[Comment removed by author]
Does gradient descent really do well for deep learning when the gradient is computed with respect to the whole dataset? I assumed that the noise in SGD played an important role for escaping local minima.
I did a similar thing for Julia: Llama2.jl contains vanilla Julia code [1] for training small Llama2-style models on the CPU. [1] https://github.com/cafaxo/Llama2.jl/tree/master/src/training
To me, the clearest way to think about signed vs. unsigned integers is that different representatives of the integers modulo n are chosen. For example, for 8-bit signed integers we choose the representatives -128, -127,…
Of course. Theoretically, the determinant answers the binary question "singular" or "nonsingular". Numerically, such a binary answer is pretty useless. Here, we need a measure of how singular/nonsingular a matrix is…
Edit: Sorry, I completely messed up my original answer here. A better version: Let's say we are in a setting where we only work with integers. A matrix is invertible iff its determinant is invertible in the underlying…
They define a generic "is_singular" function and test it with a 2x2 matrix. The problem with the determinant is not about performance. It is just useless for determining if a matrix is singular. The thing that gives it…
From page 6, when they describe their synthetic textbook dataset: "Consider the matrix A = np.array([[1, 2], [2, 4]]). We can check if this matrix is singular or nonsingular using the determinant function. [...]" No.…
GPT-4's explanation of its optimization does not make sense to me. It writes "Instead of moving it to P, we can directly use S in the following comparisons, saving one instruction." but then proceeds to use P as if that…
[Comment removed by author]
Does gradient descent really do well for deep learning when the gradient is computed with respect to the whole dataset? I assumed that the noise in SGD played an important role for escaping local minima.
I did a similar thing for Julia: Llama2.jl contains vanilla Julia code [1] for training small Llama2-style models on the CPU. [1] https://github.com/cafaxo/Llama2.jl/tree/master/src/training
To me, the clearest way to think about signed vs. unsigned integers is that different representatives of the integers modulo n are chosen. For example, for 8-bit signed integers we choose the representatives -128, -127,…
Of course. Theoretically, the determinant answers the binary question "singular" or "nonsingular". Numerically, such a binary answer is pretty useless. Here, we need a measure of how singular/nonsingular a matrix is…
Edit: Sorry, I completely messed up my original answer here. A better version: Let's say we are in a setting where we only work with integers. A matrix is invertible iff its determinant is invertible in the underlying…
They define a generic "is_singular" function and test it with a 2x2 matrix. The problem with the determinant is not about performance. It is just useless for determining if a matrix is singular. The thing that gives it…
From page 6, when they describe their synthetic textbook dataset: "Consider the matrix A = np.array([[1, 2], [2, 4]]). We can check if this matrix is singular or nonsingular using the determinant function. [...]" No.…
GPT-4's explanation of its optimization does not make sense to me. It writes "Instead of moving it to P, we can directly use S in the following comparisons, saving one instruction." but then proceeds to use P as if that…