In this post, I intended to explain in great detail why Autograd is considered to be as effective as it actually is. Fortunately, I don’t have enough time right now, so I’ll briefly explain its weaknesses.
If we need higher-order derivatives, it means we have few inputs and many outputs.
When do we actually need higher-order derivatives?
Algorithms such as K-FAC or L-BFGS where Hessian second derivatives should be calculated. Fortunately, Hessian higher-order calculations usually are approximated instead of computing exactly.
In NumPy, np.dot and np.matmul (or the @ operator) often give the same results for 1D and 2D arrays,
but they are not interchangeable in general.
Many people still think of a matrix as a tensor.
Indeed, matrix = tensor, but tensor matrix
That’s why the tensor breaks up with the scalar:
It said: “You have no dimensions to your personality. I need someone who understands my gradients.”
Sometimes even I visit, Math is Fun