Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

gist

Pedro Domingos argues that any model trained by gradient descent is approximately a kernel machine that compares a query with stored training examples. His Theorem 3 shows that, as the learning rate shrinks toward zero, the trained model equals the initial model plus a weighted sum of a path kernel over the training points, where the kernel integrates the dot product of the model's gradients along the training path. He reads deep network weights as a superposition of training examples and doubts that deep learning discovers new representations.

ideas

• The path kernel measures shared gradients. The kernel between two points integrates the dot product of the model's gradients at each point along the parameter path, and the neural tangent kernel is its integrand for multilayer perceptrons.
• The example weights depend on the query. Unlike a standard kernel machine, each example weight is a kernel-weighted average loss derivative and the bias is the initial model. The proof covers batch and stochastic gradient descent in the small-learning-rate limit.
• The result explains familiar behavior. Domingos says it accounts for deep networks performing like kernel machines, appearing to memorize training instances, and degrading as queries move away from the nearest training example.
• Representation learning is in doubt. On his reading, deep learning relies on predefined features, the gradients of a fixed function, and gradient descent only selects among them, so better representation learning remains open.
• Kernel methods gain from it too. He argues architectures can be ported into kernels, the quadratic Gram matrix is no longer needed, gradient descent works like boosting, and every convex learning problem's solution is a kernel machine.

quotes