#61: Prof. YANN LECUN: Interpolation, Extrapolation and Linearisation (w/ Dr. Randall Balestriero)

Episode Overview

• The podcast presents a deep dive into the debate on whether deep learning models interpolate or extrapolate, arguing that traditional definitions are flawed in high-dimensional spaces.
• Professor Yann LeCun contends that the primary limitations often attributed to deep learning are actually weaknesses of the supervised learning paradigm, advocating for self-supervised learning as the future.
• The conversation re-frames complex reasoning and planning not as symbolic manipulation, but as a continuous optimization problem solvable with gradient-based methods.
• The discussion highlights the critical role of human engineering in creating architectural priors (like CNNs) and the major challenges for AI, including learning world models and representing uncertainty.

Key Concepts

• Interpolation vs. Extrapolation in High Dimensions: The core argument is that our low-dimensional intuition about interpolation (points within a convex hull) is misleading. In high-dimensional spaces, nearly all new data points are outside the training data's convex hull, making most generalization a form of extrapolation. The goal of machine learning is to find "interpolative representations" where the data lies on a learned, lower-dimensional manifold.
• Critique of Supervised Learning: Many perceived failures of deep learning are attributed not to the models themselves but to the supervised learning paradigm, which is data-inefficient, task-specific, and struggles with out-of-distribution generalization.
• The Future is Self-Supervised Learning (SSL): SSL is presented as the path forward, enabling models to learn the structure of the world and build predictive models from vast amounts of unlabeled data.
• Reasoning as Continuous Optimization: Planning and reasoning are framed as the process of minimizing an energy function or cost over a sequence of actions or latent variables. This perspective makes reasoning compatible with gradient-based learning methods like backpropagation.
• Neural Networks as Piecewise Linear Functions: Models using ReLU activations are fundamentally piecewise linear, operating by partitioning the input space into a massive number of linear regions using hyperplanes.
• System 1 vs. System 2 Thinking: The speakers connect AI capabilities to human cognition, where current models excel at fast, intuitive "System 1" tasks. Human learning is described as compiling deliberate, model-based "System 2" planning into reactive "System 1" policies through practice.
• The Importance of Human Engineering: The success of modern AI is not solely due to learning algorithms but also relies heavily on human-designed architectural priors, such as convolutional neural networks, which provide essential structure.
• Major AI Challenges: Key obstacles for advancing AI include learning robust world models from observation, effectively representing and handling uncertainty, and combining discrete and continuous reasoning for complex planning.

Quotes

• At 1:42 - "In a high dimensional space, there is essentially no such thing as interpolation, everything is extrapolation." - LeCun's central argument that the traditional definitions of interpolation and extrapolation break down due to the curse of dimensionality.
• At 28:18 - "All feature engineering and representation learning in machine learning is about finding these interpolative representations." - The speaker introduces the central theme of this section, framing machine learning's goal as transforming data into a space where interpolation becomes a powerful tool for generalization.
• At 95:09 - "Basically, that's called the Kelley-Bryson algorithm... and it consists in basically doing backprop through time. It's as simple as that." - Yann LeCun explains that the core method for optimal control is fundamentally the same as backpropagation through time, linking classical planning to modern deep learning.
• At 1:04:50 - "Most of the criticism that I've heard from Gary Marcus and several others towards deep learning is not a criticism towards deep learning, it's a criticism towards supervised learning. And I agree with them. Supervised learning sucks." - LeCun redirects the blame for AI's limitations from the architecture (deep learning) to the learning paradigm (supervised learning).
• At 127:31 - "We discount all the human engineering that has gone into actually making machine learning work through specific architectures." - The speaker points out that the success of machine learning is heavily reliant on human-designed priors and architectures.

Takeaways

• Rethink interpolation in high dimensions; successful generalization depends on learning a meaningful data manifold, not just fitting points within a predefined boundary.
• The limitations of today's AI systems are often rooted in the supervised learning paradigm, not deep learning architectures. The shift towards self-supervised learning is crucial for building more general world models.
• Viewing reasoning as a continuous optimization problem, rather than a purely symbolic one, opens the door for building systems that can plan and reason using gradient-based methods.
• Human ingenuity in designing model architectures and problem representations remains a critical, and often overlooked, driver of progress in artificial intelligence.