Theoretical Foundations in Modern Machine Learning

CSCI-GA.3033-141 · Fall 2026

Instructor: Allen Liu
Meeting time: Tuesdays, 4:55–6:55 PM
Location: Bobst LL138

Course Description

This class aims to showcase various ways that theoretical insights play a role in advancing modern machine learning. Topics include scaling laws, data and compute allocation, post-training and reinforcement learning, and inference-time interventions. We will emphasize theoretical perspectives that both help us understand modern training pipelines and inform new algorithms and design principles.

Course Information

Course syllabus (PDF)

Schedule

Meeting	Theme	Detailed topic	Lecture notes	Related readings
1	LLM fundamentals	How do LLMs work, next-token prediction and autoregressive generation.	Lecture 1 (PDF)
2	LLM fundamentals	LLM architecture, self-attention and modern variations	Lecture 2 (PDF)	Attention Is All You Need Repeat After Me: Transformers are Better than State Space Models at Copying Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity Kimi Linear: An Expressive, Efficient Attention Architecture
3	Pre-training	Scaling laws I	Lecture 3 (PDF)	Scaling Laws for Neural Language Models Explaining Neural Scaling Laws Scaling Laws, Carefully A Hitchhiker's Guide to Scaling Law Estimation
4	Pre-training	Scaling laws II: data and repetition	Lecture 4 (PDF)	Internal Data Repetition Destroys Language Models
5	Pre-training	Data mixtures and data selection, metagradients		DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining Olmix: A Framework for Data Mixing Throughout LM Development Data Attribution at Scale
6	Pre-training	Data selection and metagradients (continued); systems and performance overview		MAGIC: Near-Optimal Data Attribution for Deep Learning Systems background: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness The Ultra-Scale Playbook How to Scale Your Model
7	Post-training	Introduction to post-training
8	Post-training	Policy Gradient and Policy Optimization Basics
9	Post-training
10	Post-training
11	Test-time scaling
12	Test-time scaling
13
14