Statistical Learning Theory and Sequential Prediction

Spring 2014

Time & Location: Fridays 12:30-3:30pm in JMHH F36

Instructors: Alexander Rakhlin, Karthik Sridharan

Course Description

This course will focus on theoretical aspects of Statistical Learning and Sequential Prediction (Online Learning). In the first part of the course, we will analyze learning with i.i.d. data using classical tools: concentration inequalities, random averages, covering numbers, and combinatorial parameters. We then focus on sequential prediction and develop many of the same tools for learning in this scenario. The latter part is based on recent research and offers many directions for further investigation. The minimax approach, which we emphasize throughout the course, offers a systematic way of comparing learning problems. Beyond the theoretical analysis, we will discuss learning algorithms and, in particular, an important connection between learning and optimization. Our framework will give a handle on developing near-optimal and computationally efficient algorithms. We will illustrate this on the problems of matrix completion, link prediction, and other. Time permitting, we will make excursions into Information Theory and Game Theory, and show how our new tools seamlessly yield a number of interesting results.

Prerequisites: Probability Theory and Linear Algebra.

Lecture Notes

These lecture notes are constantly evolving, so if your version says x<y today, it might say x>y tomorrow.

The "Algorithms" section will go through an overhaul this semester.

Tentative Schedule

Suggested Readings


  • Background Material
  • Statistical Learning Theory
  • Sequential Prediction / Online Learning
  • Books (not required):

  • van der Vaart and Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics.
  • Sara van de Geer. Empirical Processes in M-Estimation.
  • N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games.
  • Hiriart-Urruty and Lemaréchal. Fundamentals of Convex Analysis.
  • Other relevant courses:

  • Peter Bartlett at UC Berkeley (also the 2006 version)
  • Sham Kakade and Ambuj Tewari at TTI, UChicago
  • Maxim Raginsky at Duke
  • Shai Shalev-Shwartz at Hebrew U.
  • Dmitry Panchenko at MIT