Elements of Statistical Learning by Hastie,
Tibshirani & Friedman
Announcements and handouts
from class 1 and
code I demonstrated in class. You can also get
the raw code. (5 March)Homework 1 is now available. Due 9 April in class. Competition instructions and
sample code. (21 March)Homework 2 is now available. It uses the notes on quantile regression. Due 23 April in class. Slides
on geometry of linear regression and its bias-variance
for fitting ridge, lasso and PCA regression on our competition data,
for those who want an early start. I will discuss this code in the
next class on 9/4. (5 May)Homework 3 is now available. This is now the final version
(virtually unchanged from preliminary one).
(7 May) Class presentations:
Wallet estimation using quantile regression and
KDD-Cup 2007. Class notes on Poisson distribution and variance stabilizing
(19 May) Competition update: Moving week in our competition! We
now have four teams in the bonus, with the leader at 0.7656.
for fitting trees and bagging to our competition data.
(26 May) Competition update
: New leaders are well clear of the
field at 0.7597. Still four teams total in the bonus.
(10 June) Competition update: leaders are still at 0.7597,
still well clear of the field. We already have seven teams in the
(10 June)Homework 4 is updated with questions about
for fitting boosted trees and SVR to our competition data. Note: Teams that have passed the bonus threshold in the
competition are welcome to use boosting to improve their scores.
However, I will not accept new submissions that accomplish the bonus
by using boosting. In other words, you first have to break the 0.77
barrier without using it.
The goal of this course is to gain familiarity with the basic ideas and
methodologies of statistical (machine) learning. The focus is on
supervised learning and predictive modeling, i.e., fitting y ≈ ∧f(x), in regression
We will start by thinking about some of the simpler, but still highly effective methods, like nearest
neighbors and linear regression, and gradually learn about more complex and "modern"
methods and their close relationships with the simpler ones.
As time permits, we will also cover one or more industrial
"case studies" where we track the process from problem definition, through
development of appropriate methodology and its implementation, to deployment
of the solution and examination of its success in practice.
The homework and exam will combine hands-on programming and modeling with
theoretical analysis. Topics list:
Introduction (text chap. 1,2): Local vs. global modeling; Overview of statistical considerations: Curse of dimensionality, bias-variance tradeoff; Selection of loss functions; Basis expansions and kernels
Linear methods for regression and their extensions (text chap. 3): Regularization, shrinkage and principal components regression; Quantile regression
Linear methods for classification (text chap. 4): Linear discriminant analysis; Logistic regression; Linear support vector machines (SVM)
Classification and regression trees (text chap. 9.2)
Model assessment and selection (text chap. 7): Bias-variance decomposition; In-sample error estimates, including Cp and BIC; Cross validation; Bootstrap methods
Basis expansions, regularization and kernel methods (text chap. 5,6): Splines and polynomials; Reproducing kernel Hilbert spaces and non-linear SVM
Committee methods in embedded spaces (material from chaps 8-10): Bagging and boosting
Case studies: Customer wallet estimation; Netflix prize competition; maybe others...
Basic knowledge of mathematical foundations: Calculus; Linear Algebra; Geometry
Undergraduate courses in: Probability; Theoretical Statistics
Statistical programming experience in R is not a prerequisite,
but an advantage
Other recommended books: Modern Applied Statistics with Splus by Venables and Ripley Neural Networks for Pattern Recognition by Bishop
(Several other books on Pattern Recognition contain similar material) All of Statistics and All of Nonparametric Statistics by Wasserman
There will be about four homework assignments, which will count for
about 30% of the final grade, and a final take-home exam. Both the
homework and the exam will combine theoretical analysis with
hands-on data analysis.
We will also have an optional data modeling competition, whose
winners will get a boost in grade and present to the whole class.