Statistics of Big Data

Semester 2 2018
Wednesday 13-16, Kaplun 118
Home page on ∼ saharon/BigData.html
Lecturer: Saharon Rosset
Schreiber 022
Office hrs: Thursday 16-18 (with email coordination)

Announcements and handouts

Signup link for take home final (deadline 1 June)

(7 March) Homework 0 (warmup) is now available. Due 21/3 in class. Submission in pairs is encouraged (but not in triplets or larger, please).
This homework uses the Nature paper from 2009 introducing Google Flu Trends (GFT), and the Science paper from 2014 describing the failure of GFT since 2011.
(14 March) R code for investigating privacy of summary statistics release in GWAS.
The differential privacy topic started this week and continuing next is based on The Algorithmic Foundations of Differential Privacy by Dwork and Roth
(21 March) Paper by Wasserman and Zhou on statistical theory of differential privacy.
(26 March) Homework 1 is now available, due on 11 April in class. Submission in pairs is encouraged. This 2009 paper by Jacobs et al. may be used as reference for problem 1.
The next two classes (26/3 and 11/4) will deal with high dimensional modeling (large p, p >> n). We will discuss the statistical and computational challenges that are unique to this setting and some of the most popular solutions. Relevant reading materials include chapters 2-3 of ESL, this review I wrote on sparse modeling, and the papers on LARS by Efron et al. and its generalization by Rosset and Zhu.
(11 April) Homework 2 is now available. Due 25 April in class.
Problem 1 uses train.csv and test.csv datasets, and there is also sample code in sparse.r.
Problem 2 extra credit uses this paper.
(25 April) The next two classes (25/4 and 2/5) will include a brief introduction to Deep Learning methodology and applications. The relevant reading materials for this week include Chapter 18 of CASI and Giora Simchoni's blog entry.
(2 May) Moni's presentation from class today.
Recommended reading:
Visualizing and Understanding Convolutional Networks
Efficient Estimation of Word Representations in Vector Space
Generative Adversarial Networks (Important topic we did not get to discuss)
(7 May) Homework 3 is now available. Due 23 May in class.
It uses the code HW3-1.r (which requires installing the Keras R package, and also Python if you don't have it) and HW3-2.r.
Note: Unlike previous homeworks, this one was prepared from scratch (thanks to Moni for his help!). So despite our efforts, there might be problems or issues. If you find any, please let me know. If you find a major problem and propose an appropriate way of fixing it, you may also get a bonus on the homework!
(9 May)
Today's class uses the survey by Goldberg et al. (that appeared in 2010 in the Foundations and Trends in Machine Learning series).
Code from class for fitting models to the Sampson monks and E-Coli networks.
(16 May)
Today's class focuses on multiple testing and selective inference in big data settings.
Slides on quality preserving databases.
Some slides from Yoav Benjamini that cover many aspects of the discussion: Part 1 (pdf), Part 2 (pptx)
Why most published research is wrong by Ioannidis
Signup link for take home final (deadline 1 June)


The goal of this course is to present some of the unique statistical challenges that the new era of Big Data brings, and discuss their solutions. The course will be a topics course, meaning we will discuss various aspects that may not necessarily be related or linearly organized. Our challenge will be to cover a wide range of topics, while being specific and concrete in describing the statistical aspects of the problems and the proposed solutions, and discussing these solutions critically. We will also keep in mind other practical aspects like computation and demonstrate the ideas and results on real data when possible. Accordingly, the homework and the final exam will include a combination of hands-on programming and modeling with theoretical analysis.
Big Data is a general and rather vague term, typically referring to data and problems that share some of the following characteristics:
Some examples of typical Big Data domains gaining importance in recent years:
A key topic in data modeling in general and Big Data in particular is predictive modeling (regression, classification). Since the course Statistical Learning (given last year and next year) deals mainly with exposition and statistical analysis of algorithms in this area, it will not be a focus of this course. However, some aspects of this area that are not covered in that course, in particular the p >> n case, efficient computation, and deep learning, will be discussed in some detail.
Tentative list of topics to be covered during the semester:
We will have 3-5 guest lectures during the semester, but they will be treated as regular classes rather than enrichment classes (specifically, their material will be included in the homework and the final).

Expected background

  1. Basic knowledge of mathematical foundations:
  2. Solid fundamentals in Probability: Discrete/continuous probability definitions; Important distributions: Bernoulli/Binomial, Poisson, Geometric, Hypergeometric, Negative Binomial, Normal, Exponential/Double Exponential (Laplace), Uniform, Beta, Gamma, etc.; Limit laws: large numbers and CLT; Inequalities: Markov, Chebyshev, Hoeffding
  3. Solid fundamentals in Statistics:

Books and resources

The course does not have a specific textbook, and most lectures will be on the board and not using slides. Some of the material will closely follow chapters from books or published papers, and when this is the case it will be announced. However, it is critical that all students have all the material presented in class. If you miss classes, make sure to get the material from someone!
Relevant books:
Elements of Statistical Learning by Hastie, Tibshirani & Friedman. Including freely available pdf, data and errata)
Modern Applied Statistics with Splus by Venables and Ripley
Frontiers in Massive Data Analysis report from the National Research Council
Computer Age Statistical Inference by Efron and Hastie


There will be four-five homework assignments, which will count for about 30% of the final grade, and a final take-home exam. Both the homework and the exam will combine theoretical analysis with hands-on data analysis.


The course will require use of statistical modeling software. It is strongly recommended to use R (freely available for PC/Unix/Mac).
R Project website also contains extensive documentation.
A basic "getting you started in R" tutorial. Uses the Boston Housing Data (thanks to Giles Hooker).
Modern Applied Statistics with Splus by Venables and Ripley is an excellent source for statistical computing help for R/Splus.

File translated from TEX by TTH, version 4.08.
On 16 May 2018, 13:06.