Mart´ın Abadi, Ulfar Erlingsson, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Nicolas Papernot∗, Kunal Talwar, and Li Zhang

What’s the paper about:

A look at some of the older ideas in privacy and how they can help with keeping private the training data that machine learning models need. Particularly looking at Saltzer and Shroeder’s early work from the 70s.


  • privacy = "socially defined ability of an individual (or organization) to determine whether, when, and to whom personal (or organizational) information is to be released
  • security = “body of techniques for controlling the use or modification of computers or information”

Looking at two papers on privacy of training data for contemporary deep learning, which use noisy SGD and private aggregation of teacher ensembles (PATE). Both noisy SGD and PATE rely on differential privacy. For mitigating attacks, Saltzer and Schroeder found some useful principles for building secure systems. Problem

  • Referring to supervised classification
  • where f is some direct mapping of example to class, want to learn a function g (model) that approximates f
  • two distinct goals of attacks
  • getting some/all of training data from model g being able to check if the input/output pair is part of training data
  • want to prevent both but focus on membership tests

Differential privacy? (define!) Two types of threats that are worth considering:

  • black-box: attackers can apply model g to new inputs  
  • white-box: attackers can directly see the internals of model g

White-box threats subsume black-box threats (more severe)

  • attackers with access to model internals can apply model to infinite number of inputs
  • focus on white-box threats since does not refer to restrictions

Noisy SGD

  • many ML techniques need parametric functions as models
  • function g takes as input parameter \delta and example \x and outputs class g(\delta, \x)