└── README.md /README.md: -------------------------------------------------------------------------------- 1 | 2 | # ML@B Deep Learning Reading Group (Fall 2016) 3 | The ML@B Deep Learning Reading Group is organized by[ William Guss](http://www.wguss.ml) and [Professor {TBD sponsoring professor}](url.com) in aims to educate and inform its members in the state of the art in deep learning. The group meets two hours a week at in {TBD} and is currently limited to {TBD num members} members. The following schedule and suggested readings are open to public, as we hope to act as an aggregator of deep learning research for the UC Berkeley community. 4 | 5 | ### {DAY/TIME TBD}. {PLACE TBD}. 6 | 7 | 8 | The following are scheduled meetings the following semester and a list of suggested topics. If you plan on presenting (highly encouraged) please select a paper from the list and email wguss [at] berkeley [dot] edu or submit a pull request and add to the table! 9 | 10 | (Prior to presenting, please coordinate with Gianna (gianna [at] berkeley [dot] edu) to confirm the order for pastries at least 24 hours in advance, and please pick them up from Brewed Awakening after 10:30am - it's a catering order for Gianna - and bring them to the meeting.) 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 30 | 31 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 47 | 48 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 105 | 106 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 120 | 121 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 165 | 166 | 175 | 176 | 177 | 178 | 179 | 180 | 182 | 183 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 194 | 195 | 197 | 198 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 210 | 211 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 |
Date PresenterTopic
Jan 27 29 | 32 | 33 | all  34 | 35 | 5-10 minutes on your research (past or present or interests) 
Feb 3 Yasin 46 | 49 | 50 | [A Fast and Reliable Policy Improvement Algorithm](http://statlearning.pbworks.com/w/file/104808253/pi-aistats.pdf), Yasin Abbasi-Yadkori, Peter L. Bartlett, and Stephen Wright. Artificial Intelligence and Statistics (AISTATS), 2016.  51 | 52 |
Feb 10Billy O. Dekel, R. Eldan, T. Koren. [Bandit Smooth Convex Optimization: Improving the Bias-Variance Tradeoff.](http://tx.technion.ac.il/~tomerk/papers/bco58.pdf)
Feb 17Will  Yann Dauphin, Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.](http://arxiv.org/abs/1406.2572) 73 |
Feb 24Ben Rubinstein  84 | 85 | **12-1pm** 86 | 87 | **Title**: Private Bayesian Inference 88 | 89 |   90 | 91 | **Abstract**: Differential privacy is a leading framework for guaranteeing privacy of data when releasing aggregate statistics or models fit to data. While much is known about privatising many common learning algorithms, and frameworks such as regularised ERM, little work has focused on inference in the Bayesian setting. In that setting, the defender wishes to release a posterior on sensitive data while the untrusted third party is modelled as an adversary wishing to uncover information about the private data given query access to the defender’s release mechanism, full knowledge of the likelihood family, prior, and unbounded computation. I’ll present a natural response mechanism that simply samples from the (non-private) posterior. If either of two assumptions are met, then this mechanism is both robust and differentially private: uniformly-Lipschitz likelihoods, or a prior that concentrates on smooth likelihoods. A selection of results will be presented taken from: bounds on utility, privacy; necessary conditions; examples of common distributions; and specialisation to graphical models and alternate mechanisms which demonstrate the influence of graph structure on privacy. This is joint work with Christos Dimitrakakis, Zuhe Zhang, Katerina Mitrokotsa, Blaine Nelson; papers at ALT’14 (longer version with corrections in submission to JMLR) and AAAI’16. 92 | 93 |    94 | 95 |
Mar 2 Aldo 104 | 107 | 108 | [Learning Polynomials with Neural Networks.](http://jmlr.org/proceedings/papers/v32/andoni14.pdf) Alexandr Andoni, Rina Panigrahy, Gregory Valiant, Li Zhang. Proceedings of the 31st International Conference on Machine Learning, Beijing, China, 2014\. JMLR: W&CP volume 32.  109 | 110 |
Mar 9 Alan 119 | 122 | 123 | (1 hour: 11-12) COLT paper: Minimax Linear Regression 124 | 125 |
Mar 16 134 | 135 | **-- no meeting -- ** 136 | 137 |
Mar 23**-- no meeting -- ** 
Mar 30Sören Deep Online Convex Optimization by Putting Forecaster to Sleep. D.  Balduzzi. [https://dl.dropboxusercontent.com/u/5874168/doco.pdf](https://dl.dropboxusercontent.com/u/5874168/doco.pdf)
Apr 6 Xiang 164 | 167 | 168 | Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, and Yann LeCun.  [The loss surface of multilayer networks](http://arxiv.org/abs/1412.0233). See also: 169 | 170 | * [Complexity of random smooth functions on the high-dimensional sphere](http://arxiv.org/abs/1110.5872).  A. Auffinger and G. Ben Arous. October 2011.  171 | 172 | * [Random Matrices and complexity of Spin Glasses](http://arxiv.org/abs/1003.1129).  A. Auffinger, G. Ben Arous, and J. Cerny. March 2010. 173 | 174 |  
Apr 13 181 |  Thomas 184 | (1 hour: 11-12) Sparse and spurious: dictionary learning with noise and outliers. Rémi Gribonval, Rodolphe Jenatton, Francis Bach. 2014. [https://hal.inria.fr/hal-01025503v3](https://hal.inria.fr/hal-01025503v3)
Apr 20 193 |  Arturo 196 | [Proximal Algorithms](http://stanford.edu/~boyd/papers/pdf/prox_algs.pdf).  N. Parikh and S. Boyd.  Foundations and Trends in Optimization, 1(3):123-231, 2014. 199 | 200 |
Apr 27 Niladri 209 | Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization. 2013 - Alekh Agarwal, Animashree Anandkumar, Prateek Jain, Praneeth Netrapalli, and Rashish Tandon. [http://arxiv.org/abs/1310.7991](http://arxiv.org/abs/1310.7991) 212 |
May 4**-- no meeting -- ** 
May 11**-- no meeting -- ** 
May 18Walid 239 | 240 | On a Natural Dynamics for Linear Programming. Damian Straszak, Nisheeth K. Vishnoi. [http://arxiv.org/abs/1511.07020](http://arxiv.org/abs/1511.07020)  241 | 242 |
May 25
255 | 256 | ### Suggested topics/papers: 257 | 258 | #### Neural networks: 259 | 260 | * Feedback Stabilization Using Two-Hidden-Layer Nets.  Eduardo D. Sontag. IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 6, 981-, 1992.  261 | 262 | * Learning Polynomials with Neural Networks. Alexandr Andoni, Rina Panigrahy, Gregory Valiant, Li Zhang. Proceedings of the 31st International Conference on Machine Learning, Beijing, China, 2014\. JMLR: W&CP volume 32. 263 | 264 | * Deep Online Convex Optimization by Putting Forecaster to Sleep. D.  Balduzzi. [https://dl.dropboxusercontent.com/u/5874168/doco.pdf](https://dl.dropboxusercontent.com/u/5874168/doco.pdf) 265 | 266 | * Provable bounds for learning some deep representations.  Sanjeev Arora, Aditya Bhaskara, Rong Ge, and Tengyu Ma.  CoRR, abs/1310.6343, 2013.  267 | 268 | * Large-Scale Convex Minimization with a Low-Rank Constraint.  Shai Shalev-Shwartz, Alon Gonen, Ohad Shamir.  Proceedings of the 28 th International Conference on Machine Learning, Bellevue, WA, USA, 2011.  269 | 270 | * Yann Dauphin, Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. CoRR, abs/1406.2572, 2014. 271 | 272 | * Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, and Yann LeCun.  The loss surface of multilayer networks. CoRR, abs/1412.0233, 2014. See also: 273 | 274 | * Complexity of random smooth functions on the high-dimensional sphere.  A. Auffinger and G. Ben Arous.  ArXiv e-prints, October 2011\. arXiv math.PR 1110.5872. 275 | 276 | * Random Matrices and complexity of Spin Glasses.  A. Auffinger, G. Ben Arous, and J. Cerny.  ArXiv e-prints, March 2010\. arXiv math.PR1003.1129. 277 | 278 | * Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. CoRR, abs/1509.01240, 2015. 279 | 280 | * Matus Telgarsky.  Representation benefits of deep feedforward networks.  abs/1509.08101, 2015. 281 | 282 | * Pierre Baldi and Peter J Sadowski. Understanding dropout. In C.J.C.  Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 2814–2822\. Curran Associates, Inc., 2013. 283 | 284 | * Stefan Wager, Sida Wang, and Percy Liang. Dropout training as adaptive regularization.  In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q.  Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 351–359\.  2013. 285 | 286 | #### Dictionary learning: 287 | 288 | * J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research, 11(1):19–60, 2010. [www.di.ens.fr/~fbach/mairal10a.pdf](http://www.di.ens.fr/~fbach/mairal10a.pdf) 289 | 290 | _see also:_ 291 | 292 | * Alekh Agarwal, Animashree Anandkumar, Prateek Jain, Praneeth Netrapalli, and Rashish Tandon. Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization. 2013. [http://arxiv.org/abs/1310.7991](http://arxiv.org/abs/1310.7991) 293 | * Sanjeev Arora, Rong Ge, and Ankur Moitra. New Algorithms for Learning Incoherent and Overcomplete Dictionaries. 2013. [http://arxiv.org/abs/1308.6273](http://arxiv.org/abs/1308.6273) 294 | * Alekh Agarwal, Animashree Anandkumar, and Praneeth Netrapalli. Exact Recovery of Sparsely Used Overcomplete Dictionaries. 2013. [http://arxiv.org/abs/1309.1952v2](http://arxiv.org/abs/1309.1952v2) 295 | * Dictionary Learning Algorithms for Sparse Representation. Kenneth Kreutz-Delgado, Joseph F. Murray, Bhaskar D. Rao, Kjersti Engan, Te-Won Lee, and Terrence J. Sejnowski. Neural Comput. 2003 Feb; 15(2): 349–396\. doi:  10.1162/089976603762552951 296 | 297 | ADMM and Proximal Algorithms: 298 | 299 | * Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. S. Boyd, N. Parikh, E. Chu, B.  Peleato, and J. Eckstein. Foundations and Trends in Machine Learning, 3(1):1–122, 2011. [http://stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf](http://stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf)  300 | * Proximal Algorithms.  N. Parikh and S. Boyd.  Foundations and Trends in Optimization, 1(3):123-231, 2014. [http://stanford.edu/~boyd/papers/pdf/prox_algs.pdf](http://stanford.edu/~boyd/papers/pdf/prox_algs.pdf) 301 | 302 | Computational Implications of Reducing Data to Sufficient Statistics. Andrea Montanari. [http://arxiv.org/abs/1409.3821](http://arxiv.org/abs/1409.3821) 303 | 304 | The power of localization for efficiently learning linear separators with malicious noise.  Pranjal Awasthi, Maria-Florina Balcan, and Philip M. Long.  CoRR, abs/1307.8371, 2013.  305 | 306 | Cortical prediction markets. D. Balduzzi. 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS).[https://dl.dropboxusercontent.com/u/5874168/nmarkets.pdf](https://dl.dropboxusercontent.com/u/5874168/nmarkets.pdf) 307 | 308 | --------------------------------------------------------------------------------