Fall 2018

Non-convex optimization and deep learning

WORKSHOP & BOOTCAMP

Registration closed

Synopsis

Deep learning is transforming modern machine learning but many of its aspects still escape our theoretical understanding. The goal of this workshop is to bring together theory experts that attack this question from various angles: statistical, computational and representational. Specific topics include: generalization bounds for deep learning, robustness to adversarial corruptions, non-convex optimization and representation abilities of deep neural nets. It will explore both the strength and the limitations of various approaches.
Tangible outcomes from this multisciplinary workshop include not only well defined strategies to tackle the challenges of deep learning but also the exchange of mathematical techniques that may be the key to unlock various problems.
The workshop will be preceded by a one day bootcamp on Sunday January 27 , with the goal of presenting the basic techniques, definitions and goals in several of the communities.

Program

The workshop will take place at MIT on January 28-30, 2019. Bootcamp on January 27. Both are in room 34-101.

Bootcamp: Sunday, January 27

Time Speaker Title
9:30 - 10:00 Refreshments
10:00 - 11:00 Nati Srebro Inductive Bias and Generalization in Deep Learning
11:10 - 12:10 Rong Ge Optimization Landscape: Symmetry, Saddle Points and Beyond
12:15 - 2:00 Lunch Break
2:00 - 3:00 Andrea Montanari Mean Field Description of Two-layers Neural Networks
3:00 - 4:00 Maxim Raginsky Langevin Diffusions in Non-convex Risk Minimization

Workshop, Day 1: Monday, January 28

Time Speaker Title
9:30 - 9:45 Refreshments
9:45 - 10:00 Introduction
10:00 - 10:45 Ohad Shamir Local Minima and Optimization of Neural Networks
10:45 - 11:15 Coffee Break
11:15 - 12:00 Alexander Rakhlin Generalization, Interpolation, and Neural Nets
12:00 - 2:00 Lunch Break
2:00 - 2:45 Elad Hazan New Provable Algorithms for Control
2:45 - 3:15 Coffee Break
3:15 - 4:00 Joan Bruna Birth-Death Processes in Neural Network Optimization Dynamics
4:00 - 4:45 Maryam Fazel Convergence of Gradient-based Methods for the Linear Quadratic Regulator
4:45 - 6:30 Poster Session

Workshop, Day 2: Tuesday, January 29

Time Speaker Title
9:00 - 9:30 Refreshments
9:30 - 10:15 Sanjeev Arora Theory for Representation Learning
10:15 - 11:00 Matus Telgarsky Gradient Descent Aligns the Layers of Deep Linear Networks
11:00 - 11:30 Coffee Break
11:30 - 12:15 Suriya Gunasekar Optimization Bias in Linear Convolutional Networks
12:15 - 2:15 Lunch Break
2:15 - 3:00 Jason Lee Towards a Foundation of Deep Learning: SGD, Overparametrization, and Generalization
3:00 - 3:45 Stefanie Jegelka Representational Power of Narrow ResNet and of Graph Neural Networks
3:45 - 4:15 Coffee Break
4:15 - 5:00 Santosh Vempala Is There a Tractable (and Interesting) Theory of Nonconvex Optimization?
5:00 - 6:00 Panel Panelists: Sanjeev Arora, Andrea Montanari, Katya Scheinberg, Nati Srebro, and Antonio Torralba

Workshop, Day 3: Wednesday, January 30

Time Speaker Title
9:00 - 9:30 Refreshments
9:30 - 10:15 Ankur Moitra Learning Restricted Boltzmann Machines
10:15 - 11:00 Rachel Ward SGD with AdaGrad Adaptive Learning Rate: Strong Convergence without Step-size Tuning
11:00 - 11:30 Coffee Break
11:30 - 12:15 Satyen Kale Understanding adaptive methods for non-convex optimization
12:15 - 2:15 Lunch Break
2:15 - 3:00 Sebastien Bubeck Adversarial Examples from Computational Constraints
3:00 - 3:45 Suvrit Sra A Critical View of Global and Local Optimality in Deep Networks
3:45 - 4:15 Coffee Break

Posters

Author Title
Raman AroraOn the implicit bias of dropout
Arjun Nitin BhagojiPAC-learning in the presence of evasion adversaries
Diego CifuentesThe stability of semidefinite relaxations
Ghazal FazelniaConvex Relaxation for Probabilistic Inference and Variational Autoencoders
Dylan FosterUniform Convergence of Gradients for Non-Convex Learning and Optimization
Joseph GaudioAccelerated Learning in the Presence of Time Varying Features
maxime gazeau A general system of differential equations to model first order adaptive algorithms
Maryclare GriffinPathwise Coordinate Descent for Power Penalized Regression
Sumeet KatariyaMulti-armed Bandits for Preference Learning
Justin KhimAdversarial Risk Bounds via Function Transformation
Xiaoyu LiOn the convergence of Stochastic Gradient Descent with Adaptive Stepsizes
Hongzhou LinResNet with one-neuron hidden layers is a Universal Approximator
Elahesadat NaghibInfinite-Dimensional Fourier Constrained Optimization
Sudeep Raja PuttaExponential Weights on the Hypercube in Polynomial Time
Ali Ramezani-KebryaOn the Stability and Convergence of Stochastic Gradient Descent with Momentum
Akshay RangamaniA Measure of Flatness for Deep Network Minima Invariant to Reparameterizations
David P. SandersRigorous global optimization with interval methods in Julia
Thiago SerraEmpirical Bounds on Linear Regions of Deep Rectifier Networks
Luca VenturiSpurious Valleys in Two-layer Neural Network Optimization Landscapes
Wenda ZhouCompressibility and Generalization in Large-Scale Deep Learning

Organizers

  • Joan Bruna (New-York Univeristy)
  • Constantinos Daskalakis (MIT)
  • Stefanie Jegelka (MIT)
  • Aleksander Madry (MIT) -- Lead organizer
  • Ankur Moitra (MIT)
  • Alexander Rakhlin (MIT)
  • Shai Shalev-Shwartz (Hebrew University of Jerusalem)
  • Yaron Singer (Harvard University)
  • Harrison Zhou (Yale University)

Location and maps

Click here for location of room 34-101 on interactive campus map. The street address of the building is 50 Vassar Street, Cambridge, MA 02139. Below are directions from the Marriott Hotel

Because the workshop takes place on a weekend, many of the outside doors will be locked. MIT is unlocking some specific doors for us to enter through. These doors are the ones facing Vassar Street.

Once inside, you should be able to see our registration table in the lobby. The classroom is directly opposite the entrace to the building on Vassar Street.

Local restaurants and coffee shops

The Kendall Square area has many good food and coffee options. Here are a few options within walking distance of the workshop.

  • CAVA ($): Bowls of grains and greens with a greek flare.
  • Catalyst ($$$): Elegant American cuisine.
  • Commonwealth Market & Restaurant ($$$): Trendy, rustic market & American eatery.
  • Cafe Luna ($$): Café/Lounge for sandwiches & brunch.
  • Area 4 ($$): Artisanal pizza and coffee.
  • Abigail’s ($$): American sandwiches and BBQ.
  • Vester ($$): Hip cafe with food options (sandwiches, soups and salads)
  • Dumpling Daughter ($): Modern chinese
  • Cambridge Brewing Company ($$): Pub with local craft beer from on-site brewery.
  • Sulmona ($$): Modern Italian dining.
  • Legal Sea Foods ($$): Seafood restaurant. Popular with tourists to Boston.
  • Helmand ($$): Afghani Restaurant. Dinner only, reservation needed.
  • The Friendly Toast ($$): Hip, retro breakfast spot.
  • Mamaleh's Delicatessen ($): Hip Jewish-ish deli
  • Saloniki Greek ($): Hip, Greek plates, Gyros & more
  • Clover food Lab ($): Casual, fast-food vegetarian using local ingredients.
  • Tatte ($): Gourmet high-end bakery, sandwiches, salads, and coffee shop.
  • Darwin's ($): Coffee and sandwich shop.
  • Flour Bakery ($): Coffee and high-end pastry shop.
  • Coffee: From the list above, Vester, Darwin's, Area 4, and Flour also double as good coffee shops.