Objectives

The objective is to implement a few optimization algorithms:

• Gradient descent (GD)
• Accelerated gardient descent (momentum, NAG)
• Proximal/projected gradient descent
• Stochastic gradient descent (SGD)
• ...

And develop good development practices in Python for ML.

Schedule

• Basics about Python, NumPy & Scipy + linear regression ~ 3h
• Ridge regression (GD, NAG, SGD) ~ 3h
• Elastic-net (proximal gradient descent) ~ 3h
• Gaussian mixture model (projected gradient descent, EM) ~ 3h
• Reproductibility, metrics, and evaluation ~ 3h
• Exam ~ 3h

Exam

• 3 hours in class
• Access to online documentation
• LLMs are forbidden, we're not here to learn vibe coding
• One exercise closely related to what we've done in class

Example

• A slighly different optimization algorithm
• Applying what we've done to a real world problem