Course Review SC 607

SC 607 - Optimisation

Year: 2023-24 Spring Semester
Instructor: Prof. Avishek Ghosh

Motivation

Mathematical optimization is a cornerstone of modern science and engineering, playing a pivotal role in a wide range of applications in machine learning. By mastering these methods, students can enhance their analytical skills, develop efficient algorithms, and contribute to advancements in various fields where optimal decision-making is crucial. Whether you are aiming to improve resource allocation, streamline processes, or innovate in technology, a deep understanding of optimization will empower you to achieve superior outcomes and drive impactful solutions.

Course Content

The official course content can be found here. The updated syllabus is listed below:

• Basics of real analysis, series and sequences, supremum and infimum, Open and closed set, Continuity, limits, differentiability, local minima and Taylor theorem.
• Line segment, Affine set, Convex set, Convex hull, Cones, Hyperplanes, Eucledian balls, Ellipsoids, Norm balls, Polyhedra, PSD cone, Affine functions, Separating and supporting hyperplane theorems
• Convex function, Strong convexity, First order characterizations, Second order conditions, Multivariate calculus, Examples of convex function in n dimesional real vector space (Norms, Quadratic over linear, Max, Log-sum-exp, GM, Log-det), Sublevel sets, Epigraph, Jensen’s inequality
• Operations that preserve convexity (Non negative weighted sum, Composition, Pointwise supremum and infimum), Conjugate functions, Quasi-convex functions, Log convex and Log convex functions
• Constrained optimisation (Single equality, Single inequality, Two inequality), Lagrangian and Lagrange multipliers, Tangent cone, Constraint qualifications, LICQ, KKT conditions and its proof, Farkas lemma, Critical cone and second order conditions, MFCQ, Normal cone.
• Lagrange dual function, Connection between dual function and conjugate function, Lagrange dual problem, Weak duality, Strong duality and Slater’s CQ, Minimax interpretation of duality, KKT optimality conditions for primal-dual problems, Linear programs in its primal and dual form.
• Unconstrained optimisation, Newton’s method for equality constraints, Interior point method, Log barrier function, Centrol path, Eucledian projection, Projected gradient descent method, Convergence analysis of PGD, Frank-Wolfe method, Augmented Lagrangian method, ADMM.

Feedback on Lectures

• Teaching Style: As a first-time instructor, there were moments of underconfidence, yet his solid communication skills and willingness to address and resolve doubts spontaneously created a supportive learning environment. While his teaching skills are still developing, and there is room for improvement, particularly in providing more visual interpretations rather than relying solely on writing on the board, his dedication to helping students understand the material is evident. The use of board work, often copied from his notebook, could benefit from a more dynamic and visually engaging approach to enhance comprehension and retention.
• Attendence: Not Taken.

Feedback on Assignments and Exams

• Weightage: Assignments - 25%, Midsem - 25%, Lecture scribes - 10%, Participation - 10%, Endsem - 30%
• Pattern: The course assignments featured a mix of straightforward, lollipop questions while the mid-semester exam was exceptionally challenging, out-of-the-box problems that truly tested students’ understanding. The end-semester exam, while still proof-based, was relatively easier in comparison. Evaluation was lenient, with partial credit awarded, ensuring that students’ efforts were recognized even if complete solutions were not achieved.

Difficulty Level

The course strikes a balance between accessibility and challenge. While the mini projects are manageable and offer practical insights, the exams are notably more demanding, requiring a solid grasp of mathematical concepts. To excel in these assessments, a significant amount of self-study is essential. Overall, the course is of moderate difficulty, providing a comprehensive and rewarding learning experience for those willing to invest the necessary time and effort.

Prerequisites

The course is not excessively difficult but requires dedicated self-study to tackle textbook problems effectively. Given its proof-based nature, significant memorization and understanding of concepts are essential. The problems are not straightforward and demand considerable practice at home. However, the assignments are manageable and not time-consuming. While the exams are challenging and require sincere preparation, overall, the course is manageable and considered lighter compared to other advanced courses, offering a balanced and rewarding learning experience.

Grading Stats

Reference Books

• Convex Optimization by Stephen Boyd and Lieven Vandenberghe
• Numerical Optimisation by Wright
• Optimisation for Data Analysis by Wright

Reviewed by

Soumen Mondal (Email: 23m2157@iitb.ac.in)