CS 480/680 - Introduction to Machine Learning - Spring 2022

Logistics

Weekly Schedule

There will be two lectures each week, on Monday and Wednesday.
There will also be one assignment posted (roughly) every other week, for a total of five.
There will be a midterm (on June 21) and a final exam (date TBD).
There will be a final project (for CS 680 only) at the end of the course.
All deliverables are due at 11:59 PM Waterloo time on the stated date.

Lectures

All future lecture are tentative. ISL is the most recommended reading when available.
Lecture Number Date Topic Slides Readings
1 5/2/2022 Logistics and Intro, Perceptron PDF, PDF Lecture 1 of Yaoliang's notes
UML Section 9.1
ESL Section 4.5
2 5/4/2022 Perceptron continued, Linear Regression PDF UML Section 9.2, 11.2
ESL Section 3.2, 3.4, 7.10
ISL Section 3.1-3.2, 5.1, 6.2
Calculus derivation
3 5/9/2022 Finish Linear Regression PDF Same as last lecture
4 5/11/2022 k-Nearest Neighbour Classification, Logistic Regression PDF, PDF UML Section 19
ESL Section 2.3.2, 13.3
ISL Section 2.2

UML Section 9.3
ESL Section 4.4
ISL Section 4.3
5 5/16/2022 Finish Logistic Regression PDF Last lecture's logistic regression content
6 5/18/2022 Hard-Margin Support Vector Machines PDF UML Section 15
ESL Section 12.1-12.3
ISL Section 9.1-9.2
7 5/25/2022 Soft-Margin Support Vector Machines, start Kernels PDF Last lecture's SVM content

UML Section 16
ESL Section 12.3
ISL Section 9.3
8 5/29/2022 Finish Kernels, start Decision trees PDF Last lecture's Kernels content

UML Section 18
ESL Section 9.2
ISL Section 8.1
9 5/31/2022 Finish Decision trees, Bagging PDF Last lecture's Decision Tree content

UML Section 10
ESL Section 8.2, 8.7, 10
ISL Section 8.2
10 6/6/2022 Boosting PDF Last lecture's Boosting content
11 6/8/2022 Multilayer Perceptrons PDF DL Section 6
D2L Section 4
UML Section 20
ISL Section 10.1, 10.2, 10.7
3Blue1Brown videos on backprop: 1, 2
12 6/13/2022 Deep Networks PDF DL Section 7, 8
D2L Section 5, 11
An overview of gradient descent optimization algorithms, by Sebastian Ruder
13 6/15/2022 Optimization, Convolutional Neural Networks PDF DL Section 9
D2L Section 6, 7
ISL Section 10.3
14 6/20/2022 Convolutional Neural Networks Continued PDF DL Section 9
D2L Section 6, 7
ISL Section 10.3
15 6/22/2022 Recurrent Neural Networks PDF DL Section 10
D2L Section 8, 9
ISL Section 10.5
16 6/27/2022 k-Means and Gaussian Mixture Models PDF UML Section 22.2, 24.4
ESL Section 6.8, 8.5, 14.3
ISL Section 10.3
17 6/29/2022 Finish GMMs, Autoencoders PDF DL Section 14, 20
Building Autoencoders in Keras, by Francois Chollet
An Introduction to Variational Autoencoders, by Diederik Kingma and Max Welling
18 7/4/2022 Variational Autoencoders PDF Same as last time
19 7/6/2022 Generative Adversarial Networks PDF DL Section 20
D2L Section 17
20 7/11/2022 Robustness PDF Adversarial Robustness - Theory and Practice by Zico Kolter and Aleksander Madry
21 7/13/2022 Privacy PDF Notes
22 7/18/2022 Attention PDF The Annotated Transformer, by Sasha Rush
D2L Section 10
Attention Is All You Need
Improving Language Understanding with Unsupervised Learning
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Better Language Models and Their Implications
Language Models are Few-Shot Learners
23 7/20/2022 Ethics PDF Private traits and attributes are predictable from digital records of human behavior
Sharing learnings about our image cropping algorithm
Gender Shades
Machine Bias: There’s software used across the country to predict future criminals. And it’s biased against blacks.
24 7/25/2022 Flows PDF Video

Assignments

There will be five assignments, roughly every other week. All future dates are tentative, and may be shifted by a few days.

Late Policy

Two 48 hour extensions per student are provided. They may be each be used on one of the five homework assignments (at most one may be used per homework assignment). Email the instructor and the TA for the assignment at least 24 hours before the deadline to let us know that you're using it, and why. There will be no explicit email confirmation of an extension being granted, but at least 12 hours before the deadline, your deadline on LEARN will be adjusted appropriately. If your deadline has not been adjusted on LEARN 12 hours before the deadline despite asking 24 hours before the deadline, then (and only then) further contact the course staff to fix things. Any deviation from this protocol (including extra emails or Piazza posts inquiring whether the extension has been granted) may result in your request being (silently) ignored. There are no extensions for the final project or any other deadline. Beyond this, we do not accept any late homework submissions, unless you have a legitimate reason with formal proof (e.g. hospitalization, family urgency, etc.). Traveling, busy with other stuff, or simply forgetting to submit, are not considered legitimate.

Project (CS 680 only)

You need to conduct a research project, which could be an attempt to beat the state-of-the-art performance on an interesting dataset, or an unexpected application of machine learning algorithms to a different field, or designing a novel algorithm to address a need in machine learning, or theoretically analyzing the performance of a machine learning algorithm (new or old). A literature survey on a new hot topic in ML is also possible. You may work in pairs for this project. All parts of the project are submitted by email to the instructor.

You project should The project proposal will be due on June 15, and worth 5% of the final grade. Please concisely describe what your project is about, some related works (no need to be thorough yet), what is your execution plan, what do you expect to learn/contribute, and how are you going to evaluate your results. We expect the proposal to be 2 to 4 pages (excluding references).

An (ungraded) update is due on July 15. This should describe what you've done so far, and what is left to be done. You are expected to have measurable "work done" by this point.

The project report will be due on July 26 (though I will accept late submissions without penalty until August 12), and worth 20% of the final grade. Please summarize all your findings (empirical, algorithmic, theoretical) in a scientific report. We expect there is an introduction section, a background section, a main result section, and a conclusion section. Depending on your project, you may include an experimental section and/or discussion section. Please always give proper citations to prior work or results. Be precise and concise. We expect the report to be at most 8 pages (excluding references).

Your project report will be evaluated on its clarity, significance, rigor, presentation, and completeness.

Books and References

Readings for each lecture will be linked. Additionally, the following are excellent resources. If NumPy is new to you, you might consult these resources.

Grading

Grades for this course will be determined with the following breakdown.
CS 480: CS 680: There are five total assignments, uniformly weighted. If you submit a good faith effort (at the discretion of the instructor) to try to solve all parts of all five assignments, your lowest assignment score will be dropped. In particular, the "good faith effort" condition of this policy is to prevent students from skipping assignments.

Resources

Here are some top conferences and journals in machine learning. They may be good places to look for project ideas. I believe all of them should be open access. More generally, you should never have to pay to access an academic article, especially when you are affiliated with a university. Contact me if you run into any issues accessing any academic article and I'll write up some directions.

Policies

Academic Integrity: In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. Check the university website for more information.

Grievance: A student who believes that a decision affecting some aspect of his/her university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70, Student Petitions and Grievances, Section 4. When in doubt please be certain to contact the department's administrative assistant who will provide further assistance.

Discipline: A student is expected to know what constitutes academic integrity to avoid committing an academic offence, and to take responsibility for his/her actions. A student who is unsure whether an action constitutes an offence, or who needs help in learning how to avoid offences (e.g., plagiarism, cheating) or about "rules" for group work/collaboration should seek guidance from the course instructor, academic advisor, or the undergraduate Associate Dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline. For typical penalties check Guidelines for the Assessment of Penalties.

Appeals: A decision made or penalty imposed under Policy 70 (Student Petitions and Grievances) (other than a petition) or Policy 71 (Student Discipline) may be appealed if there is a ground. A student who believes he/she has a ground for an appeal should refer to Policy 72 (Student Appeals).

Note for Students with Disabilities: The Office for Persons with Disabilities (OPD), located in Needles Hall, Room 1132, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with the OPD at the beginning of each academic term.

Mental Health: If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support.

On-campus Resources: Off-campus Resources: Diversity: It is our intent that students from all diverse backgrounds and perspectives be well served by this course, and that students' learning needs be addressed both in and out of class. We recognize the immense value of the diversity in identities, perspectives, and contributions that students bring, and the benefit it has on our educational environment. Your suggestions are encouraged and appreciated. Please let us know ways to improve the effectiveness of the course for you personally or for other students or student groups. In particular: