| While modern deep learning has achieved remarkable success in supervised learning, e.g., image classification, speech recognition, machine translation, and game playing, this success crucially hinges on the assumption that the training distribution (approximately) matches the test distribution. However in real-world applications such an assumption often does not hold. This course will cover topics related to machine learning under the scenario where the training and test distributions are related, but not the same. We will study how to quantify the relatedness between distributions, tasks, and how the structure between them could be used to facilitate more efficient and effective learning. In particular, this course will cover the following topics, often from a representation learning perspective: domain adaptation/generalization, multitask learning, and meta-learning. |
| Week | Date | Lecture | Slides | Note |
|---|---|---|---|---|
| Part I: Lectures | ||||
| 1 | 08/25 | Course Overview & Introduction | slides | overview, Bayes error |
| 08/27 | Supervised Learning: Binary Classification (iid) | slides | error decomposition, Perceptron | |
| 2 | 09/01 | Generalization (I) | slides | PAC, generalization error |
| 09/03 | Generalization (II) | slides, whiteboard | covering number, sample complexity | |
| 3 | 09/08 | Domain Adaptation / Generalization (Algorithm I) | slides | adversarial training, minimax game |
| 09/10 | Domain Adaptation / Generalization (Theory I) | slides | statistical distances, distinguishing game | |
| 4 | 09/15 | Domain Adaptation / Generalization (Algorithm II) | slides | adaptation upper bound, joint error lower bound |
| 09/17 | Domain Adaptation / Generalization (Theory II) | slides, whiteboard | data processing inequality, JS divergence | |
| 5 | 09/22 | Invariant Causal Predictors | slides, whiteboard | |
| 09/24 | Invariant Risk Minimization | slides, whiteboard | ||
| 6 | 09/29 | Unifying Invariant Representations and Predictors | slides, whiteboard | Project proposal due |
| 10/01 | Multitask Representation Learning | slides | ||
| 7 | 10/06 | Meta-Learning | slides | MTL generalization bound, MAML |
| 10/08 | Meta-Learning (Guest Lecture) | slides | Fast variants of Meta-Learning algorithms | |
| Part II: Paper Presentations | ||||
| 8 | 10/13 | "A Unified View of Label Shift Estimation" by Daniel Campos | slides | |
| 10/15 | Geometry of High-dimensional Data | slides, whiteboard | Random projection, JL-Lemma | |
| 9 | 10/20 | Johnson-Lindenstrauss Lemma | slides | |
| 10/22 | "Group Distributionally Robust Optimization" by Haoxiang Wang, Lang Yin, and Haozhe Si | slides | ||
| 10 | 10/27 | "Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks" by Zhen Zhu, Nengyu Wang | slides | |
| 10/29 | "Meta-Learning with Implicit Gradients" by Yunyi Zhang, Jiaxin Huang; "Transfer Learning based Mixture Models for Anomaly Detection" by Jay Shenoy, Nathanael Assefa | slides_1, slides_2 | ||
| 11 | 11/03 | "What is being transferred in transfer learning?" by Xiang Li, Mingyuan Wu; "On the Theory of Transfer Learning: the Importance of Task Diversity" by Seiyun Shin, Supawit Chockchowwat; "Multi-Task Learning as Multi-Objective Optimization" by Zhenbang Wu | slides_1, slides_2, slides_3 | |
| 11/05 | "Meta Pseudo Labels" by Anirudh; "Joint Distribution Optimal Transportation for Domain Adaptation" by Enyi Jiang, Carl Norbert, Akul Goyal; "Impossibility Theorems for Domain Adaptation" by Gargi Balasubramaniam, Yifei He,Sam Cheng | slides_1, slides_2, slides_3 | ||
| 12 | 11/10 | "Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-task Model Fusion" by Ruijie Wang, Jianing Zhou; " Multi-Task Learning Using Uncertainty to Weight Losses for Scene Geometry and Semantics" by David Kolschowksy, Adit Bhagat | slides_1, slides_2 | |
| 11/12 | "On First-Order Meta-Learning Algorithms" by Francis Yu, Zhenhailong Wang; "Generalizing to Unseen Domains via Adversarial Data Augmentation" by Liliang Ren, Zexing Xu; " On the Generalization Effects of Linear Transformations in Data Augmentation" by Chulin Xie, Evelyn Ma, Fan Wu | slides_1, slides_2, slides_3 | ||
| 13 | 11/17 | "An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization" by Nicole Chiou, Olawale Salaudeen; "Debiased Contrastive Learning" by Meilu Yuan | slides_1, slides_2 | |
| 11/19 | "Which Tasks Should Be Learned Together in Multi-task Learning?" by Jiayi Luo, Junze Wu, Bharat Bojja; "Action Embedding for Transfer Reinforcement Learning" by Emma Yu, Stefan Ivanovic, Bochao Li | slides_1, slides_2 | ||
| Fall Break (no class) | ||||
| 15 | 12/01 | Final Project Presentation (part 1) | ||
| 12/03 | Final Project Presentation (part 2) | |||
| 16 | 12/08 | Final Project Presentation (part 3) | ||
| 12/10 | Final project report due | |||
| This course will be a mix of lectures and student presentations/discussions. For the first half of the course, we will cover both the algorithms and theory of domain adaptation/generalization, multitask learning and meta-learning. The second half of the course will mostly be seminar-style paper presentations/discussions. In this part, students will be responsible for paper readings and in-class presentation. Students will also form into groups of 1-3 to complete a course project. More details on the course project will come soon. Note: this course does not have any tests or exams. |
| Paper Presentation | 30% | Presenting one research paper (related to transfer learning) in class |
| Course Project | 60% | This includes a break down of 10% for project proposal, 20% for final presentation and 30% for the project report |
| Participation | 10% | This includes both in-class discussion and Piazza participation (sufficient to participate in one of them) |
| Probability and statistics, linear algebra, calculus and basic information theory. Machine Learning (CS 446) or an equivalent introductory course on machine learning is highly recommended. |
| Jeff Erickson maintains a nice page on this. Please also check the University of Illinois at Urbana-Champaign Student Code, Article 1 Part 4 for more details on what constitutes an academic integrity violation. A violation of academic integrity will lead to a failing grade for the corresponding assignment, and this will be recorded at the university level as well. |
|
Q: I am interested in taking this course, but the current enrollment is full, could you add me to
the waiting list?
A: Sorry, but the department of computer science does not do waiting list. You need to wait for a seat to open. You can go into course explorer and set it to ping you when a seat opens. For more questions about course registration, please contact Heather Mihaly for more information. Q: Will this course be video recorded? How could I have access to the recorded videos? A: Yes, all the lectures will be video recorded, except for the second part on paper discussion (due to privacy concerns). All the registered students of this course will be added to a channel on Mediaspace, where we will post the videos. Q: I am an undergrad student who is interested in this course, could I register? A: For sure, just go through the Undergrad Application for Access to CS Grad Sections petition and make a request. As long as you have satisfied the prerequisites of this course, I will go ahead and approve. Q: My schedule is quite tight this semester. Could I audit this course instead of taking it for credit? A: Yes. Just go through the usual process for auditing. |
| Following University policy, all students are required to engage in appropriate behavior to protect the health and safety of the community. Students are also required to follow the campus COVID-19 protocols. Students who feel ill must not come to class. In addition, students who test positive for COVID-19 or have had an exposure that requires testing and/or quarantine must not attend class. The University will provide information to the instructor, in a manner that complies with privacy laws, about students in these latter categories. These students are judged to have excused absences for the class period and should contact the instructor via email about making up the work. Students who fail to abide by these rules will first be asked to comply; if they refuse, they will be required to leave the classroom immediately. If a student is asked to leave the classroom, the non-compliant student will be judged to have an unexcused absence and reported to the Office for Student Conflict Resolution for disciplinary action. Accumulation of non-compliance complaints against a student may result in dismissal from the University. All students, faculty, staff, and visitors are required to wear face coverings in classrooms and university spaces. This is in accordance with CDC guidance and University policy and expected in this class. Please refer to the University of Illinois Urbana-Champaign’s COVID-19 website for further information on face coverings. For more details about the University's Covid-19 policy, please visit the website. |
| To obtain disability-related academic adjustments and/or auxiliary aids, students with disabilities should contact the course instructor and the Disability Resources and Educational Services (DRES, 1207 S. Oak St., Champaign) as soon as possible. Please refer to http://www.disability.illinois.edu/ for more resources and details. |