CS 598: Transfer Learning (Fall 2021)


Instructor: Han Zhao
Email: hanzhao [AT] illinois (DOT) edu
Office: 3320 Siebel Center for Computer Science
Office Hours: F 3:30 pm to 4:30 pm
Teaching Assistant: Ruicheng Xian
Office: 3405 Siebel Center for Computer Science
Office Hours: Th 3:00 pm to 4:00 pm
Time: WF 2:00 pm - 3:15 pm
Location: 0216 Siebel Center for Computer Science
Discussion Group: Piazza

Course Overview

While modern deep learning has achieved remarkable success in supervised learning, e.g., image classification, speech recognition, machine translation, and game playing, this success crucially hinges on the assumption that the training distribution (approximately) matches the test distribution. However in real-world applications such an assumption often does not hold. This course will cover topics related to machine learning under the scenario where the training and test distributions are related, but not the same. We will study how to quantify the relatedness between distributions, tasks, and how the structure between them could be used to facilitate more efficient and effective learning. In particular, this course will cover the following topics, often from a representation learning perspective: domain adaptation/generalization, multitask learning, and meta-learning.

Schedule (Tentative)

We may update the following tentative schedule as the course progresses.
Week Date Lecture Slides Note
Part I: Lectures
1 08/25 Course Overview & Introduction slides overview, Bayes error
08/27 Supervised Learning: Binary Classification (iid) slides error decomposition, Perceptron
2 09/01 Generalization (I) slides PAC, generalization error
09/03 Generalization (II) slides, whiteboard covering number, sample complexity
3 09/08 Domain Adaptation / Generalization (Algorithm I) slides adversarial training, minimax game
09/10 Domain Adaptation / Generalization (Theory I) slides statistical distances, distinguishing game
4 09/15 Domain Adaptation / Generalization (Algorithm II) slides adaptation upper bound, joint error lower bound
09/17 Domain Adaptation / Generalization (Theory II) slides, whiteboard data processing inequality, JS divergence
5 09/22 Invariant Causal Predictors slides, whiteboard
09/24 Invariant Risk Minimization slides, whiteboard
6 09/29 Unifying Invariant Representations and Predictors slides, whiteboard Project proposal due
10/01 Multitask Representation Learning slides
7 10/06 Meta-Learning slides MTL generalization bound, MAML
10/08 Meta-Learning (Guest Lecture) slides Fast variants of Meta-Learning algorithms
Part II: Paper Presentations
8 10/13 "A Unified View of Label Shift Estimation" by Daniel Campos slides
10/15 Geometry of High-dimensional Data slides, whiteboard Random projection, JL-Lemma
9 10/20 Johnson-Lindenstrauss Lemma slides
10/22 "Group Distributionally Robust Optimization" by Haoxiang Wang, Lang Yin, and Haozhe Si slides
10 10/27 "Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks" by Zhen Zhu, Nengyu Wang slides
10/29 "Meta-Learning with Implicit Gradients" by Yunyi Zhang, Jiaxin Huang; "Transfer Learning based Mixture Models for Anomaly Detection" by Jay Shenoy, Nathanael Assefa slides_1, slides_2
11 11/03 "What is being transferred in transfer learning?" by Xiang Li, Mingyuan Wu; "On the Theory of Transfer Learning: the Importance of Task Diversity" by Seiyun Shin, Supawit Chockchowwat; "Multi-Task Learning as Multi-Objective Optimization" by Zhenbang Wu slides_1, slides_2, slides_3
11/05 "Meta Pseudo Labels" by Anirudh; "Joint Distribution Optimal Transportation for Domain Adaptation" by Enyi Jiang, Carl Norbert, Akul Goyal; "Impossibility Theorems for Domain Adaptation" by Gargi Balasubramaniam, Yifei He,Sam Cheng slides_1, slides_2, slides_3
12 11/10 "Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-task Model Fusion" by Ruijie Wang, Jianing Zhou; " Multi-Task Learning Using Uncertainty to Weight Losses for Scene Geometry and Semantics" by David Kolschowksy, Adit Bhagat slides_1, slides_2
11/12 "On First-Order Meta-Learning Algorithms" by Francis Yu, Zhenhailong Wang; "Generalizing to Unseen Domains via Adversarial Data Augmentation" by Liliang Ren, Zexing Xu; " On the Generalization Effects of Linear Transformations in Data Augmentation" by Chulin Xie, Evelyn Ma, Fan Wu slides_1, slides_2, slides_3
13 11/17 "An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization" by Nicole Chiou, Olawale Salaudeen; "Debiased Contrastive Learning" by Meilu Yuan slides_1, slides_2
11/19 "Which Tasks Should Be Learned Together in Multi-task Learning?" by Jiayi Luo, Junze Wu, Bharat Bojja; "Action Embedding for Transfer Reinforcement Learning" by Emma Yu, Stefan Ivanovic, Bochao Li slides_1, slides_2
Fall Break (no class)
15 12/01 Final Project Presentation (part 1)
12/03 Final Project Presentation (part 2)
16 12/08 Final Project Presentation (part 3)
12/10 Final project report due

Course Format and Grading

This course will be a mix of lectures and student presentations/discussions. For the first half of the course, we will cover both the algorithms and theory of domain adaptation/generalization, multitask learning and meta-learning. The second half of the course will mostly be seminar-style paper presentations/discussions. In this part, students will be responsible for paper readings and in-class presentation. Students will also form into groups of 1-3 to complete a course project. More details on the course project will come soon. Note: this course does not have any tests or exams.
Your course grade will be based on three components:
Paper Presentation 30% Presenting one research paper (related to transfer learning) in class
Course Project 60% This includes a break down of 10% for project proposal, 20% for final presentation and 30% for the project report
Participation 10% This includes both in-class discussion and Piazza participation (sufficient to participate in one of them)

Prerequisites

Probability and statistics, linear algebra, calculus and basic information theory. Machine Learning (CS 446) or an equivalent introductory course on machine learning is highly recommended.

Academic Integrity

Jeff Erickson maintains a nice page on this. Please also check the University of Illinois at Urbana-Champaign Student Code, Article 1 Part 4 for more details on what constitutes an academic integrity violation. A violation of academic integrity will lead to a failing grade for the corresponding assignment, and this will be recorded at the university level as well.

FAQs

Q: I am interested in taking this course, but the current enrollment is full, could you add me to the waiting list?
A: Sorry, but the department of computer science does not do waiting list. You need to wait for a seat to open. You can go into course explorer and set it to ping you when a seat opens. For more questions about course registration, please contact Heather Mihaly for more information.

Q: Will this course be video recorded? How could I have access to the recorded videos?
A: Yes, all the lectures will be video recorded, except for the second part on paper discussion (due to privacy concerns). All the registered students of this course will be added to a channel on Mediaspace, where we will post the videos.

Q: I am an undergrad student who is interested in this course, could I register?
A: For sure, just go through the Undergrad Application for Access to CS Grad Sections petition and make a request. As long as you have satisfied the prerequisites of this course, I will go ahead and approve.

Q: My schedule is quite tight this semester. Could I audit this course instead of taking it for credit?
A: Yes. Just go through the usual process for auditing.

Covid-19 Policy

Following University policy, all students are required to engage in appropriate behavior to protect the health and safety of the community. Students are also required to follow the campus COVID-19 protocols.

Students who feel ill must not come to class. In addition, students who test positive for COVID-19 or have had an exposure that requires testing and/or quarantine must not attend class. The University will provide information to the instructor, in a manner that complies with privacy laws, about students in these latter categories. These students are judged to have excused absences for the class period and should contact the instructor via email about making up the work.

Students who fail to abide by these rules will first be asked to comply; if they refuse, they will be required to leave the classroom immediately. If a student is asked to leave the classroom, the non-compliant student will be judged to have an unexcused absence and reported to the Office for Student Conflict Resolution for disciplinary action. Accumulation of non-compliance complaints against a student may result in dismissal from the University.

All students, faculty, staff, and visitors are required to wear face coverings in classrooms and university spaces. This is in accordance with CDC guidance and University policy and expected in this class. Please refer to the University of Illinois Urbana-Champaign’s COVID-19 website for further information on face coverings. For more details about the University's Covid-19 policy, please visit the website.

Accommodations

To obtain disability-related academic adjustments and/or auxiliary aids, students with disabilities should contact the course instructor and the Disability Resources and Educational Services (DRES, 1207 S. Oak St., Champaign) as soon as possible. Please refer to http://www.disability.illinois.edu/ for more resources and details.