Guidelines

Each group of student(s) will work on a course project, write a project report, and give a presentation about the project at the end of this course. Projects should be done either individually (literature review) or in a team of 2~3 students (research project). Students are encouraged to talk to and discuss with the instructor about the potential topics for the course project. Overall, the course project will be worth 60% of the total grade. The concrete breakdown of each part is listed as follows: The course project will be evaluated on three parts: Project Proposal 10% Due on Sep. 29th, 2 pages, must use NeurIPS LaTeX template Final Presentation 20% The last three lectures, (10 mins) Project Report 30% Due on Dec. 10th, 9 pages + unlimited reference and appendix, must use NeurIPS LaTeX template A course project could be either in the form of a literature review or a research project. In order to balance the per-student workload, literature review could only be done individually, and a research project should be finished in a team of 2~3 students. For both types of course projects, you are free to work on any topics that are broadly related to transfer learning, e.g., recent advances in meta-learning / multitask learning, out-of-domain generalization, distributionally robust learning, domain adaptation / generalization etc.

Project Proposal

Proposals should be roughly two pages long, and should at least contain the following information: Project title, type of project (literature review or research project?), and team members; Overview: Give a concise description on the research topic and problem that you are going to work on, or state the papers that you plan to survey. Describe the desired outcome and goals. Explain your motivation and provide a list appropriate references; Schedule: List a rough timeline of the project;

Final Presentation

All project teams will present their work at the end of the semester. Each team should prepare a 10-min talk (8-min presentation + 2-min Q&A). Each team needs to send the slides to the instructor at least one week before the final presentation. The project presentation will be graded based on the following criterion: Clarity of the presentation; Background and motivation of the work; Introduction to the theory / algorithm / method (if applicable) used in the project; Detailed discussion on the experimental setup, i.e., the datasets, evaluation metrics, and results (if applicable); Answers to the potential questions from the audience.

The list of slides for the project presentations:

• How To Compress Your BERT: Compound Compression during pre-training and Fine Tunning
• Super Subset Selection (S3)
• An Information-Theoretic Approach to Modality Selection
• Domain adaptation: Learning bounds and algorithms
• Invariant Risk Minimization
• Correcting Sample Selection Bias by Unlabeled Data
• Impossibility Theorems for Domain Adaptation
• Robust Supervised Learning
• Joint Distribution Optimal Transportation for Domain Adaptation
• The Risks of Invariant Risk Minimization
• Domain Generalization via Invariant Feature Representation
• Invariant Risk Minimization Games
• Generalizing from Several Related Classification Tasks to a New Unlabeled Sample
• In Search of Lost Domain Generalization
• A Unified View of Label Shift Estimation
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation
• A Convex Formulation for Learning Task Relationships in Multi-Task Learning
• Multi-Task Feature Learning
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-Task Model Fusion
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Multi-Task Learning as Multi-Objective Optimization
• Pareto Multi-Task Learning
• Bayesian Multitask Learning with Latent Hierarchies
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Provable Meta-Learning of Linear Representations
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• On First-Order Meta-Learning Algorithms
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Meta-Learning with Implicit Gradients

Potential Topics

The following list of papers serve as a starting point for project ideas to choose from. You are encouraged to read some of the papers and the reference therein. Of course, you are more than welcome to discuss other project ideas with the instructor as well.

• Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
• LEEP: A New Measure to Evaluate Transferability of Learned Representations
• mixup: Beyond Empirical Risk Minimization
• Domain adaptation: Learning bounds and algorithms
• Invariant Risk Minimization
• Correcting Sample Selection Bias by Unlabeled Data
• Impossibility Theorems for Domain Adaptation
• Robust Supervised Learning
• Joint Distribution Optimal Transportation for Domain Adaptation
• The Risks of Invariant Risk Minimization
• Domain Generalization via Invariant Feature Representation
• Invariant Risk Minimization Games
• Generalizing from Several Related Classification Tasks to a New Unlabeled Sample
• In Search of Lost Domain Generalization
• A Unified View of Label Shift Estimation
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation
• A Convex Formulation for Learning Task Relationships in Multi-Task Learning
• Multi-Task Feature Learning
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-Task Model Fusion
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Multi-Task Learning as Multi-Objective Optimization
• Pareto Multi-Task Learning
• Bayesian Multitask Learning with Latent Hierarchies
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Provable Meta-Learning of Linear Representations
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• On First-Order Meta-Learning Algorithms
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Meta-Learning with Implicit Gradients