Transfer Learning and Domain Adaptation

“Domain” and “Task”

• Domain relates to the feature space of a specific dataset and the marginal probability distribution of features
• Task relates to the label space of a dataset and an objective predictive function

Transfer Learning

• goal is to transfer the knowledge learned from the Task(a) on Domain A to the Task(b) on Domain B
• common to update the last several layers of the pre-trained network towards a new given task with a new label space (the supporting assumption is that the early layers in the network extract low-level features which are universal for vision tasks)
  • Target Domain에 label이 없다면 학습이 불가능

Domain Adaptation

• particular type of transfer learning
• goal is to transfer knowledge from S Domain to T Domain to perform a specific Task on T (Domain Shift)
  
  • Task is shared
  • the marginal distributions are different between the source(S) Domain and target(T) Domain

Types of Domain Adaptation

1. Supervised DA
   
   • small number labeled data in the target domain are available
   • pre-train the model with source dataset and fine-tune with target dataset (Transfer learning)
2. Semi-supervised DA
   
   • small number of labeled data as well as redundant unlabeled in the target domain are available
3. Unsupervised DA / Self-supervised DA
   
   • only unlabeled target data are available
   • 대표논문. [DANN] Domain-Adversarial Training of Neural Networks
   • self-supervised a branch of unsupervised learning: Obtain labels form the data itself
4. Source-free DA
   • only a source pre-trained model and an unlabeled target domain dataset are available
   • 대표논문. [DistillAdapt] Source-Free Activate Visual Domain Adaptation

 

Reference

https://lhw0772.medium.com/study-da-domain-adaptation-%EC%95%8C%EC%95%84%EB%B3%B4%EA%B8%B0-%EA%B8%B0%EB%B3%B8%ED%8E%B8-4af4ab63f871

Guan, Hao, and Mingxia Liu. "Domain adaptation for medical image analysis: a survey." IEEE Transactions on Biomedical Engineering 69.3 (2021)