本文共 1651 字，大约阅读时间需要 5 分钟。
一、
- SimCLR seems to benefit from deep models and long training more than SL
- how we handle the weight initialization may play a big role in the final features
- We also should probably move away from comparing to supervised learning features as they may not be the golden standard (e.g., mid-range attributes)
- Probably a combination of both discriminative and aligning principles (through multi task learning) is a plausible direction (see also Feng et al, 2019)
二、
- Three phases of self-supervised learning: Classical – Expansion – Uncurated; Uncurated is less explored.
- Self-supervision from uncurated data, i.e. no pre-defined datasets, instead: – Random YouTube videos, so not class balanced, long tailed – Daily life videos, e.g. Vlogs, babycams
- New learning schedules: • Curriculum learning • How to obtain informative (hard) samples
- Universal networks: able to ingest multiple-modalities and carry out multiple tasks
- Curated datasets still have their uses: become new evaluation benchmarks
三、
- Data is infinite. Test set is contemporary. So network always can find a way to treat test set. We need to get rid of it.
- Real-world motivation: Biological agents never see the same data twice! Repeating the same sample might encourage memorization/discourage generalization.
- Self-supervision can help adapt to new domain.
- Inspiration: one sample learning (An image specific )
- Datasets were useful but they might be hunting us and thus missing something important.
- Let's start thinking beyond fixed train/test datasets, and more about continuous data streams (which happen naturally and do not need some designs such as crop or data augmentation)----videos
转载地址：http://yfeqf.baihongyu.com/