├── README.md ├── blocks ├── __init__.py ├── feedforward_block.py └── lowrank_bilinear_block.py ├── datasets ├── __init__.py ├── data_loader.py └── mimic_dataset.py ├── evaluation └── evaler.py ├── experiments ├── V1 │ └── config.yml └── V1_rl │ └── config.yml ├── images └── framework.jpg ├── layers ├── __init__.py ├── attention.py ├── basic_att.py ├── low_rank.py ├── positional_encoding.py └── sc_att.py ├── lib ├── config.py └── utils.py ├── losses ├── __init__.py ├── cross_entropy.py ├── focalloss.py ├── label_smoothing.py └── reward_criterion.py ├── lr_scheduler ├── __init__.py ├── multi_step_lr.py └── noam_lr.py ├── main.py ├── models ├── __init__.py ├── att_basic_model.py ├── basic_model.py ├── mutimodel_beam.py └── xtransformer.py ├── optimizer ├── __init__.py ├── optimizer.py └── radam.py ├── requirements.txt ├── samplers ├── __init__.py └── distributed.py ├── scorer ├── __init__.py └── scorer.py └── tools ├── CLIP ├── CLIP.png ├── LICENSE ├── MANIFEST.in ├── README.md ├── clip │ ├── __init__.py │ ├── bpe_simple_vocab_16e6.txt.gz │ ├── clip.py │ ├── model.py │ └── simple_tokenizer.py ├── data │ ├── country211.md │ ├── prompts.md │ ├── rendered-sst2.md │ └── yfcc100m.md ├── hubconf.py ├── model-card.md ├── notebooks │ ├── Interacting_with_CLIP.ipynb │ └── Prompt_Engineering_for_ImageNet.ipynb ├── requirements.txt ├── setup.py └── tests │ └── test_consistency.py ├── build_vocab.py ├── cider_cache.py └── extract_clip_feature.py /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/README.md -------------------------------------------------------------------------------- /blocks/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/blocks/__init__.py -------------------------------------------------------------------------------- /blocks/feedforward_block.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/blocks/feedforward_block.py -------------------------------------------------------------------------------- /blocks/lowrank_bilinear_block.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/blocks/lowrank_bilinear_block.py -------------------------------------------------------------------------------- /datasets/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /datasets/data_loader.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/datasets/data_loader.py -------------------------------------------------------------------------------- /datasets/mimic_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/datasets/mimic_dataset.py -------------------------------------------------------------------------------- /evaluation/evaler.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/evaluation/evaler.py -------------------------------------------------------------------------------- /experiments/V1/config.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/experiments/V1/config.yml -------------------------------------------------------------------------------- /experiments/V1_rl/config.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/experiments/V1_rl/config.yml -------------------------------------------------------------------------------- /images/framework.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/images/framework.jpg -------------------------------------------------------------------------------- /layers/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/layers/__init__.py -------------------------------------------------------------------------------- /layers/attention.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/layers/attention.py -------------------------------------------------------------------------------- /layers/basic_att.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/layers/basic_att.py -------------------------------------------------------------------------------- /layers/low_rank.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/layers/low_rank.py -------------------------------------------------------------------------------- /layers/positional_encoding.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/layers/positional_encoding.py -------------------------------------------------------------------------------- /layers/sc_att.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/layers/sc_att.py -------------------------------------------------------------------------------- /lib/config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/lib/config.py -------------------------------------------------------------------------------- /lib/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/lib/utils.py -------------------------------------------------------------------------------- /losses/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/losses/__init__.py -------------------------------------------------------------------------------- /losses/cross_entropy.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/losses/cross_entropy.py -------------------------------------------------------------------------------- /losses/focalloss.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/losses/focalloss.py -------------------------------------------------------------------------------- /losses/label_smoothing.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/losses/label_smoothing.py -------------------------------------------------------------------------------- /losses/reward_criterion.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/losses/reward_criterion.py -------------------------------------------------------------------------------- /lr_scheduler/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/lr_scheduler/__init__.py -------------------------------------------------------------------------------- /lr_scheduler/multi_step_lr.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/lr_scheduler/multi_step_lr.py -------------------------------------------------------------------------------- /lr_scheduler/noam_lr.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/lr_scheduler/noam_lr.py -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/main.py -------------------------------------------------------------------------------- /models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/models/__init__.py -------------------------------------------------------------------------------- /models/att_basic_model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/models/att_basic_model.py -------------------------------------------------------------------------------- /models/basic_model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/models/basic_model.py -------------------------------------------------------------------------------- /models/mutimodel_beam.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/models/mutimodel_beam.py -------------------------------------------------------------------------------- /models/xtransformer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/models/xtransformer.py -------------------------------------------------------------------------------- /optimizer/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /optimizer/optimizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/optimizer/optimizer.py -------------------------------------------------------------------------------- /optimizer/radam.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/optimizer/radam.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/requirements.txt -------------------------------------------------------------------------------- /samplers/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /samplers/distributed.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/samplers/distributed.py -------------------------------------------------------------------------------- /scorer/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /scorer/scorer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/scorer/scorer.py -------------------------------------------------------------------------------- /tools/CLIP/CLIP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/CLIP.png -------------------------------------------------------------------------------- /tools/CLIP/LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/LICENSE -------------------------------------------------------------------------------- /tools/CLIP/MANIFEST.in: -------------------------------------------------------------------------------- 1 | include clip/bpe_simple_vocab_16e6.txt.gz 2 | -------------------------------------------------------------------------------- /tools/CLIP/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/README.md -------------------------------------------------------------------------------- /tools/CLIP/clip/__init__.py: -------------------------------------------------------------------------------- 1 | from .clip import * 2 | -------------------------------------------------------------------------------- /tools/CLIP/clip/bpe_simple_vocab_16e6.txt.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/clip/bpe_simple_vocab_16e6.txt.gz -------------------------------------------------------------------------------- /tools/CLIP/clip/clip.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/clip/clip.py -------------------------------------------------------------------------------- /tools/CLIP/clip/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/clip/model.py -------------------------------------------------------------------------------- /tools/CLIP/clip/simple_tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/clip/simple_tokenizer.py -------------------------------------------------------------------------------- /tools/CLIP/data/country211.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/data/country211.md -------------------------------------------------------------------------------- /tools/CLIP/data/prompts.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/data/prompts.md -------------------------------------------------------------------------------- /tools/CLIP/data/rendered-sst2.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/data/rendered-sst2.md -------------------------------------------------------------------------------- /tools/CLIP/data/yfcc100m.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/data/yfcc100m.md -------------------------------------------------------------------------------- /tools/CLIP/hubconf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/hubconf.py -------------------------------------------------------------------------------- /tools/CLIP/model-card.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/model-card.md -------------------------------------------------------------------------------- /tools/CLIP/notebooks/Interacting_with_CLIP.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/notebooks/Interacting_with_CLIP.ipynb -------------------------------------------------------------------------------- /tools/CLIP/notebooks/Prompt_Engineering_for_ImageNet.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/notebooks/Prompt_Engineering_for_ImageNet.ipynb -------------------------------------------------------------------------------- /tools/CLIP/requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/requirements.txt -------------------------------------------------------------------------------- /tools/CLIP/setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/setup.py -------------------------------------------------------------------------------- /tools/CLIP/tests/test_consistency.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/CLIP/tests/test_consistency.py -------------------------------------------------------------------------------- /tools/build_vocab.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/build_vocab.py -------------------------------------------------------------------------------- /tools/cider_cache.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/cider_cache.py -------------------------------------------------------------------------------- /tools/extract_clip_feature.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wang-zhanyu/MSAT/HEAD/tools/extract_clip_feature.py --------------------------------------------------------------------------------