├── .gitignore ├── LICENSE ├── README.md ├── TODO.md ├── assets ├── overview.png ├── pie.png └── vt_type.png ├── experiments ├── additional_for_social_bias.sentiment.sh ├── experiment.qa_qg.sh ├── experiment.sentiment.sh ├── experiment.xnli.sh ├── finetune_sentiment.py ├── finetune_xnli.py ├── misc │ ├── model_list.py │ ├── plot_js.py │ ├── plot_tables.py │ ├── result.xlm_r.py │ └── result.xlm_v.py ├── plot_line.py ├── result.qa_qg.py ├── result.sentiment_xnli.py └── social_bias │ ├── __init__.py │ ├── preprocess.py │ └── sentiment_bias.py ├── hf_operations ├── check_models.py ├── delete_models.py └── move_models.py ├── model_card.md ├── setup.cfg ├── setup.py ├── tests ├── test_mbart.py ├── test_mt5.py └── test_xlm.py └── vocabtrimmer ├── __init__.py ├── base_trimmer.py ├── cl ├── __init__.py └── trimming.py ├── util.py └── vocab_miner.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/README.md -------------------------------------------------------------------------------- /TODO.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/TODO.md -------------------------------------------------------------------------------- /assets/overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/assets/overview.png -------------------------------------------------------------------------------- /assets/pie.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/assets/pie.png -------------------------------------------------------------------------------- /assets/vt_type.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/assets/vt_type.png -------------------------------------------------------------------------------- /experiments/additional_for_social_bias.sentiment.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/additional_for_social_bias.sentiment.sh -------------------------------------------------------------------------------- /experiments/experiment.qa_qg.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/experiment.qa_qg.sh -------------------------------------------------------------------------------- /experiments/experiment.sentiment.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/experiment.sentiment.sh -------------------------------------------------------------------------------- /experiments/experiment.xnli.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/experiment.xnli.sh -------------------------------------------------------------------------------- /experiments/finetune_sentiment.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/finetune_sentiment.py -------------------------------------------------------------------------------- /experiments/finetune_xnli.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/finetune_xnli.py -------------------------------------------------------------------------------- /experiments/misc/model_list.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/misc/model_list.py -------------------------------------------------------------------------------- /experiments/misc/plot_js.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/misc/plot_js.py -------------------------------------------------------------------------------- /experiments/misc/plot_tables.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/misc/plot_tables.py -------------------------------------------------------------------------------- /experiments/misc/result.xlm_r.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/misc/result.xlm_r.py -------------------------------------------------------------------------------- /experiments/misc/result.xlm_v.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/misc/result.xlm_v.py -------------------------------------------------------------------------------- /experiments/plot_line.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/plot_line.py -------------------------------------------------------------------------------- /experiments/result.qa_qg.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/result.qa_qg.py -------------------------------------------------------------------------------- /experiments/result.sentiment_xnli.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/result.sentiment_xnli.py -------------------------------------------------------------------------------- /experiments/social_bias/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /experiments/social_bias/preprocess.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/social_bias/preprocess.py -------------------------------------------------------------------------------- /experiments/social_bias/sentiment_bias.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/experiments/social_bias/sentiment_bias.py -------------------------------------------------------------------------------- /hf_operations/check_models.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/hf_operations/check_models.py -------------------------------------------------------------------------------- /hf_operations/delete_models.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/hf_operations/delete_models.py -------------------------------------------------------------------------------- /hf_operations/move_models.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/hf_operations/move_models.py -------------------------------------------------------------------------------- /model_card.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/model_card.md -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/setup.py -------------------------------------------------------------------------------- /tests/test_mbart.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/tests/test_mbart.py -------------------------------------------------------------------------------- /tests/test_mt5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/tests/test_mt5.py -------------------------------------------------------------------------------- /tests/test_xlm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/tests/test_xlm.py -------------------------------------------------------------------------------- /vocabtrimmer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/vocabtrimmer/__init__.py -------------------------------------------------------------------------------- /vocabtrimmer/base_trimmer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/vocabtrimmer/base_trimmer.py -------------------------------------------------------------------------------- /vocabtrimmer/cl/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /vocabtrimmer/cl/trimming.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/vocabtrimmer/cl/trimming.py -------------------------------------------------------------------------------- /vocabtrimmer/util.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/vocabtrimmer/util.py -------------------------------------------------------------------------------- /vocabtrimmer/vocab_miner.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/asahi417/lm-vocab-trimmer/HEAD/vocabtrimmer/vocab_miner.py --------------------------------------------------------------------------------