├── .github ├── CODEOWNERS ├── ISSUE_TEMPLATE │ ├── bug.md │ ├── feature.md │ └── todo.md ├── PULL_REQUEST_TEMPLATE.md └── workflows │ └── pull_request.yml ├── .gitignore ├── .pre-commit-config.yaml ├── LICENSE ├── README.md ├── ahn ├── __init__.py ├── train_sbatch_whitespace.sh ├── train_sbatch_zero.sh └── training │ ├── __init__.py │ ├── _decoder.py │ ├── _factory.py │ ├── _utils.py │ ├── configs │ ├── __init__.py │ ├── bloomz-1b1-train-zero.yaml │ ├── bloomz-7b1-mt-train-zero.yaml │ ├── example-230314.yaml │ ├── xglm-train-lora.yaml │ ├── xglm-train-whitespace.yaml │ └── xglm-train-zero.yaml │ ├── requirements.txt │ ├── train.py │ └── train_zero.py ├── data ├── README.md └── src │ ├── TranslateJsonl.py │ ├── cross_lingual_plus_formatting_fixes.py │ └── translation_m2m100_with_quality_check.py └── requirements.txt /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @hyunwoongko 2 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/.github/ISSUE_TEMPLATE/bug.md -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/.github/ISSUE_TEMPLATE/feature.md -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/todo.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/.github/ISSUE_TEMPLATE/todo.md -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/.github/PULL_REQUEST_TEMPLATE.md -------------------------------------------------------------------------------- /.github/workflows/pull_request.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/.github/workflows/pull_request.yml -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/.gitignore -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/.pre-commit-config.yaml -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/README.md -------------------------------------------------------------------------------- /ahn/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /ahn/train_sbatch_whitespace.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/train_sbatch_whitespace.sh -------------------------------------------------------------------------------- /ahn/train_sbatch_zero.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/train_sbatch_zero.sh -------------------------------------------------------------------------------- /ahn/training/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /ahn/training/_decoder.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/_decoder.py -------------------------------------------------------------------------------- /ahn/training/_factory.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/_factory.py -------------------------------------------------------------------------------- /ahn/training/_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/_utils.py -------------------------------------------------------------------------------- /ahn/training/configs/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /ahn/training/configs/bloomz-1b1-train-zero.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/configs/bloomz-1b1-train-zero.yaml -------------------------------------------------------------------------------- /ahn/training/configs/bloomz-7b1-mt-train-zero.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/configs/bloomz-7b1-mt-train-zero.yaml -------------------------------------------------------------------------------- /ahn/training/configs/example-230314.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/configs/example-230314.yaml -------------------------------------------------------------------------------- /ahn/training/configs/xglm-train-lora.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/configs/xglm-train-lora.yaml -------------------------------------------------------------------------------- /ahn/training/configs/xglm-train-whitespace.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/configs/xglm-train-whitespace.yaml -------------------------------------------------------------------------------- /ahn/training/configs/xglm-train-zero.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/configs/xglm-train-zero.yaml -------------------------------------------------------------------------------- /ahn/training/requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/requirements.txt -------------------------------------------------------------------------------- /ahn/training/train.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/train.py -------------------------------------------------------------------------------- /ahn/training/train_zero.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/ahn/training/train_zero.py -------------------------------------------------------------------------------- /data/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/data/README.md -------------------------------------------------------------------------------- /data/src/TranslateJsonl.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/data/src/TranslateJsonl.py -------------------------------------------------------------------------------- /data/src/cross_lingual_plus_formatting_fixes.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/data/src/cross_lingual_plus_formatting_fixes.py -------------------------------------------------------------------------------- /data/src/translation_m2m100_with_quality_check.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/data/src/translation_m2m100_with_quality_check.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LAION-AI/Anh/HEAD/requirements.txt --------------------------------------------------------------------------------