├── .gitignore ├── signwriting_clip ├── __init__.py ├── data │ ├── __init__.py │ ├── preprocess_images_for_clip.py │ └── create_dataset_csv.py ├── create_dataset.sh └── train_model.sh ├── assets ├── distribution │ ├── CLIP.png │ └── SignWritingCLIP.png └── matches │ ├── M520x520S14c20480x484S27106505x480 │ ├── ref.png │ ├── CLIP │ │ ├── 0.png │ │ ├── 1.png │ │ ├── 2.png │ │ ├── 3.png │ │ ├── 4.png │ │ ├── 5.png │ │ ├── 6.png │ │ ├── 7.png │ │ ├── 8.png │ │ └── 9.png │ └── SignWritingCLIP │ │ ├── 0.png │ │ ├── 1.png │ │ ├── 2.png │ │ ├── 3.png │ │ ├── 4.png │ │ ├── 5.png │ │ ├── 6.png │ │ ├── 7.png │ │ ├── 8.png │ │ └── 9.png │ ├── M533x518S2ff00482x483S15a11510x487S26500508x469 │ ├── ref.png │ ├── CLIP │ │ ├── 0.png │ │ ├── 1.png │ │ ├── 2.png │ │ ├── 3.png │ │ ├── 4.png │ │ ├── 5.png │ │ ├── 6.png │ │ ├── 7.png │ │ ├── 8.png │ │ └── 9.png │ └── SignWritingCLIP │ │ ├── 0.png │ │ ├── 1.png │ │ ├── 2.png │ │ ├── 3.png │ │ ├── 4.png │ │ ├── 5.png │ │ ├── 6.png │ │ ├── 7.png │ │ ├── 8.png │ │ └── 9.png │ └── M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482 │ ├── ref.png │ ├── CLIP │ ├── 0.png │ ├── 1.png │ ├── 2.png │ ├── 3.png │ ├── 4.png │ ├── 5.png │ ├── 6.png │ ├── 7.png │ ├── 8.png │ └── 9.png │ └── SignWritingCLIP │ ├── 0.png │ ├── 1.png │ ├── 2.png │ ├── 3.png │ ├── 4.png │ ├── 5.png │ ├── 6.png │ ├── 7.png │ ├── 8.png │ └── 9.png ├── .github └── workflows │ └── lint.yaml ├── pyproject.toml └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ -------------------------------------------------------------------------------- /signwriting_clip/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /signwriting_clip/data/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /assets/distribution/CLIP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/distribution/CLIP.png -------------------------------------------------------------------------------- /assets/distribution/SignWritingCLIP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/distribution/SignWritingCLIP.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/ref.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/ref.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/0.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/1.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/2.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/3.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/4.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/5.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/6.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/7.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/8.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/CLIP/9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/CLIP/9.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/ref.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/ref.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/0.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/1.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/2.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/3.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/4.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/5.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/6.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/7.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/8.png -------------------------------------------------------------------------------- /assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M520x520S14c20480x484S27106505x480/SignWritingCLIP/9.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/0.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/1.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/2.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/3.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/4.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/5.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/6.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/7.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/8.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/CLIP/9.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/ref.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/ref.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/0.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/1.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/2.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/3.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/4.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/5.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/6.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/7.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/8.png -------------------------------------------------------------------------------- /assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M533x518S2ff00482x483S15a11510x487S26500508x469/SignWritingCLIP/9.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/0.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/1.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/2.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/3.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/4.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/5.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/6.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/7.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/8.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/CLIP/9.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/0.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/1.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/2.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/3.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/4.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/5.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/6.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/7.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/8.png -------------------------------------------------------------------------------- /assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sign-language-processing/signwriting-clip/main/assets/matches/M528x557S14c21473x531S2890a499x527S30a00482x482S33e00482x482/SignWritingCLIP/9.png -------------------------------------------------------------------------------- /.github/workflows/lint.yaml: -------------------------------------------------------------------------------- 1 | name: Lint 2 | 3 | 4 | on: 5 | push: 6 | branches: [ master, main ] 7 | pull_request: 8 | branches: [ master, main ] 9 | 10 | 11 | jobs: 12 | test: 13 | name: Lint 14 | runs-on: ubuntu-latest 15 | 16 | steps: 17 | - uses: actions/checkout@v3 18 | - uses: actions/setup-python@v4 19 | with: 20 | python-version: '3.10' 21 | 22 | - name: Install Requirements 23 | run: pip install .[dev] 24 | 25 | - name: Lint Code 26 | run: pylint signwriting_clip 27 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "signwriting-clip" 3 | description = "CLIP model for SignWriting" 4 | version = "0.0.1" 5 | authors = [ 6 | { name = "Amit Moryossef", email = "amitmoryossef@gmail.com" } 7 | ] 8 | readme = "README.md" 9 | dependencies = [ 10 | "tqdm", 11 | "transformers", 12 | "Pillow", 13 | "torchvision", 14 | "numpy", 15 | "signwriting @ git+https://github.com/sign-language-processing/signwriting" 16 | ] 17 | 18 | [project.optional-dependencies] 19 | dev = [ 20 | "pytest", 21 | "pylint" 22 | ] 23 | 24 | [tool.yapf] 25 | based_on_style = "google" 26 | column_limit = 120 27 | 28 | [tool.pylint] 29 | max-line-length = 120 30 | disable = [ 31 | "C0114", # Missing module docstring 32 | "C0116", # Missing function or method docstring 33 | ] 34 | 35 | [tool.setuptools] 36 | packages = [ 37 | "signwriting_clip" 38 | ] 39 | 40 | [tool.pytest.ini_options] 41 | addopts = "-v" 42 | testpaths = ["signwriting_clip"] 43 | -------------------------------------------------------------------------------- /signwriting_clip/create_dataset.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | #SBATCH --job-name=create-dataset 4 | #SBATCH --time=168:00:00 5 | #SBATCH --cpus-per-task=8 6 | #SBATCH --mem=32GB 7 | #SBATCH --output=dataset.out 8 | 9 | #SBATCH --ntasks=1 10 | 11 | set -e # exit on error 12 | set -x # echo commands 13 | 14 | module load anaconda3 15 | source activate clip 16 | 17 | # Download the SignBank repository if not exists 18 | SIGNBANK_DIR="/home/amoryo/sign-language/signbank-annotation/signbank-plus" 19 | [ ! -d "$SIGNBANK_DIR" ] && \ 20 | git clone https://github.com/sign-language-processing/signbank-plus.git "$SIGNBANK_DIR" 21 | 22 | # Process data for machine translation if not exists 23 | [ ! -d "$SIGNBANK_DIR/data/parallel/cleaned" ] && \ 24 | python "$SIGNBANK_DIR/signbank_plus/prep_nmt.py" 25 | 26 | # Create images from all written signs 27 | PROCESSED_DATA_DIR="/data/$(whoami)/SignWritingImages" 28 | mkdir -p $PROCESSED_DATA_DIR 29 | 30 | # For CLIP, images should be image_size = 224. We should pad small images to 224x224, and remove larger images 31 | pip install Pillow 32 | 33 | # This takes about 30 minutes for 240k images 34 | for name in "raw" "sign2mint" "signsuisse" 35 | do 36 | python data/preprocess_images_for_clip.py \ 37 | --input-csv="$SIGNBANK_DIR/data/$name.csv" \ 38 | --output-path="$PROCESSED_DATA_DIR" 39 | done 40 | 41 | 42 | # Convert to huggingface dataset 43 | HF_DATASET_CSV="$PROCESSED_DATA_DIR/sign-writing-clip.csv" 44 | 45 | [ ! -f "$HF_DATASET_CSV" ] && \ 46 | python data/create_dataset_csv.py \ 47 | --images-directory="$PROCESSED_DATA_DIR" \ 48 | --csv="$SIGNBANK_DIR/data/parallel/cleaned/train.csv" \ 49 | --output-path="$HF_DATASET_CSV" 50 | -------------------------------------------------------------------------------- /signwriting_clip/train_model.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | #SBATCH --job-name=train-clip-hf 4 | #SBATCH --time=168:00:00 5 | #SBATCH --cpus-per-task=8 6 | #SBATCH --mem=32GB 7 | #SBATCH --output=train.out 8 | 9 | #SBATCH --ntasks=1 10 | #SBATCH --gres gpu:1 11 | #SBATCH --constraint=GPUMEM80GB|GPUMEM32GB 12 | 13 | set -e # exit on error 14 | set -x # echo commands 15 | 16 | module load gpu 17 | module load cuda 18 | 19 | module load anaconda3 20 | source activate clip 21 | 22 | # Download transformers repository if not exists 23 | [ ! -d "transformers" ] && \ 24 | git clone https://github.com/huggingface/transformers.git 25 | 26 | # Install transformers from source 27 | #pip install ./transformers accelerate huggingface-hub wandb 28 | #pip install -r transformers/examples/pytorch/contrastive-image-text/requirements.txt 29 | 30 | huggingface-cli login --token $HUGGINGFACE_TOKEN 31 | 32 | OUTPUT_DIR="/scratch/$(whoami)/models/signwriting-clip/clip" 33 | mkdir -p $OUTPUT_DIR 34 | 35 | # Remove dataset filtering from `run_clip.py` 36 | TRAIN_SCRIPT="transformers/examples/pytorch/contrastive-image-text/run_clip.py" 37 | sed -i '/train_dataset\.filter(/,+2 s/^/#/' "$TRAIN_SCRIPT" 38 | 39 | export WANDB_PROJECT="signwriting-clip" 40 | 41 | # Train the model 42 | python "$TRAIN_SCRIPT" \ 43 | --output_dir "$OUTPUT_DIR" \ 44 | --model_name_or_path "openai/clip-vit-base-patch32" \ 45 | --max_seq_length 77 \ 46 | --freeze_text_model \ 47 | --train_file "/data/$(whoami)/SignWritingImages/sign-writing-clip.csv" \ 48 | --image_column "image_path" \ 49 | --caption_column "caption" \ 50 | --remove_unused_columns=False \ 51 | --do_train \ 52 | --per_device_train_batch_size="64" \ 53 | --per_device_eval_batch_size="64" \ 54 | --num_train_epochs="200" \ 55 | --learning_rate="5e-5" --warmup_steps="0" --weight_decay 0.1 \ 56 | --overwrite_output_dir \ 57 | --report_to "wandb" \ 58 | --push_to_hub \ 59 | --hub_model_id "sign/signwriting-clip" 60 | 61 | 62 | # sbatch train.sh 63 | # srun --pty -n 1 -c 1 --time=01:00:00 --gres=gpu:1 --constraint=GPUMEM80GB --mem=32G bash -l 64 | # srun --pty -n 1 -c 1 --time=01:00:00 --mem=32G bash -l -------------------------------------------------------------------------------- /signwriting_clip/data/preprocess_images_for_clip.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import csv 3 | import hashlib 4 | from csv import DictReader 5 | 6 | from pathlib import Path 7 | from PIL import Image 8 | from tqdm import tqdm 9 | 10 | from signwriting.visualizer.visualize import signwriting_to_image 11 | 12 | 13 | def process_image_clip(fsw: str, dst_path: Path): 14 | # Create a 224x224 RGB image with a white background 15 | size = 224 16 | img = signwriting_to_image(fsw) # this is RGBA 17 | 18 | if img.width > size or img.height > size: 19 | return 20 | 21 | new_img = Image.new('RGB', (size, size), (255, 255, 255)) 22 | 23 | # Calculate the position to paste the image so that it's centered 24 | x_offset = (size - img.width) // 2 25 | y_offset = (size - img.height) // 2 26 | offset = (x_offset, y_offset) 27 | 28 | # Paste the output_im image onto the white background 29 | new_img.paste(img, offset, img) 30 | 31 | new_img.save(dst_path) 32 | 33 | 34 | def main(): 35 | parser = argparse.ArgumentParser() 36 | parser.add_argument("--input-csv", type=str, required=True) 37 | parser.add_argument("--output-path", type=str, required=True) 38 | args = parser.parse_args() 39 | 40 | input_csv = Path(args.input_csv) 41 | output_path = Path(args.output_path) 42 | 43 | output_path.mkdir(parents=True, exist_ok=True) 44 | existing_files = set(map(str, output_path.glob('*.png'))) 45 | print(f"Found {len(existing_files)} existing files.") 46 | 47 | with open(input_csv, 'r', encoding="utf-8") as csv_f: 48 | csv.field_size_limit(2 ** 20) # Increase limit to 1MB (2^20 characters) 49 | reader = DictReader(csv_f) 50 | unique_fsw = set(row["sign_writing"] for row in reader) 51 | 52 | for fsw in tqdm(unique_fsw): 53 | fsw_md5 = hashlib.md5(fsw.encode('utf-8')).hexdigest() 54 | output_file = output_path / f"{fsw_md5}.png" 55 | if str(output_file) not in existing_files: 56 | try: 57 | process_image_clip(fsw, output_file) 58 | except Exception as exception: # pylint: disable=broad-except 59 | print(f"Failed to process {fsw}: {exception}") 60 | 61 | 62 | if __name__ == "__main__": 63 | main() 64 | -------------------------------------------------------------------------------- /signwriting_clip/data/create_dataset_csv.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import csv 3 | import hashlib 4 | from pathlib import Path 5 | 6 | 7 | def remove_dollar_tokens(text: str): 8 | return " ".join(token for token in text.split(" ") if not token.startswith("$")).strip() 9 | 10 | 11 | def generate_samples(images_directory: Path, csv_path: Path): 12 | # Load data from CSV file 13 | signwritings = set() 14 | data = [] 15 | with open(csv_path, 'r', encoding="utf-8") as csv_file: 16 | csv.field_size_limit(2 ** 20) 17 | reader = csv.DictReader(csv_file) 18 | for row in reader: 19 | signwriting, caption = row["source"], row["target"] 20 | 21 | # There is a bug in the training script that does not allow the value "None" or "null" 22 | if caption.strip().lower() in ["none", "null", "na"]: 23 | continue 24 | 25 | signwriting = remove_dollar_tokens(signwriting) 26 | signwritings.add(signwriting) 27 | data.append((signwriting, caption)) 28 | 29 | # map signwriting to image path 30 | images = list(images_directory.glob("*.png")) 31 | image_names = set(image.stem for image in images) 32 | signwriting_to_image = {} 33 | for signwriting in signwritings: 34 | fsignwriting_md5 = hashlib.md5(signwriting.encode('utf-8')).hexdigest() 35 | if fsignwriting_md5 in image_names: 36 | signwriting_to_image[signwriting] = images_directory / f"{fsignwriting_md5}.png" 37 | 38 | # filter out data without images 39 | data = [(signwriting, caption) for signwriting, caption in data if signwriting in signwriting_to_image] 40 | 41 | for i, (signwriting, caption) in enumerate(data): 42 | yield i, { 43 | "caption": caption, 44 | "image_path": signwriting_to_image[signwriting], 45 | } 46 | 47 | 48 | def save_dataset_csv(dataset, csv_output_path: Path): 49 | with open(csv_output_path, 'w', encoding="utf-8") as csv_file: 50 | writer = csv.DictWriter(csv_file, fieldnames=["image_path", "caption"], 51 | quoting=csv.QUOTE_NONNUMERIC, quotechar='"') 52 | writer.writeheader() 53 | for _, example in dataset: 54 | writer.writerow({ 55 | "image_path": example["image_path"], 56 | "caption": example["caption"], 57 | }) 58 | 59 | 60 | def main(): 61 | parser = argparse.ArgumentParser() 62 | parser.add_argument("--images-directory", type=str, required=True) 63 | parser.add_argument("--csv", type=str, required=True) 64 | parser.add_argument("--output-path", type=str, required=True) 65 | args = parser.parse_args() 66 | 67 | images_directory = Path(args.images_directory) 68 | output_path = Path(args.output_path) 69 | csv_path = Path(args.csv) 70 | 71 | dataset = generate_samples(images_directory, csv_path) 72 | save_dataset_csv(dataset, output_path) 73 | 74 | 75 | if __name__ == "__main__": 76 | main() 77 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # SignWritingCLIP 2 | 3 | CLIP model to embed SignWriting images. 4 | 5 | Can be used in [signwriting-evaluation](https://github.com/sign-language-processing/signwriting-evaluation). 6 | 7 | Trained on [SignBank+](https://github.com/sign-language-processing/signbank-plus), 8 | by correlating the SignWriting images with their corresponding texts. 9 | This does not guarantee that the embeddings are useful for lexical tasks like transcription, 10 | and an empirical evaluation is needed. 11 | 12 | Preliminary results show that the model can be used for more semantic tasks, but not for lexical tasks. 13 | The model is hosted on [HuggingFace](https://huggingface.co/sign/signwriting-clip). 14 | 15 | ## Training a model 16 | 17 | ```bash 18 | # 0. Setup the environment. 19 | conda create --name vq python=3.11 20 | conda activate clip 21 | pip install . 22 | 23 | # 1. Creates a dataset of SignWriting images. 24 | DATA_DIR=/scratch/amoryo/clip 25 | sbatch scripts/create_dataset.sh "$DATA_DIR" 26 | 27 | # 2. Trains the model and reports to `wandb`. 28 | sbatch scripts/train_model.sh "$DATA_DIR" 29 | ``` 30 | 31 | ## CLIP vs SignWritingCLIP 32 | 33 | ### Cosine Similarity Distribution 34 | 35 | The original CLIP model encodes all SignWriting images as very similar embeddings, 36 | as evident by the distribution of cosine similarities between embeddings of random signs. 37 | The distribution is head-heavy, with most similarities between 0.8 and 0.9. 38 | 39 | ![cosine similarity distribution for CLIP](assets/distribution/CLIP.png) 40 | 41 | On the other hand, SignWritingCLIP encodes SignWriting images as more diverse embeddings. 42 | This is evident by a more tail-heavy distribution. 43 | 44 | ![cosine similarity distribution for SignWritingCLIP](assets/distribution/SignWritingCLIP.png) 45 | 46 | ### Nearest Neighbors 47 | 48 | For three signs representing the sign for "hello", the nearest neighbors are shown below. 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 |
CLIPSignWritingCLIPCLIPSignWritingCLIPCLIPSignWritingCLIP
1
2
3
4
5
6
7
8
9
10
--------------------------------------------------------------------------------