├── .github ├── ISSUE_TEMPLATE │ └── 01-request.md └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE └── README.md /.github/ISSUE_TEMPLATE/01-request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: "\U0001F381 Feature Request" 3 | about: For the proposal to improve or enhance this repository 4 | 5 | --- 6 | 7 | Feature description 8 | --------- 9 | 10 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | # Description 2 | 3 | 4 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Created by .ignore support plugin (hsz.mobi) 2 | ### Python template 3 | # Byte-compiled / optimized / DLL files 4 | __pycache__/ 5 | *.py[cod] 6 | *$py.class 7 | 8 | # C extensions 9 | *.so 10 | 11 | # Distribution / packaging 12 | .Python 13 | env/ 14 | build/ 15 | develop-eggs/ 16 | dist/ 17 | downloads/ 18 | eggs/ 19 | .eggs/ 20 | lib/ 21 | lib64/ 22 | parts/ 23 | sdist/ 24 | var/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .coverage 43 | .coverage.* 44 | .cache 45 | nosetests.xml 46 | coverage.xml 47 | *,cover 48 | .hypothesis/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | 58 | # Flask stuff: 59 | instance/ 60 | .webassets-cache 61 | 62 | # Scrapy stuff: 63 | .scrapy 64 | 65 | # Sphinx documentation 66 | docs/_build/ 67 | 68 | # PyBuilder 69 | target/ 70 | 71 | # Jupyter Notebook 72 | .ipynb_checkpoints 73 | 74 | # pyenv 75 | .python-version 76 | 77 | # celery beat schedule file 78 | celerybeat-schedule 79 | 80 | # dotenv 81 | .env 82 | 83 | # virtualenv 84 | .venv/ 85 | venv/ 86 | ENV/ 87 | 88 | # Spyder project settings 89 | .spyderproject 90 | 91 | # Rope project settings 92 | .ropeproject 93 | ### JetBrains template 94 | # Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and Webstorm 95 | # Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839 96 | 97 | # User-specific stuff: 98 | .idea/workspace.xml 99 | .idea/tasks.xml 100 | 101 | # Sensitive or high-churn files: 102 | .idea/dataSources/ 103 | .idea/dataSources.ids 104 | .idea/dataSources.xml 105 | .idea/dataSources.local.xml 106 | .idea/sqlDataSources.xml 107 | .idea/dynamic.xml 108 | .idea/uiDesigner.xml 109 | 110 | # Gradle: 111 | .idea/gradle.xml 112 | .idea/libraries 113 | 114 | # Mongo Explorer plugin: 115 | .idea/mongoSettings.xml 116 | 117 | ## File-based project format: 118 | *.iws 119 | 120 | ## Plugin-specific files: 121 | 122 | # IntelliJ 123 | /out/ 124 | 125 | # mpeltonen/sbt-idea plugin 126 | .idea_modules/ 127 | 128 | # JIRA plugin 129 | atlassian-ide-plugin.xml 130 | 131 | # Crashlytics plugin (for Android Studio and IntelliJ) 132 | com_crashlytics_export_strings.xml 133 | crashlytics.properties 134 | crashlytics-build.properties 135 | fabric.properties 136 | ### Windows template 137 | # Windows image file caches 138 | Thumbs.db 139 | ehthumbs.db 140 | 141 | # Folder config file 142 | Desktop.ini 143 | 144 | # Recycle Bin used on file shares 145 | $RECYCLE.BIN/ 146 | 147 | # Windows Installer files 148 | *.cab 149 | *.msi 150 | *.msm 151 | *.msp 152 | 153 | # Windows shortcuts 154 | *.lnk 155 | ### macOS template 156 | *.DS_Store 157 | .AppleDouble 158 | .LSOverride 159 | 160 | # Icon must end with two \r 161 | Icon 162 | 163 | 164 | # Thumbnails 165 | ._* 166 | 167 | # Files that might appear in the root of a volume 168 | .DocumentRevisions-V100 169 | .fseventsd 170 | .Spotlight-V100 171 | .TemporaryItems 172 | .Trashes 173 | .VolumeIcon.icns 174 | .com.apple.timemachine.donotpresent 175 | 176 | # Directories potentially created on remote AFP share 177 | .AppleDB 178 | .AppleDesktop 179 | Network Trash Folder 180 | Temporary Items 181 | .apdisk 182 | ### VirtualEnv template 183 | # Virtualenv 184 | # http://iamzed.com/2009/05/07/a-primer-on-virtualenv/ 185 | [Bb]in 186 | [Ii]nclude 187 | [Ll]ib 188 | [Ll]ib64 189 | [Ll]ocal 190 | [Ss]cripts 191 | pyvenv.cfg 192 | .venv 193 | pip-selfcheck.json 194 | 195 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at [@Hironsan](https://github.com/Hironsan). All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to contribute to auto-labeling-client? 2 | 3 | 1. Fork the [repository](https://github.com/doccano/auto-labeling-client) by clicking on the 'Fork' button on the repository's page. This creates a copy of the code under your GitHub user account. 4 | 5 | 2. Clone your fork to your local disk, and add the base repository as a remote: 6 | 7 | ```bash 8 | git clone git@github.com:/auto-labeling-client.git 9 | cd auto-labeling-client 10 | git remote add upstream https://github.com/doccano/auto-labeling-client.git 11 | ``` 12 | 13 | 3. Create a new branch to hold your development changes: 14 | 15 | ```bash 16 | git checkout -b a-descriptive-name-for-my-changes 17 | ``` 18 | 19 | **do not** work on the `master` branch. 20 | 21 | 4. Set up a development environment by running the following command in a virtual environment: 22 | 23 | ```bash 24 | pipenv install --dev 25 | ``` 26 | 27 | 5. Develop the features on your branch. 28 | 29 | 6. Format your code. Run flake8 and isort so that your newly added files look nice with the following command: 30 | 31 | ```bash 32 | pipenv run flake8 33 | pipenv run isort 34 | ``` 35 | 36 | 7. Once you're happy with your script file, add your changes and make a commit to record your changes locally: 37 | 38 | ```bash 39 | git add auto-labeling-client/ 40 | git commit 41 | ``` 42 | 43 | It is a good idea to sync your copy of the code with the original 44 | repository regularly. This way you can quickly account for changes: 45 | 46 | ```bash 47 | git fetch upstream 48 | git rebase upstream/master 49 | ``` 50 | 51 | Push the changes to your account using: 52 | 53 | ```bash 54 | git push -u origin a-descriptive-name-for-my-changes 55 | ``` 56 | 57 | 8. Once you are satisfied, go the webpage of your fork on GitHub. Click on "Pull request" to send your to the project maintainers for review. 58 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Hiroki Nakayama 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # awesome-annotation-tools 2 | 3 | A curated list of awesome annotation tools. Please feel free to pull requests to add links. 4 | 5 | ## Text 6 | 7 | Caution: This is based on my survey about a year ago. I will update it asap. 8 | 9 | ### Supporting Tasks 10 | 11 | | | Classification | Sequence Labeling | Seq2seq | Relation | Dictionary | Choice | 12 | | ------------------------------------------------------------------------------------------------------------- | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: | 13 | | [doccano](https://github.com/doccano/doccano) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | 14 | | [Prodigy](https://prodi.gy) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | 15 | | [Prodigy Teams](https://support.prodi.gy/t/prodigy-annotation-manager-update-prodigy-scale-prodigy-teams/805) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | :heavy_check_mark: | 16 | | [tagtog](https://www.tagtog.net) | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | | 17 | | [LightTag](https://www.lighttag.io) | :heavy_check_mark: | :heavy_check_mark: | | | | | 18 | | [Dataturks](https://dataturks.com) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | 19 | | [Smart](https://github.com/RTIInternational/SMART) | :heavy_check_mark: | | | | | :heavy_check_mark: | 20 | | [Universal Data Tool](https://universaldatatool.com/) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | :heavy_check_mark: | 21 | | [Label Studio(OSS)](https://github.com/heartexlabs/label-studio) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | :heavy_check_mark: | 22 | | [Label Studio(Enterprise)](https://www.heartex.ai/) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | :heavy_check_mark: | 23 | | [Humanloop](https://humanloop.com/) | :heavy_check_mark: | :heavy_check_mark: | ? | :heavy_check_mark: | ? | :heavy_check_mark: | 24 | | [Snorkel Flow](https://snorkel.ai/) | :heavy_check_mark: | ? | ? | ? | ? | :heavy_check_mark: | 25 | | [Taggle](https://www.nlp.town/taggle/) | :heavy_check_mark: | :heavy_check_mark: | ? | :heavy_check_mark: | ? | :heavy_check_mark: | 26 | | [UBIAI](https://ubiai.tools/) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | 27 | | [Annotation Lab](https://www.johnsnowlabs.com/annotation-lab/) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | 28 | 29 | ### Supporting Active Learning or Auto Labeling 30 | 31 | | | Active Learning | Auto Labeling | Model Training | API Access | 32 | | ------------------------ | :----------------: | :----------------: | :----------------: | :----------------: | 33 | | doccano | | | | :heavy_check_mark: | 34 | | prodigy | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | 35 | | Prodigy Teams | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | 36 | | tagtog | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | 37 | | LightTag | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | 38 | | Dataturks | | | | :heavy_check_mark: | 39 | | Smart | :heavy_check_mark: | :heavy_check_mark: | | | 40 | | Universal Data Tool | | | | | 41 | | Label Studio(OSS) | :heavy_check_mark: | :heavy_check_mark: | | ? | 42 | | Label Studio(Enterprise) | :heavy_check_mark: | :heavy_check_mark: | ? | ? | 43 | | Humanloop | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | 44 | | Snorkel Flow | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | 45 | | Taggle | :heavy_check_mark: | :heavy_check_mark: | ? | ? | 46 | | UBIAI | ? | :heavy_check_mark: | :heavy_check_mark: | ? | 47 | | Annotation Lab | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | 48 | 49 | ### Price and other 50 | 51 | | | Price range(12 month) | Customizable | On-premise support | Collaboration | OSS | Quality Control (e.g. benchmarking, IRR/IAA) | 52 | | ------------------------ | --------------------- | :----------------: | :----------------: | :----------------: | :----------------: | :------------------------------------------: | 53 | | doccano | \$0 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | 54 | | Prodigy | \$390-$10,000 | | :heavy_check_mark: | | | | 55 | | Prodigy Team | ? | | :heavy_check_mark: | :heavy_check_mark: | | | 56 | | tagtog | \$0-$12,000 | | :heavy_check_mark: | :heavy_check_mark: | | | 57 | | LightTag | \$1,200 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | 58 | | Dataturks | \$0-3,600 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | 59 | | Smart | \$0 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | IRR | 60 | | Universal Data Tool | \$0 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | Course/Benchmarking | 61 | | Label Studio(OSS) | \$0 | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | 62 | | Label Studio(Enterprise) | ? | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | 63 | | Humanloop | \$0 - $999 | ? | :heavy_check_mark: | :heavy_check_mark: | | | 64 | | Snorkel Flow | ? | ? | ? | :heavy_check_mark: | | | 65 | | Taggle | ? | ? | ? | :heavy_check_mark: | | 66 | | UBIAI | \$0 - $1000 | :heavy_check_mark: | ? | :heavy_check_mark: | | IAA| 67 | | Annotation Lab | Free | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Analytics & IAA| 68 | 69 | ## Contribution 70 | 71 | Welcome to the Pull Requests! 72 | --------------------------------------------------------------------------------