├── README.md ├── requirements.txt ├── setup.py ├── tokenizer.gitignore └── tokenizer ├── README.md ├── __init__.py ├── data └── emoticons.txt ├── reg.py ├── requirements.txt ├── setup.py ├── test_regularizer.py ├── test_tokenizer.py ├── tokenizer.gitignore └── tokenizer.py /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/README.md -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | nltk==3.2.1 2 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/setup.py -------------------------------------------------------------------------------- /tokenizer.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer.gitignore -------------------------------------------------------------------------------- /tokenizer/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/README.md -------------------------------------------------------------------------------- /tokenizer/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | -------------------------------------------------------------------------------- /tokenizer/data/emoticons.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/data/emoticons.txt -------------------------------------------------------------------------------- /tokenizer/reg.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/reg.py -------------------------------------------------------------------------------- /tokenizer/requirements.txt: -------------------------------------------------------------------------------- 1 | nltk==3.2.1 2 | -------------------------------------------------------------------------------- /tokenizer/setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/setup.py -------------------------------------------------------------------------------- /tokenizer/test_regularizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/test_regularizer.py -------------------------------------------------------------------------------- /tokenizer/test_tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/test_tokenizer.py -------------------------------------------------------------------------------- /tokenizer/tokenizer.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/tokenizer.gitignore -------------------------------------------------------------------------------- /tokenizer/tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikavaris/tokenizer/HEAD/tokenizer/tokenizer.py --------------------------------------------------------------------------------