├── .gitignore ├── LICENSE.txt ├── Makefile ├── README.md ├── media ├── Ruler_EMNLP2020.pdf ├── concept_detail.png ├── fast-exploration-alt-2.gif ├── fast-exploration-alt.gif ├── fast-exploration.gif ├── fast-exporation-thin.gif ├── labeling_pane.png ├── overview.png ├── qualitative.png ├── quantitative.png ├── ruler_teaser.gif ├── ruler_teaser_tall.png ├── ruler_teaser_wide.png └── ruler_ui.png ├── server ├── .gitignore ├── api │ ├── __init__.py │ ├── active_sampler.py │ ├── dataset.py │ ├── endpoints.py │ ├── project.py │ ├── server.py │ └── swagger.yml ├── config.py ├── datasets │ ├── .placeholder │ ├── README.md │ └── spam_example │ │ ├── example_dataset.csv │ │ └── processed.csv ├── environment.yml ├── models │ ├── .placeholder │ └── README.md ├── requirements.txt ├── synthesizer │ ├── __init__.py │ ├── gll.py │ ├── parser.py │ └── synthesizer.py ├── tests │ ├── __init__.py │ ├── test_api.py │ ├── test_dataset.py │ ├── test_modeler.py │ ├── test_project.py │ └── test_translator.py └── verifier │ ├── __init__.py │ ├── eval_utils.py │ ├── interaction_db.py │ ├── keraslogreg.py │ ├── labeling_function.py │ ├── modeler.py │ └── translator.py ├── ui ├── .env ├── .gitignore ├── .idea │ ├── .gitignore │ ├── dpbd.iml │ ├── inspectionProfiles │ │ └── Project_Default.xml │ ├── misc.xml │ ├── modules.xml │ └── vcs.xml ├── README.md ├── package-lock.json ├── package.json ├── public │ ├── favicon.png │ ├── index.html │ ├── manifest.json │ └── robots.txt └── src │ ├── AnnotationBuilder.js │ ├── AnnotationDisplay.js │ ├── AnnotationDisplayCollapse.js │ ├── App.js │ ├── App.test.js │ ├── ClassLabelsCollection.js │ ├── ColorPalette.js │ ├── Concept.js │ ├── ConceptCollection.js │ ├── ConceptElement.js │ ├── Footer.js │ ├── GithubIcon.js │ ├── LFPanel.js │ ├── LabelingFunctionsSelected.js │ ├── LabelingFunctionsSuggested.js │ ├── LeftDrawer.js │ ├── Link.js │ ├── Main.js │ ├── MegagonIcon.js │ ├── Navigation.js │ ├── NavigationBar.js │ ├── NavigationButtons.js │ ├── ProjectCreation │ ├── DatasetSelect.js │ ├── LabelConfig.js │ ├── ModelSelect.js │ └── ProjectCreation.js │ ├── ProjectGrid.js │ ├── RichTextUtils.js │ ├── SaveButton.js │ ├── SelectedSpan.js │ ├── SortingTableUtils.js │ ├── Span.js │ ├── StatisticsLRPane.js │ ├── StatisticsPane.js │ ├── actions │ ├── ConceptStyler.js │ ├── annotate.js │ ├── concepts.js │ ├── connectivesAndKeyTypes.js │ ├── datasets.js │ ├── getStatistics.js │ ├── getText.js │ ├── interaction.js │ ├── labelAndSuggestLF.js │ ├── labelClasses.js │ ├── loadingBar.js │ ├── model.js │ ├── save.js │ └── submitLFs.js │ ├── errorSnackbar.js │ ├── index.js │ ├── reducers.js │ ├── reducers │ ├── ConnectivesKeyTypesReducer.js │ ├── annotationsReducer.js │ ├── conceptsReducer.js │ ├── datasetsReducer.js │ ├── interactionHistoryReducer.js │ ├── labelAndSuggestLFReducer.js │ ├── labelClassesReducer.js │ ├── labelExampleReducer.js │ ├── loadingBarReducer.js │ ├── modelsReducer.js │ ├── reducers.js │ ├── saveFileReducer.js │ ├── selectedLFReducer.js │ ├── statisticsReducer.js │ └── textReducer.js │ ├── serviceWorker.js │ └── store.js └── user_study ├── .ipynb_checkpoints └── ruler_user_study_figures-checkpoint.ipynb ├── README.md ├── background_survey_anon.csv ├── exit_survey_anon.csv ├── final_survey_anon.csv ├── full_study_data.csv └── ruler_user_study_figures.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | *.DS_Store 2 | */data/* 3 | venv/* 4 | *.egg 5 | server/Sphinx* 6 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | VENV_NAME?=venv 2 | PYTHON=$(PWD)/$(VENV_NAME)/bin/python 3 | PIP=$(PWD)/$(VENV_NAME)/bin/pip 4 | export DATA_DIR = $(PWD)server/datasets 5 | 6 | 7 | venv: FORCE 8 | test -d $(VENV_NAME) || python3 -m venv $(VENV_NAME) 9 | . $(VENV_NAME)/bin/activate 10 | 11 | install: venv 12 | $(PIP) install -r server/requirements.txt 13 | @cd ui; npm install 14 | @$(PYTHON) -m spacy download en_core_web_sm 15 | 16 | build: venv 17 | npm build app/react-app 18 | 19 | test: venv 20 | cd server; $(PYTHON) -m unittest discover 21 | 22 | server: venv FORCE 23 | cd server; $(PYTHON) api/server.py 24 | 25 | ui: venv FORCE 26 | cd ui; npm start 27 | 28 | gitclean: 29 | #TODO 30 | 31 | uninstall: 32 | #TODO 33 | 34 | FORCE: 35 | -------------------------------------------------------------------------------- /media/Ruler_EMNLP2020.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/Ruler_EMNLP2020.pdf -------------------------------------------------------------------------------- /media/concept_detail.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/concept_detail.png -------------------------------------------------------------------------------- /media/fast-exploration-alt-2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/fast-exploration-alt-2.gif -------------------------------------------------------------------------------- /media/fast-exploration-alt.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/fast-exploration-alt.gif -------------------------------------------------------------------------------- /media/fast-exploration.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/fast-exploration.gif -------------------------------------------------------------------------------- /media/fast-exporation-thin.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/fast-exporation-thin.gif -------------------------------------------------------------------------------- /media/labeling_pane.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/labeling_pane.png -------------------------------------------------------------------------------- /media/overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/overview.png -------------------------------------------------------------------------------- /media/qualitative.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/qualitative.png -------------------------------------------------------------------------------- /media/quantitative.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/quantitative.png -------------------------------------------------------------------------------- /media/ruler_teaser.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/ruler_teaser.gif -------------------------------------------------------------------------------- /media/ruler_teaser_tall.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/ruler_teaser_tall.png -------------------------------------------------------------------------------- /media/ruler_teaser_wide.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/ruler_teaser_wide.png -------------------------------------------------------------------------------- /media/ruler_ui.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/media/ruler_ui.png -------------------------------------------------------------------------------- /server/.gitignore: -------------------------------------------------------------------------------- 1 | *.aux 2 | *.bbl 3 | *.log 4 | *.blg 5 | *.ent 6 | *.idx 7 | *.out 8 | *.synctex.gz 9 | __pycache__/ 10 | .RData 11 | .Rhistory 12 | main.pdf 13 | .DS_Store 14 | code/venv 15 | code/.idea 16 | venv/* 17 | 18 | # User-specific stuff 19 | datasets/* 20 | models/* 21 | .idea/**/workspace.xml 22 | .idea/**/tasks.xml 23 | .idea/**/usage.statistics.xml 24 | .idea/**/dictionaries 25 | .idea/**/shelf 26 | 27 | # Generated files 28 | .idea/**/contentModel.xml 29 | 30 | # Sensitive or high-churn files 31 | .idea/**/dataSources/ 32 | .idea/**/dataSources.ids 33 | .idea/**/dataSources.local.xml 34 | .idea/**/sqlDataSources.xml 35 | .idea/**/dynamic.xml 36 | .idea/**/uiDesigner.xml 37 | .idea/**/dbnavigator.xml 38 | 39 | # Gradle 40 | .idea/**/gradle.xml 41 | .idea/**/libraries 42 | 43 | # Gradle and Maven with auto-import 44 | # When using Gradle or Maven with auto-import, you should exclude module files, 45 | # since they will be recreated, and may cause churn. Uncomment if using 46 | # auto-import. 47 | # .idea/modules.xml 48 | # .idea/*.iml 49 | # .idea/modules 50 | # *.iml 51 | # *.ipr 52 | 53 | # CMake 54 | cmake-build-*/ 55 | 56 | # Mongo Explorer plugin 57 | .idea/**/mongoSettings.xml 58 | 59 | # File-based project format 60 | *.iws 61 | 62 | # IntelliJ 63 | out/ 64 | 65 | # mpeltonen/sbt-idea plugin 66 | .idea_modules/ 67 | 68 | # JIRA plugin 69 | atlassian-ide-plugin.xml 70 | 71 | # Cursive Clojure plugin 72 | .idea/replstate.xml 73 | 74 | # Crashlytics plugin (for Android Studio and IntelliJ) 75 | com_crashlytics_export_strings.xml 76 | crashlytics.properties 77 | crashlytics-build.properties 78 | fabric.properties 79 | 80 | # Editor-based Rest Client 81 | .idea/httpRequests 82 | 83 | # Android studio 3.1+ serialized cache file 84 | .idea/caches/build_file_checksums.ser 85 | 86 | snorkel.db 87 | .idea 88 | *.json 89 | *.pkl 90 | *.csv 91 | *.zip 92 | *.pyc 93 | *.txt 94 | -------------------------------------------------------------------------------- /server/api/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/server/api/__init__.py -------------------------------------------------------------------------------- /server/api/active_sampler.py: -------------------------------------------------------------------------------- 1 | from math import log 2 | 3 | def next_text(modeler, dataset, subset_size=50): 4 | # Limit our search to the least seen examples 5 | # TODO it's a little hacky to use the dataframe underlying the dataset... 6 | min_seen = dataset.df["seen"].min() 7 | least_seen_examples = dataset.df[dataset.df["seen"]==min_seen] 8 | 9 | if ((len(least_seen_examples)==1) \ 10 | or (len(modeler.get_lfs())==0)): 11 | # We have no labelling functions, or only one example hasn't been seen: 12 | res_idx = least_seen_examples.sample(1).index[0] 13 | else: 14 | modeler.fit(dataset) 15 | # Sample at most subset_size examples 16 | subset_size = min(subset_size, len(least_seen_examples)) 17 | subset = least_seen_examples.sample(subset_size) 18 | 19 | probs = modeler.predict(subset) 20 | entropies = [entropy(x) for x in probs] 21 | subset = subset[entropies==max(entropies)] 22 | 23 | res_idx = subset.sample(1).index[0] 24 | dataset.df.at[res_idx, "seen"] += 1 25 | return {"text": dataset.df.at[res_idx, "text"], "id": int(res_idx)} 26 | 27 | def entropy(prob_dist): 28 | #return(-(L_row_i==-1).sum()) 29 | return(-sum([x*log(x) for x in prob_dist])) -------------------------------------------------------------------------------- /server/api/server.py: -------------------------------------------------------------------------------- 1 | """ 2 | Main module of the server file 3 | """ 4 | 5 | import connexion 6 | 7 | from flask import redirect 8 | from flask import render_template 9 | from flask_cors import CORS 10 | 11 | 12 | 13 | # create the application instance 14 | app = connexion.App(__name__, specification_dir="./") 15 | 16 | # Cead the swagger.yml file to configure the endpoints 17 | app.add_api("swagger.yml") 18 | 19 | CORS(app.app, supports_credentials=True, resources={r"/*": {"origins": "*"}}) 20 | 21 | # Create a URL route in our application for "/" 22 | @app.route("/") 23 | def home(): 24 | """ 25 | This function just responds to the browser URL 26 | localhost:5000/ 27 | 28 | :return: the rendered templates "home.html" 29 | """ 30 | return redirect("/api/ui", code=302) 31 | 32 | 33 | 34 | if __name__ == "__main__": 35 | try: 36 | app.run(debug=True, threaded=False) 37 | except KeyboardInterrupt: 38 | pass -------------------------------------------------------------------------------- /server/config.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | # Names of the directories to store models and datasets 4 | dir_path = os.path.dirname(os.path.realpath(__file__)) 5 | MODELS_PATH = os.path.join(dir_path, "models") 6 | DATASETS_PATH = os.path.join(dir_path, "datasets") 7 | 8 | # Ruler will preprocess your data (detecting named entities, for example) and store them in a csv file 9 | PROCESSED_FILE_NAME = 'processed.csv' 10 | 11 | 12 | 13 | # CUSTOMIZABLE SETTINGS 14 | 15 | # If the size of the labelled data is smaller than MIN_LABELLED_SIZE, a warning will be logged 16 | MIN_LABELLED_SIZE = 20 17 | 18 | # To keep response times short, you can cap the number of unlabelled training examples to use when developing your model 19 | DEFAULT_MAX_TRAINING_SIZE = 3000 -------------------------------------------------------------------------------- /server/datasets/.placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/server/datasets/.placeholder -------------------------------------------------------------------------------- /server/datasets/README.md: -------------------------------------------------------------------------------- 1 | # Datasets 2 | This folder is where you will upload the data you would like to use with Ruler. 3 | You can place it directly in this folder, or you can upload a csv file via the UI-- either way it will end up here. 4 | 5 | ## Data Format 6 | Data should be uploaded as a csv file, with a column named `text` denoting the text that you would like to classify/annotate. 7 | To get the most out of Ruler, you should have a small labelled development set as well. 8 | This enables the interactive statistics that tell you how your functions are performing. This column should be named `label`. 9 | 10 | For example, data for a spam classification task (`{0: HAM, 1: SPAM}`) might have this format: 11 | 12 | | id | text | label | 13 | |----|----------------------------------------------|-------| 14 | | 0 | Buy my sand pastries! They are mostly edible | 1 | 15 | | 1 | I love this video | 0 | 16 | 17 | ## Example Data 18 | You can download a sample spam classification dataset [here](https://archive.ics.uci.edu/ml/datasets/YouTube+Spam+Collection). 19 | The provided description: "It is a public set of comments collected for spam research. It has five datasets composed by 1,956 real messages extracted from five videos that were among the 10 most viewed on the collection period." 20 | 21 | 22 | -------------------------------------------------------------------------------- /server/datasets/spam_example/example_dataset.csv: -------------------------------------------------------------------------------- 1 | ,COMMENT_ID,AUTHOR,DATE,CONTENT,CLASS 2 | 0,0,Black Widow,2014-07-22T15:27:50,i love this so much. AND also I Generate Free Leads on Auto Pilot & You Can Too! http://www.some.url.seems.spammy.tk,1 3 | 1,1,Dracula,2014-07-27T01:57:16,http://my/fake/website/fan-face-off-round-3 Vote for me so I can be on the cover of FACES: HOW TO HAVE A FACE,1 4 | 2,2,Frankenstein,2014-07-27T01:57:16,If you don't like this comment I will find you and confront you IRL.,1 5 | 3,3,Your Inner Child,2014-07-27T01:57:16,https://www.law.berkeley.edu/article/spinning-around-the-laugh-track/,1 6 | 4,4,E. Coli,2014-07-27T01:57:16,https://www.law.berkeley.edu/article/spinning-around-the-laugh-track/,1 7 | 5,5,Don Jr's subconscious,2014-07-27T01:57:16,http://sanfranciscocomedycompetition.com/2018-competition/hanna-evensen/,1 8 | 6,6,Rick,2014-07-27T01:57:16,http://sanfranciscocomedycompetition.com/2018-competition/hanna-evensen/ no one will notice if I leave this here?,1 9 | 7,7,Morty,2014-07-27T01:57:22,http://sanfranciscocomedycompetition.com/2018-competition/hanna-evensen/ if you're reading the test cases...why?,1 10 | 8,8,Popeye,2014-07-27T01:57:16,https://pi.ytmnd.com/,1 11 | 9,9,Erica,2014-07-27T02:51:43,"""https://pi.ytmnd.com/""",1 12 | 10,10,Tim,2014-08-01T12:27:48,Check out my Youtube channel where I make pastries out of sand,1 13 | 11,11,Timm,2014-08-01T12:27:48,Check out my Youtube channel where I make pastries out of sand,1 14 | 12,12,Tim Tam,2014-08-01T12:27:48,Check out my Youtube channel where I make pastries out of sand,1 15 | 13,13,Timmy Jr,2014-08-01T12:27:48,Check out my Youtube channel where I make pastries out of sand,1 16 | 14,14,Timothy,2014-08-01T12:27:48,Check out my Youtube channel where I make pastries out of sand,1 17 | 15,15,Bernadette,2014-08-01T12:27:48,Check out my Youtube channel where I make pastries out of sand,1 18 | 16,16,Sam,2014-08-01T12:27:48,Buy my sand pastries! They are mostly edible,1 19 | 17,17,Ari,2014-08-01T12:27:48,Buy my sand pastries! They are mostly edible,1 20 | 18,18,Ted,2014-08-01T12:27:48,This is the best music video I ever saw,0 21 | 19,25,Ted,2014-08-01T12:27:48,This comment is so relevant to the music video I'm watching,0 22 | 20,35,Ted,2014-08-01T12:27:48,I like music,0 23 | 21,45,Ted,2014-08-01T12:27:48,This video makes me cry every time,0 24 | 22,55,Ted,2014-08-01T12:27:48,This is the best music video I ever saw,0 25 | 23,65,Ted,2014-08-01T12:27:48,This is the best music video I ever saw,0 26 | 24,66,Tom,2014-08-01T12:27:48,So good,0 27 | 25,67,Myra,2014-08-01T12:27:48,Great,0 28 | -------------------------------------------------------------------------------- /server/datasets/spam_example/processed.csv: -------------------------------------------------------------------------------- 1 | ,Unnamed: 0,COMMENT_ID,AUTHOR,DATE,text,label,file,split,CARDINAL,EVENT,FAC,GPE,LANGUAGE,LAW,LOC,MONEY,NORP,ORDINAL,ORG,PERCENT,PERSON,PRODUCT,QUANTITY,TIME,WORK_OF_ART 2 | 0,22,55,Ted,False,This is the best music video I ever saw,0,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 3 | 1,8,8,Popeye,False,https://pi.ytmnd.com/,1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 4 | 2,23,65,Ted,False,This is the best music video I ever saw,0,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 5 | 3,18,18,Ted,False,This is the best music video I ever saw,0,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 6 | 4,20,35,Ted,False,I like music,0,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 7 | 5,3,3,Your Inner Child,False,https://www.law.berkeley.edu/article/spinning-around-the-laugh-track/,1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 8 | 6,9,9,Erica,False,"""https://pi.ytmnd.com/""",1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 9 | 7,1,1,Dracula,False,http://my/fake/website/fan-face-off-round-3 Vote for me so I can be on the cover of FACES: HOW TO HAVE A FACE,1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,"(81, 83)",False,False,False,False,False,False 10 | 8,10,10,Tim,False,Check out my Youtube channel where I make pastries out of sand,1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 11 | 9,19,25,Ted,False,This comment is so relevant to the music video I'm watching,0,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 12 | 10,17,17,Ari,False,Buy my sand pastries! They are mostly edible,1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 13 | 11,6,6,Rick,False,http://sanfranciscocomedycompetition.com/2018-competition/hanna-evensen/ no one will notice if I leave this here?,1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 14 | 12,24,66,Tom,False,So good,0,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 15 | 13,2,2,Frankenstein,False,If you don't like this comment I will find you and confront you IRL.,1,datasets/spam_example/example_dataset.csv,dev,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 16 | 14,5,5,Don Jr's subconscious,False,http://sanfranciscocomedycompetition.com/2018-competition/hanna-evensen/,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 17 | 15,7,7,Morty,False,http://sanfranciscocomedycompetition.com/2018-competition/hanna-evensen/ if you're reading the test cases...why?,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 18 | 16,11,11,Timm,False,Check out my Youtube channel where I make pastries out of sand,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 19 | 17,12,12,Tim Tam,False,Check out my Youtube channel where I make pastries out of sand,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 20 | 18,25,67,Myra,False,Great,0,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 21 | 19,4,4,E. Coli,False,https://www.law.berkeley.edu/article/spinning-around-the-laugh-track/,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 22 | 20,16,16,Sam,False,Buy my sand pastries! They are mostly edible,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 23 | 21,14,14,Timothy,False,Check out my Youtube channel where I make pastries out of sand,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 24 | 22,21,45,Ted,False,This video makes me cry every time,0,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 25 | 23,15,15,Bernadette,False,Check out my Youtube channel where I make pastries out of sand,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 26 | 24,0,0,Black Widow,False,i love this so much. AND also I Generate Free Leads on Auto Pilot & You Can Too! http://www.some.url.seems.spammy.tk,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,"(70, 81)" 27 | 25,13,13,Timmy Jr,False,Check out my Youtube channel where I make pastries out of sand,1,datasets/spam_example/example_dataset.csv,train,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False 28 | -------------------------------------------------------------------------------- /server/environment.yml: -------------------------------------------------------------------------------- 1 | name: IDEA2 2 | channels: 3 | - conda-forge 4 | - pytorch 5 | dependencies: 6 | - snorkel==0.9.0 7 | - spacy 8 | - tensorflow 9 | - flask 10 | - connexion 11 | - flask-cors 12 | 13 | -------------------------------------------------------------------------------- /server/models/.placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/server/models/.placeholder -------------------------------------------------------------------------------- /server/models/README.md: -------------------------------------------------------------------------------- 1 | # Models 2 | This is where all your saved models are stored. There are several files that describe a model: 3 | 4 | ### concept_collection.json 5 | All of the concepts created in conjunction with this model, stored as json. 6 | 7 | ### custom_functions.py 8 | __Directly edit this file with your custom python functions.__ 9 | 10 | Write your functions in this file, along with any necessary imports, and next you load the model those functions will be visible in the UI for you to debug (see false positives, conflicts, etc.) 11 | Make sure you have all necessary dependencies for your functions available. 12 | 13 | ### label_model.pkl 14 | The serialized label_model 15 | 16 | ### labels.json 17 | Definitions of the label classes for your model/task. 18 | For example: 19 | 20 | `{"name": "spam", "dict": {"ABSTAIN": -1, "labels": {"non spam": 0, "spam": 1}}, "count": 1}` 21 | 22 | ### LF_DB.json 23 | The functions created with Ruler, in JSON format. 24 | -------------------------------------------------------------------------------- /server/requirements.txt: -------------------------------------------------------------------------------- 1 | connexion==2.4.0 2 | Flask==1.1.1 3 | Flask-Cors==3.0.8 4 | nltk==3.5 5 | numpy==1.17.3 6 | pandas==0.24.2 7 | scikit-learn==0.21.3 8 | scipy==1.4.1 9 | snorkel==0.9.0 10 | spacy==2.2.3 11 | swagger-ui-bundle==0.0.6 12 | tensorflow==2.3.0 13 | tqdm==4.39.0 -------------------------------------------------------------------------------- /server/synthesizer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/server/synthesizer/__init__.py -------------------------------------------------------------------------------- /server/synthesizer/parser.py: -------------------------------------------------------------------------------- 1 | import re 2 | import spacy 3 | from synthesizer.gll import Token 4 | from typing import List 5 | 6 | 7 | nlp = spacy.load("en_core_web_sm") 8 | nlp.add_pipe(nlp.create_pipe('sentencizer'), first=True) 9 | 10 | 11 | def parse(annotations: List[dict], origin_text: str, delimiter: str, concepts: dict): 12 | token_list = [] 13 | rel_code_list = [] 14 | 15 | origin_doc = nlp(origin_text) 16 | 17 | sentences = list(origin_doc.sents) 18 | 19 | for annotation in annotations: 20 | 21 | # crnt_token: text 22 | crnt_token_text = origin_text[annotation["start_offset"]:annotation["end_offset"]] 23 | # initialize a new Token class 24 | crnt_token = Token(crnt_token_text) 25 | token_list.append(crnt_token) 26 | 27 | # Find the concept from annotation 28 | crnt_concept_name = None 29 | if ("label" in annotation) and (annotation["label"]): 30 | crnt_concept_name = annotation["label"] 31 | 32 | crnt_token.assign_concept(crnt_concept_name) 33 | 34 | # Find the relationship from annotation 35 | crnt_rel_code = None 36 | if "link" in annotation: 37 | if not annotation["link"] is None: 38 | crnt_rel_code = int(annotation["link"]) 39 | 40 | rel_code_list.append(crnt_rel_code) 41 | 42 | # Find the index of current annotation 43 | flag: bool = False 44 | # print(annotated_text) 45 | for crnt_sent in sentences: 46 | sent_start = origin_doc[crnt_sent.start].idx 47 | sent_end = origin_doc[crnt_sent.end-1].idx + len(origin_doc[crnt_sent.end-1]) 48 | if (annotation["start_offset"] >= sent_start) and (annotation["end_offset"]<=sent_end): 49 | crnt_token.assign_sent_idx(sentences.index(crnt_sent)) 50 | flag = True 51 | break 52 | if not flag: 53 | print("No sentence found for the annotation: \"{}\"\nsentences: {}".format(annotation, sentences)) 54 | 55 | # Find the named entity of current annotation 56 | for ent in origin_doc.ents: 57 | if crnt_token.text in ent.text: 58 | crnt_token.assign_ner_type(ent.label_) 59 | break 60 | 61 | # Match existing concepts 62 | augment_concept(token_list, concepts) 63 | 64 | return token_list, rel_code_list 65 | 66 | 67 | def augment_concept(token_list, concepts: dict): 68 | for crnt_token in token_list: 69 | if crnt_token.concept_name is not None: 70 | continue 71 | 72 | for key in concepts.keys(): 73 | if crnt_token.text in concepts[key]: 74 | crnt_token.assign_concept(key) 75 | break 76 | 77 | 78 | # remove stop word and punct 79 | def simple_parse(text: str, concepts: dict): 80 | token_list = [] 81 | doc = nlp(text) 82 | tokens = [token.text for token in doc if not token.is_stop and not token.is_punct] 83 | #print(tokens) 84 | 85 | if len(doc) == len(tokens): 86 | # early return 87 | return token_list 88 | 89 | ner_dict = dict() 90 | 91 | # merge multiple tokens if falling into one ent.text 92 | for ent in doc.ents: 93 | matched_text = [] 94 | for i in range(len(tokens)): 95 | if tokens[i] in ent.text: 96 | matched_text.append(tokens[i]) 97 | 98 | if len(matched_text) > 1: 99 | new_text = "" 100 | for crnt_text in matched_text: 101 | new_text += crnt_text 102 | tokens.remove(crnt_text) 103 | 104 | tokens.append(ent.text) 105 | ner_dict[ent.text] = ent.label_ 106 | 107 | for crnt_text in tokens: 108 | crnt_token = Token(crnt_text) 109 | if crnt_text in ner_dict.keys(): 110 | crnt_token.assign_ner_type(ner_dict[crnt_text]) 111 | token_list.append(crnt_token) 112 | 113 | augment_concept(token_list, concepts) 114 | 115 | return token_list 116 | 117 | -------------------------------------------------------------------------------- /server/synthesizer/synthesizer.py: -------------------------------------------------------------------------------- 1 | from synthesizer.parser import parse, simple_parse 2 | from synthesizer.gll import * 3 | 4 | 5 | class Synthesizer: 6 | 7 | token_list = [] 8 | rel_code_list = [] 9 | 10 | def __init__(self, t_origin, annots, t_label, de, cs: dict): 11 | self.origin_text = t_origin 12 | self.annotations = annots 13 | self.delimiter = de 14 | self.concepts = cs 15 | self.label = t_label 16 | self.instances = [] 17 | 18 | def print(self): 19 | print(self.instances) 20 | 21 | def run(self): 22 | """ 23 | Based on one label, suggest LFs 24 | 25 | Returns: 26 | List(Dict): labeling functions represented as dicts with fields: 27 | 'Conditions', 'Connective', 'Direction', 'Label', and 'Weight' 28 | """ 29 | self.token_list, self.rel_code_list = \ 30 | parse(self.annotations, self.origin_text, self.delimiter, self.concepts) 31 | 32 | assert len(self.token_list) == len(self.rel_code_list) 33 | 34 | relationship_set = None 35 | relationship_undirected = dict() 36 | relationship_directed = dict() 37 | 38 | for i in range(len(self.rel_code_list)): 39 | crnt_rel_code = self.rel_code_list[i] 40 | crnt_token = self.token_list[i] 41 | 42 | if crnt_rel_code is None: 43 | if relationship_set is None: 44 | relationship_set = Relationship(RelationshipType.SET) 45 | relationship_set.add(crnt_token, crnt_rel_code) 46 | 47 | else: 48 | if abs(crnt_rel_code) < 100: 49 | # no direction relationship 50 | if crnt_rel_code not in relationship_undirected: 51 | relationship_undirected[crnt_rel_code] = Relationship(RelationshipType.UNDIRECTED) 52 | 53 | relationship_undirected[crnt_rel_code].add(crnt_token, crnt_rel_code) 54 | 55 | else: 56 | # directed relationship 57 | abs_code = abs(crnt_rel_code) 58 | if abs_code not in relationship_directed: 59 | relationship_directed[abs_code] = Relationship(RelationshipType.DIRECTED) 60 | 61 | relationship_directed[abs_code].add(crnt_token, crnt_rel_code) 62 | 63 | # for each relationship, generate instances 64 | if relationship_set is not None: 65 | self.instances.extend(relationship_set.get_instances(self.concepts)) 66 | for k, v in relationship_undirected.items(): 67 | self.instances.extend(v.get_instances(self.concepts)) 68 | for k, v in relationship_directed.items(): 69 | self.instances.extend(v.get_instances(self.concepts)) 70 | 71 | # more processing on single-condition 72 | if len(self.instances) == 1: 73 | self.single_condition() 74 | 75 | # add label to each instance 76 | for crnt_instance in self.instances: 77 | crnt_instance[LABEL] = self.label 78 | # remove repeated conditions 79 | conditions = crnt_instance[CONDS] 80 | crnt_instance[CONDS] = [dict(t) for t in {tuple(d.items()) for d in conditions}] 81 | 82 | # sort instances based on weight 83 | calc_weight(self.instances, self.concepts) 84 | self.instances.sort(key=lambda x: x[WEIGHT], reverse=True) 85 | 86 | print(self.instances) 87 | return self.instances[:15] 88 | 89 | def single_condition(self): 90 | extended_instances = [] 91 | for crnt_instance in self.instances: 92 | if len(crnt_instance[CONDS]) == 1: 93 | single_cond = crnt_instance[CONDS][0] 94 | crnt_text = list(single_cond.keys())[0] 95 | crnt_type = single_cond[crnt_text] 96 | 97 | # only process when the highlighted is token 98 | if crnt_type == KeyType[TOKEN]: 99 | # pipeline to process the crnt_text 100 | # remove stopwords and punct 101 | token_list = simple_parse(crnt_text, self.concepts) 102 | 103 | if len(token_list) == 0: 104 | return 105 | 106 | # relationship set 107 | relationship_set = Relationship(RelationshipType.SET) 108 | for crnt_token in token_list: 109 | relationship_set.add(crnt_token, None) 110 | extended_instances.extend( relationship_set.get_instances(self.concepts) ) 111 | else: 112 | for condition in crnt_instance[CONDS]: 113 | new_inst = crnt_instance.copy() 114 | new_inst[CONDS] = [condition] 115 | extended_instances.extend([new_inst]) 116 | self.instances.extend(extended_instances) 117 | 118 | 119 | def calc_weight(instances, concepts): 120 | # TODO better weight calc 121 | for crnt_instance in instances: 122 | crnt_weight = 0 123 | for crnt_cond in crnt_instance[CONDS]: 124 | k = crnt_cond.get("string") 125 | v = crnt_cond.get("type") 126 | if v == KeyType[CONCEPT]: 127 | crnt_weight += len(concepts[k]) 128 | elif v == KeyType[NER]: 129 | crnt_weight += 1 130 | crnt_instance[WEIGHT] = crnt_weight 131 | 132 | 133 | def test_synthesizer(): 134 | text_origin = "the room is clean." 135 | annotations = [{"id":5, "start_offset": 5, "end_offset":9}, {"id":12, "start_offset": 12, "end_offset":17}] 136 | label = 1 137 | de = '#' 138 | concepts = dict() 139 | 140 | concepts['Hotel'] = ['room'] 141 | 142 | crnt_syn = Synthesizer(text_origin, annotations, label, de, concepts) 143 | print(crnt_syn.run()) 144 | 145 | 146 | #test_synthesizer() 147 | -------------------------------------------------------------------------------- /server/tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/server/tests/__init__.py -------------------------------------------------------------------------------- /server/tests/test_dataset.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import os 3 | import pandas as pd 4 | import unittest 5 | 6 | from api.dataset import Dataset, DataPreparer 7 | from verifier.labeling_function import LabelingFunction 8 | 9 | 10 | class datasetTest(unittest.TestCase): 11 | 12 | def setUp(self): 13 | lfs = [ 14 | LabelingFunction(name="1", f=lambda x: 1), 15 | LabelingFunction(name="0", f=lambda x: 0), 16 | LabelingFunction(name="pos",f=lambda x: 1 if x.text=="positive" else -1), 17 | LabelingFunction(name="neg",f=lambda x: 1 if x.text=="negative" else 0) 18 | ] 19 | self.lfs = lfs 20 | 21 | sample = [ 22 | [0,"People ask me ""Who was that singing? It was amazing!""",1, 'dev'], 23 | [1,"Best purchase ever",1, 'dev'], 24 | [2,"After about a year, the batteries would not hold a charge. ",0, 'dev'], 25 | [3,"So many bugs!",0, 'dev'], 26 | [4,"What a terrible product", -1, 'train'], 27 | [5,"What a terrible product", -1, 'train'], 28 | [6,"Nice stuff they're doing these days with the innovation and such", -1, 'train'], 29 | [7,"This is not what I wanted for Christmas", -1, 'train'], 30 | ] 31 | df = pd.DataFrame(sample, columns=["idx", "text", "label", "split"]) 32 | self.dataset = Dataset(df) 33 | 34 | self.data_preparer = DataPreparer() 35 | 36 | def test_apply(self): 37 | L = self.dataset.apply_lfs(self.lfs) 38 | self.assertEqual(L.shape, (len(self.dataset), len(self.lfs))) 39 | 40 | L2 = self.dataset.apply_lfs(self.lfs) 41 | self.assertTrue((L==L2).all()) 42 | 43 | def test_save(self): 44 | path = 'datasets/sentiment_example/processed.csv' 45 | self.dataset.save(path) 46 | L = self.dataset.apply_lfs(self.lfs) 47 | d2 = Dataset.load(path) 48 | L2 = d2.apply_lfs(self.lfs) 49 | self.assertTrue((L==L2).all()) 50 | 51 | """Make sure data upload works for the formats we want to support 52 | """ 53 | def test_upload_amazon_reviews(self): 54 | sample = [ 55 | [0,"People ask me ""Who was that singing? It was amazing!""",1], 56 | [1,"Best purchase ever",1], 57 | [2,"Best purchase ever",1], 58 | [3,"Best purchase ever",1], 59 | [4,"Best purchase ever",1], 60 | [5,"Best purchase ever",1], 61 | [6,"After about a year, the batteries would not hold a charge. ",0], 62 | [7,"After about a year, the batteries would not hold a charge. ",0], 63 | [8,"After about a year, the batteries would not hold a charge. ",0], 64 | [9,"After about a year, the batteries would not hold a charge. ",0], 65 | [10,"So many bugs!",0]] 66 | 67 | df = pd.DataFrame(sample) 68 | 69 | data_preparer = DataPreparer() 70 | 71 | pdf = data_preparer.set_headers(df) 72 | df_split = data_preparer.make_splits(pdf, test_split=False) 73 | 74 | df_final = data_preparer.precompute_values(df_split) 75 | dsets = data_preparer.split(df_final) 76 | 77 | for dset in dsets.values(): 78 | dset.apply_lfs(self.lfs) 79 | self.assertEqual(len(dsets['dev']['label'].value_counts()), 2) 80 | 81 | def test_upload_youtube_comments(self): 82 | df = pd.read_csv("datasets/spam_example/example_dataset.csv", header=0) 83 | 84 | data_preparer = DataPreparer() 85 | 86 | pdf = data_preparer.set_headers(df) 87 | df_split = data_preparer.make_splits(pdf) 88 | df_final = data_preparer.precompute_values(df_split) 89 | dsets = data_preparer.split(df_final) 90 | 91 | for split in ['train', 'test', 'dev', 'valid']: 92 | dsets[split].apply_lfs(self.lfs) 93 | 94 | self.assertEqual(len(dsets['dev']['label'].value_counts()), 2) 95 | 96 | 97 | def test_upload_newsgroups(self): 98 | sample = { 99 | "data": ['My harddrive catches on fire every time a watch cat videos. Do I need to download more RAM?', 100 | 'From: Sasha \nSubject: Specialty Ammo\n Where do people buy silver bullets in bulk?', 101 | 'Send me your address and I\'ll mail you gunpowder. First ten people to comment!'], 102 | "filenames": ['/file/path/one', 103 | '/file/path/two', 104 | '/file/path/three'], 105 | "target_names": ['comp.electronics', 'politics.guns', 'politics.guns'], 106 | "target": [7, 5, 0], 107 | "DESCR": 'This is a sample in the format of the 20 newsgroups dataset' 108 | } 109 | 110 | df = pd.DataFrame(sample) 111 | 112 | data_preparer = DataPreparer() 113 | 114 | pdf = data_preparer.set_headers(df) 115 | 116 | df_split = data_preparer.make_splits(pdf, test_split=False) 117 | df_final = data_preparer.precompute_values(df_split) 118 | dsets = data_preparer.split(df_final) 119 | 120 | for dset in dsets.values(): 121 | dset.apply_lfs(self.lfs) 122 | 123 | self.assertEqual(len(df_final['label'].value_counts()), 3) 124 | 125 | 126 | if __name__ == '__main__': 127 | unittest.main() -------------------------------------------------------------------------------- /server/tests/test_project.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import unittest 3 | 4 | from api.dataset import Dataset 5 | from api.project import Project 6 | 7 | 8 | class projectTest(unittest.TestCase): 9 | def setUp(self): 10 | self.project = Project() 11 | 12 | def test_name(self): 13 | self.project.set_name("test_project") 14 | self.assertEqual(self.project.name, "test_project") 15 | 16 | def test_labels(self): 17 | empty_labels = self.project.get_labels() 18 | self.assertEqual(empty_labels, {}) 19 | 20 | labels = {"NOT_SPAM": 0, "SPAM": 1} 21 | self.project.add_labels(labels) 22 | fetched_labels = self.project.get_labels() 23 | self.assertEqual(fetched_labels, labels) 24 | 25 | def test_set_datasets(self): 26 | 27 | df = pd.read_csv('datasets/spam_example/processed.csv', header=0) 28 | dataset = Dataset(df) 29 | 30 | test_datasets_dict = { 31 | "train": dataset, 32 | "dev": dataset, 33 | "test": dataset, 34 | "valid": dataset 35 | } 36 | self.project.set_datasets(test_datasets_dict) 37 | 38 | def test_prep_datasets(self): 39 | self.project.prep_datasets('spam_example', test_split=False) 40 | 41 | def test_readiness(self): 42 | # TODO: assert labels and cardinality match 43 | # make sure that if self.project.ready() is true, all the endpoints work 44 | pass -------------------------------------------------------------------------------- /server/verifier/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/server/verifier/__init__.py -------------------------------------------------------------------------------- /server/verifier/eval_utils.py: -------------------------------------------------------------------------------- 1 | # evaluation utilities 2 | import numpy as np 3 | import pandas as pd 4 | import sys 5 | 6 | from sklearn import metrics 7 | from snorkel.utils import probs_to_preds 8 | from snorkel.labeling import filter_unlabeled_dataframe 9 | 10 | def lf_analysis(modeler, train_data, dev_data=None): 11 | # get analysis over training and dev sets 12 | if not modeler.has_lfs(): 13 | print("lf_analysis called with no LFs") 14 | return None 15 | analys_train = modeler.analyze_lfs(train_data) 16 | stats_to_include = ["Polarity", "Coverage", "Conflicts", "Overlaps"] 17 | analys_train.rename(columns = { 18 | stat: stat + " Train" for stat in stats_to_include 19 | }, inplace = True) 20 | 21 | if dev_data is not None: 22 | # if development data is available 23 | analys_dev = modeler.analyze_lfs(dev_data, labels=dev_data["label"].values) 24 | analys_dev.rename(columns = { 25 | stat: stat + " Dev." for stat in stats_to_include 26 | }, inplace = True) 27 | analys_dev.drop(["j"], axis=1, inplace=True) 28 | 29 | # Merge these analyses 30 | analysis = analys_train.merge(analys_dev, how="inner", left_index=True, right_index=True) 31 | else: 32 | analysis = analys_train 33 | 34 | # Add GLL metadata 35 | analysis = pd.concat([analysis, modeler.GLL_repr()], axis=1) 36 | 37 | # Add weights 38 | analysis['Weight'] = modeler.get_weights() 39 | 40 | # Field "LABEL" may be empty for custom functions. We can fill it using observed labels. 41 | analysis["Label"].fillna(analysis["Polarity Train"]) 42 | 43 | # Detect duplicate labeling signatures 44 | analysis["Duplicate"] = None 45 | for dupe, OG in detect_duplicate_labeling_signature(modeler, [train_data, dev_data]).items(): 46 | print("Duplicate labeling signature detected") 47 | print(dupe, OG) 48 | analysis.at[dupe, "Duplicate"] = OG 49 | 50 | return analysis 51 | 52 | def lf_examples(labeling_function, data, max_examples = 5): 53 | label_vector = apply_one_lf(labeling_function, data) 54 | examples = data[label_vector!=-1] 55 | samples = examples.sample(min(max_examples, len(examples)), random_state=13) 56 | return samples["text"].values 57 | 58 | def lf_mistakes(labeling_function, data, max_examples = 5): 59 | label_vector = apply_one_lf(labeling_function, data) 60 | examples = data[(label_vector!=-1) & (label_vector != data["label"].values)] 61 | samples = examples.sample(min(max_examples, len(examples)), random_state=13) 62 | return samples["text"].values 63 | 64 | def apply_one_lf(labeling_function, data): 65 | try: 66 | label_matrix = data.apply_lfs([labeling_function]) 67 | except AttributeError: 68 | applier = PandasLFApplier(lfs=[labeling_function]) 69 | label_matrix = applier.apply(df = data) 70 | return label_matrix 71 | 72 | def detect_duplicate_labeling_signature(modeler, datasets: list): 73 | label_matrix = np.vstack([modeler.apply(dset) for dset in datasets]) 74 | seen_signatures = {} 75 | dupes = {} 76 | lfs = modeler.get_lfs() 77 | signatures = [hash(label_matrix[:,i].tostring()) for i in range(len(lfs))] 78 | for i, s in enumerate(signatures): 79 | lf = lfs[i] 80 | if s in seen_signatures: 81 | dupes[lf.name] = seen_signatures[s] 82 | else: 83 | seen_signatures[s] = lf.name 84 | return dupes 85 | 86 | def get_label_model_stats(): 87 | pass 88 | 89 | def get_stats_from_modeler(modeler, dataset, metrics_to_use=["f1", "precision", "recall"], average='weighted'): 90 | L = modeler.apply(dataset) 91 | y = modeler.predict(dataset) 92 | 93 | df_train_filtered, probs_train_filtered = filter_unlabeled_dataframe( 94 | X=dataset.df, y=y, L=L 95 | ) 96 | coverage = len(probs_train_filtered)/len(dataset) 97 | preds = probs_to_preds(modeler.predict(dataset)) 98 | ground_truth = dataset["label"].values 99 | 100 | res = { 101 | #"accuracy": metrics.accuracy_score(ground_truth, preds), 102 | "f1": metrics.f1_score(ground_truth, preds, average=average), 103 | "precision": metrics.precision_score(ground_truth, preds, average=average), 104 | "recall": metrics.recall_score(ground_truth, preds, average=average), 105 | "coverage": coverage 106 | } 107 | return res 108 | 109 | def get_stats_from_probs(probs, ground_truth): 110 | preds = probs_to_preds(probs) 111 | return { 112 | "f1": metrics.f1_score(ground_truth, preds), 113 | "precision": metrics.precision_score(ground_truth, preds), 114 | "recall": metrics.recall_score(ground_truth, preds), 115 | } 116 | 117 | -------------------------------------------------------------------------------- /server/verifier/interaction_db.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | import json 3 | import os 4 | import pandas as pd 5 | from verifier.labeling_function import make_lf 6 | 7 | 8 | class InteractionDBSingleton: 9 | filename = "interactionDB.json" 10 | def __init__(self): 11 | self.db = {} 12 | self.count = 0 13 | 14 | def add(self, interaction: dict): 15 | if "index" in interaction: 16 | index = interaction["index"] 17 | interaction['time_submitted'] = str(datetime.now()) 18 | self.db[index].update(interaction) 19 | else: 20 | index = self.count 21 | interaction["index"] = index 22 | interaction['time_first_seen'] = str(datetime.now()) 23 | self.db[index] = interaction 24 | self.count += 1 25 | return index 26 | 27 | def update(self, index: int, selected_lf_ids: list): 28 | self.db[index]["lfs"] = selected_lf_ids 29 | 30 | def get(self, index: int): 31 | return self.db[index] 32 | 33 | def save(self, dirname): 34 | with open(os.path.join(dirname, self.filename), "w+") as file: 35 | json.dump({ 36 | "db": self.db, 37 | "count": self.count 38 | }, file, default=str) 39 | 40 | def load(self, dirname): 41 | with open(os.path.join(dirname, self.filename), "r") as file: 42 | data = json.load(file) 43 | self.db = data['db'] 44 | self.count = data['count'] 45 | 46 | interactionDB = InteractionDBSingleton() -------------------------------------------------------------------------------- /server/verifier/keraslogreg.py: -------------------------------------------------------------------------------- 1 | """END MODEL 2 | To better evaluate the quality of our generated training data, 3 | we evaluate an end model trained on this data. 4 | 5 | Here, the end model is implemented as a logistic regression bag of words model. 6 | However, you can replace this with any model of your choosing, as long as it has 7 | functions "fit" and "predict" with the specifications outlined below. 8 | """ 9 | import tensorflow as tf 10 | 11 | from numpy.random import seed as np_seed 12 | from random import seed as py_seed 13 | from snorkel.utils import set_seed as snork_seed 14 | from snorkel.utils import preds_to_probs 15 | from sklearn.feature_extraction.text import CountVectorizer 16 | 17 | 18 | 19 | def get_keras_logreg(input_dim, output_dim=2): 20 | """Create a simple logistic regression model (using keras) 21 | """ 22 | model = tf.keras.Sequential() 23 | if output_dim == 1: 24 | loss = "binary_crossentropy" 25 | activation = tf.nn.sigmoid 26 | else: 27 | loss = "categorical_crossentropy" 28 | activation = tf.nn.softmax 29 | dense = tf.keras.layers.Dense( 30 | units=output_dim, 31 | input_dim=input_dim, 32 | activation=activation, 33 | kernel_regularizer=tf.keras.regularizers.l2(0.001), 34 | ) 35 | model.add(dense) 36 | opt = tf.keras.optimizers.Adam(lr=0.01) 37 | model.compile(optimizer=opt, loss=loss, metrics=["accuracy"]) 38 | 39 | return model 40 | 41 | 42 | 43 | def get_keras_early_stopping(patience=10): 44 | """Create early stopping condition 45 | """ 46 | return tf.keras.callbacks.EarlyStopping( 47 | monitor="val_accuracy", patience=10, verbose=1, restore_best_weights=True 48 | ) 49 | 50 | 51 | class KerasLogReg: 52 | """This logistic regression model is trained on the labels that Ruler generates, and then evaluated against a test set. 53 | This provides a more complete picture of the quality of the generated training data. 54 | 55 | Attributes: 56 | cardinality (int): Number of output classes 57 | keras_model (a Keras logistic regression model): Description 58 | vectorizer (CountVectorizer): Object with fit and transform functions, which transforms texts into vectors 59 | """ 60 | 61 | def __init__(self, cardinality=2): 62 | """Summary 63 | 64 | Args: 65 | cardinality (int, optional): Number of output classes 66 | """ 67 | # Set all random seeds 68 | snork_seed(123) 69 | tf.random.set_seed(123) 70 | np_seed(123) 71 | py_seed(123) 72 | self.cardinality = cardinality 73 | self.keras_model = None 74 | 75 | def fit(self, X_train, Y_train, X_valid, Y_valid): 76 | """Train the model using the given training and validation data. 77 | 78 | Args: 79 | X_train (list(str)): Training text examples, length n 80 | Y_train (matrix): Training labels, size n*m, where m is the cardinality 81 | X_valid (list(str)): Validation test examples, length p 82 | Y_valid (matrix): Validation labels, size p*m 83 | """ 84 | if self.keras_model is None: 85 | self.vectorizer = CountVectorizer(ngram_range=(1, 2)) 86 | self.vectorizer.fit(X_train) 87 | X_train = self.vectorizer.transform(X_train) 88 | X_valid = self.vectorizer.transform(X_valid) 89 | if self.keras_model is None: 90 | self.keras_model = get_keras_logreg(input_dim=X_train.shape[1], output_dim=self.cardinality) 91 | self.keras_model.fit( 92 | x=X_train, 93 | y=Y_train, 94 | validation_data=(X_valid, Y_valid), 95 | callbacks=[get_keras_early_stopping()], 96 | epochs=20, 97 | verbose=0, 98 | ) 99 | 100 | def predict(self, X): 101 | """Predict probabilities that each sample in X belongs to each class. 102 | 103 | Args: 104 | X (list(str)): Texts to predict class, length n 105 | 106 | Returns: 107 | matrix: size n*m, where m is the cardinality of the model 108 | """ 109 | X_v = self.vectorizer.transform(X) 110 | return self.keras_model.predict(x=X_v) 111 | 112 | -------------------------------------------------------------------------------- /server/verifier/labeling_function.py: -------------------------------------------------------------------------------- 1 | import inspect 2 | import marshal 3 | import types 4 | 5 | from datetime import datetime 6 | from snorkel.labeling import LabelingFunction as SnorkelLF 7 | from synthesizer.gll import CONDS 8 | from synthesizer.gll import CONNECTIVE 9 | from synthesizer.gll import DIRECTION 10 | from synthesizer.gll import LABEL 11 | from synthesizer.gll import WEIGHT 12 | from verifier.translator import to_lf 13 | 14 | 15 | def make_lf(instance, concepts): 16 | """wrapper to return a LabelingFunction object from an explanation dictionary 17 | 18 | Args: 19 | instance (dict): describes the LABEL, DIRECTION, CONNECTIVE, and CONDITIONS of the rule 20 | concepts (dict): all the defined concepts 21 | 22 | Returns: 23 | LabelingFunction 24 | """ 25 | try: 26 | concepts = concepts.get_dict() 27 | except AttributeError: 28 | pass 29 | return LabelingFunction( 30 | name=lf_to_hash(instance), 31 | f=to_lf, 32 | resources=dict(instance=instance, concepts=concepts), 33 | ) 34 | 35 | def lf_to_hash(lf_dict): 36 | """Create a unique string representing an LF 37 | 38 | Args: 39 | lf_dict (dict): A labeling function (LF) 40 | 41 | Returns: 42 | str: unique hash 43 | """ 44 | def conditions_to_string(cond_list): 45 | conds = [] 46 | for x in cond_list: 47 | conds.append("".join([str(val) for val in x.values()])) 48 | return "-".join(sorted(conds)) 49 | lf_hash = "" 50 | for key, value in sorted(lf_dict.items()): 51 | if key == "Conditions": 52 | lf_hash += conditions_to_string(value) 53 | else: 54 | if key in ["Connective", "Direction", "Label", "Context"]: 55 | lf_hash += "_" + str(value) 56 | return lf_hash 57 | 58 | class LabelingFunction(SnorkelLF): 59 | 60 | """Summary 61 | 62 | Attributes: 63 | active (bool): if the LF is active (If it's inactive we still save it, just don't apply to a dataset) 64 | as_string (str): if it's a custom function, the string definition of the function. 65 | GLL_fields (list): names of required fields for a Generalized Labeling Language function 66 | stats (dict): most recently computed statistics (like coverage, accuracy, conflict) 67 | time_created (datetime): when it was created 68 | time_submitted (datetime): when it was submitted to the modeler 69 | uuid (str): UUID identifier. This should change whenever a change occurs that will affect the function's labeling signature. 70 | Currently, only updating a concept will do this. 71 | name (str): Unique name for this function. The name will never change over the lifetime of the function 72 | """ 73 | 74 | def __init__(self, 75 | name, # a unique name is required 76 | f, 77 | as_string=None, 78 | active = True, 79 | stats = {}, 80 | time_created = datetime.now(), 81 | **kwargs): 82 | super().__init__(name=name, f=f, **kwargs) 83 | 84 | # TODO hard coding these is a little hacky 85 | self.GLL_fields = [CONDS, CONNECTIVE, DIRECTION, LABEL, WEIGHT] 86 | 87 | self.time_created = time_created 88 | self.active = active 89 | self.stats = stats 90 | self.as_string = as_string 91 | 92 | # Set UUID. Unlike name, this changes when the function updates (ex: concept changes) 93 | self.uuid = name 94 | 95 | def new_uuid(self): 96 | self.uuid = self.uuid + "_i" 97 | 98 | def submit(self): 99 | self.time_submitted = datetime.now() 100 | 101 | def activate(self): 102 | self.active = True 103 | 104 | def deactivate(self): 105 | self.active = False 106 | 107 | def update(self, stats): 108 | self.stats.update(stats) 109 | 110 | def to_json(self): 111 | obj = { 112 | "name": self.name, 113 | "resources": self._resources, 114 | # TODO make sure all resources are json serializable 115 | "active": self.active, 116 | "stats": self.stats, 117 | "time_created": str(self.time_created) 118 | } 119 | return obj 120 | 121 | def has_GLL_repr(self): 122 | return "instance" in self._resources 123 | 124 | def GLL_repr(self): 125 | """Get a human-readable representation of this LF. 126 | If it's Ruler-generated, it has a GLL (Generalized Labeling Language) representation, 127 | otherwise, there is a field "as_string" showing the function definition as a string. 128 | """ 129 | rep = {} 130 | if self.has_GLL_repr(): 131 | rep = self._resources['instance'] 132 | else: 133 | rep = {field: None for field in self.GLL_fields} 134 | 135 | # See if we can observe the output label(s) 136 | if "Polarity Train" in self.stats: 137 | rep[LABEL] = self.stats["Polarity Train"] 138 | 139 | # add string representation of function 140 | if self.as_string is not None: 141 | rep["as_string"] = self.as_string 142 | rep['active'] = self.active 143 | return rep 144 | 145 | @staticmethod 146 | def read_json(json_obj): 147 | name = json_obj.pop('name') 148 | return LabelingFunction(name=name, f=to_lf, **json_obj) 149 | -------------------------------------------------------------------------------- /server/verifier/translator.py: -------------------------------------------------------------------------------- 1 | """Translate a dictionary explaining a labeling rule into a function 2 | """ 3 | import re 4 | import sys 5 | 6 | from synthesizer.gll import * 7 | from synthesizer.parser import nlp 8 | 9 | def raw_stringify(s): 10 | """From a regex create a regular expression that finds occurences of the string as entire words 11 | 12 | Args: 13 | s (string): the string to look for 14 | 15 | Returns: 16 | string: a regular expession that looks for this string surrounded by non word characters 17 | """ 18 | return "(?:(?<=\W)|(?<=^))({})(?=\W|$)".format(re.escape(s)) 19 | 20 | def find_indices(cond_dict: dict, text: str): 21 | """Find all instances of this condition in the text 22 | """ 23 | v = cond_dict["type"] 24 | k = cond_dict["string"] 25 | case_sensitive = True if cond_dict.get("case_sensitive") else False 26 | 27 | if v == KeyType[NER]: 28 | doc = nlp(text) 29 | for ent in doc.ents: 30 | if ent.label_ == k: 31 | return [(doc[ent.start].idx, doc[ent.end-1].idx + len(doc[ent.end-1].text))] 32 | return [] 33 | if case_sensitive: 34 | return [(m.start(), m.end()) for m in re.finditer(k, text)] 35 | else: 36 | return [(m.start(), m.end()) for m in re.finditer(k, text, re.IGNORECASE)] 37 | 38 | def to_lf(x, instance, concepts): 39 | """from one instance (in dict) to one labeling function 40 | 41 | Args: 42 | x: Object with an attribute "text", which is a string 43 | instance (dict): describes the LABEL, DIRECTION, CONNECTIVE, and CONDITIONS of the rule 44 | concepts (dict): all the defined concepts 45 | 46 | Returns: 47 | int: the label assigned by this function 48 | """ 49 | label = instance.get(LABEL) 50 | direction = bool(instance.get(DIRECTION)) 51 | conn = instance.get(CONNECTIVE) 52 | conds = instance.get(CONDS) 53 | context = instance.get(CONTEXT) 54 | 55 | text = x.text 56 | 57 | matched_idxes = [] 58 | 59 | for crnt_cond in conds: 60 | # crnt_cond is a dict 61 | v = crnt_cond["type"] 62 | k = crnt_cond["string"] 63 | assert(v in KeyType.values()) 64 | idx = [] 65 | 66 | # if current condition is a named entity type, see if it's pre-computed 67 | if v == KeyType[NER]: 68 | try: 69 | if x[k]: 70 | idx.extend([x[k]]) 71 | except (KeyError, TypeError): 72 | idx.extend(find_indices(crnt_cond, text)) 73 | elif v == KeyType[CONCEPT]: 74 | # match a concept 75 | assert k in concepts, "{} has no element {}".format(concepts, k) 76 | try: 77 | print(concepts.get_dict()) 78 | except AttributeError: 79 | pass 80 | for crnt_token in concepts[k]: 81 | idx.extend(find_indices(crnt_token, text)) 82 | else: 83 | idx.extend(find_indices(crnt_cond, text)) 84 | 85 | matched_idxes.append(idx) 86 | 87 | if (conn == ConnectiveType[AND]) and (len(idx)==0): 88 | return -1 89 | 90 | 91 | if not direction: 92 | if conn == ConnectiveType[OR]: 93 | # no direction, OR 94 | if any(len(m) > 0 for m in matched_idxes): 95 | return label 96 | else: 97 | # no direction, AND 98 | if all(len(m) > 0 for m in matched_idxes): 99 | if conn == ConnectiveType[AND]: 100 | if conn == ConnectiveType[AND]: 101 | doc = nlp(text) 102 | if (len(list(doc.sents))==1): 103 | return label 104 | for sent in doc.sents: 105 | if to_lf(sent, instance, concepts): 106 | return label 107 | else: 108 | return label 109 | else: 110 | assert len(matched_idxes) == 2 111 | if all(len(m) > 0 for m in matched_idxes): 112 | min_idx = matched_idxes[0][0][0] 113 | max_idx = matched_idxes[1][-1][0] 114 | 115 | if min_idx < max_idx: 116 | if conn == ConnectiveType[AND]: 117 | if context == ContextType[SENTENCE]: 118 | doc = nlp(text) 119 | if (len(list(doc.sents))==1): 120 | return label 121 | for sent in doc.sents: 122 | if to_lf(sent, instance, concepts): 123 | return label 124 | else: 125 | return label 126 | return -1 127 | 128 | -------------------------------------------------------------------------------- /ui/.env: -------------------------------------------------------------------------------- 1 | DOMAIN=http://localhost:5000 2 | REACT_APP_SERVER=$DOMAIN/api 3 | REACT_APP_API_UI=$DOMAIN/api/ui -------------------------------------------------------------------------------- /ui/.gitignore: -------------------------------------------------------------------------------- 1 | # See https://help.github.com/articles/ignoring-files/ for more about ignoring files. 2 | 3 | venv/* 4 | # dependencies 5 | /node_modules 6 | /.pnp 7 | .pnp.js 8 | 9 | # testing 10 | /coverage 11 | 12 | # production 13 | /build 14 | 15 | # misc 16 | .DS_Store 17 | .env.local 18 | .env.development.local 19 | .env.test.local 20 | .env.production.local 21 | 22 | npm-debug.log* 23 | yarn-debug.log* 24 | yarn-error.log* 25 | -------------------------------------------------------------------------------- /ui/.idea/.gitignore: -------------------------------------------------------------------------------- 1 | # Default ignored files 2 | /workspace.xml -------------------------------------------------------------------------------- /ui/.idea/dpbd.iml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /ui/.idea/inspectionProfiles/Project_Default.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 6 | -------------------------------------------------------------------------------- /ui/.idea/misc.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 6 | -------------------------------------------------------------------------------- /ui/.idea/modules.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | -------------------------------------------------------------------------------- /ui/.idea/vcs.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /ui/README.md: -------------------------------------------------------------------------------- 1 | 2 | The UI for data programming by demonstration (DPBD) 3 | 4 | ## Available Scripts 5 | 6 | In the project directory, you can run: 7 | 8 | ### `npm start` 9 | 10 | Runs the app in the development mode.
11 | Open [http://localhost:3000](http://localhost:3000) to view it in the browser. 12 | 13 | The page will reload if you make edits.
14 | You will also see any lint errors in the console. 15 | 16 | ### `npm test` 17 | 18 | Launches the test runner in the interactive watch mode.
19 | See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information. 20 | 21 | ### `npm run build` 22 | 23 | Builds the app for production to the `build` folder.
24 | It correctly bundles React in production mode and optimizes the build for the best performance. 25 | 26 | The build is minified and the filenames include the hashes.
27 | Your app is ready to be deployed! 28 | 29 | See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information. 30 | 31 | ### `npm run eject` 32 | 33 | **Note: this is a one-way operation. Once you `eject`, you can’t go back!** 34 | 35 | If you aren’t satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project. 36 | 37 | Instead, it will copy all the configuration files and the transitive dependencies (Webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own. 38 | 39 | You don’t have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it. 40 | 41 | ## Learn More 42 | 43 | You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started). 44 | 45 | To learn React, check out the [React documentation](https://reactjs.org/). 46 | 47 | ### Code Splitting 48 | 49 | This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting 50 | 51 | ### Analyzing the Bundle Size 52 | 53 | This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size 54 | 55 | ### Making a Progressive Web App 56 | 57 | This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app 58 | 59 | ### Advanced Configuration 60 | 61 | This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration 62 | 63 | ### Deployment 64 | 65 | This section has moved here: https://facebook.github.io/create-react-app/docs/deployment 66 | 67 | ### `npm run build` fails to minify 68 | 69 | This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify 70 | -------------------------------------------------------------------------------- /ui/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "dpbd", 3 | "version": "0.1.0", 4 | "private": true, 5 | "dependencies": { 6 | "@material-ui/core": "^4.4.0", 7 | "@material-ui/icons": "^4.2.1", 8 | "@material-ui/styles": "^4.0.0-beta.1", 9 | "axios": "^0.19.0", 10 | "clsx": "^1.0.4", 11 | "d3": "^5.12.0", 12 | "file-saver": "^2.0.2", 13 | "jquery": "^3.4.0", 14 | "material-ui-dropzone": "^3.2.1", 15 | "prop-types": "^15.7.2", 16 | "react": "^16.9.0", 17 | "react-dom": "^16.9.0", 18 | "react-redux": "^7.1.1", 19 | "react-router-dom": "^5.2.0", 20 | "react-scripts": "3.1.1", 21 | "redux": "^4.0.4", 22 | "redux-thunk": "^2.3.0", 23 | "typeface-roboto": "0.0.75" 24 | }, 25 | "scripts": { 26 | "start": "react-scripts start", 27 | "build": "react-scripts build", 28 | "test": "react-scripts test", 29 | "eject": "react-scripts eject" 30 | }, 31 | "eslintConfig": { 32 | "extends": "react-app" 33 | }, 34 | "browserslist": { 35 | "production": [ 36 | ">0.2%", 37 | "not dead", 38 | "not op_mini all" 39 | ], 40 | "development": [ 41 | "last 1 chrome version", 42 | "last 1 firefox version", 43 | "last 1 safari version" 44 | ] 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /ui/public/favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/megagonlabs/ruler/57112a414165a92ae56d145ee9f6a301c32e0765/ui/public/favicon.png -------------------------------------------------------------------------------- /ui/public/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 12 | 16 | 17 | 26 | Data Programming By Demonstration 27 | 28 | 29 | 30 |
31 | 41 | 42 | 43 | -------------------------------------------------------------------------------- /ui/public/manifest.json: -------------------------------------------------------------------------------- 1 | { 2 | "short_name": "Ruler", 3 | "name": "Ruler: Data Programming by Demonstration for Text", 4 | "icons": [ 5 | { 6 | "src": "favicon.png", 7 | "sizes": "64x64", 8 | "type": "image/x-png" 9 | } 10 | ], 11 | "start_url": ".", 12 | "display": "standalone", 13 | "theme_color": "#000000", 14 | "background_color": "#ffffff" 15 | } 16 | -------------------------------------------------------------------------------- /ui/public/robots.txt: -------------------------------------------------------------------------------- 1 | # https://www.robotstxt.org/robotstxt.html 2 | User-agent: * 3 | -------------------------------------------------------------------------------- /ui/src/AnnotationDisplayCollapse.js: -------------------------------------------------------------------------------- 1 | import React from "react"; 2 | import PropTypes from "prop-types"; 3 | 4 | import AnnotationDisplay from "./AnnotationDisplay"; 5 | import ExpandMoreIcon from "@material-ui/icons/ExpandMore"; 6 | import ExpandLessIcon from "@material-ui/icons/ExpandLess"; 7 | import Grid from "@material-ui/core/Grid"; 8 | 9 | class AnnotationDisplayCollapse extends React.Component { 10 | constructor(props) { 11 | super(props); 12 | this.state = { 13 | open: false, 14 | }; 15 | this.collapseText(this.props.text); 16 | } 17 | 18 | collapseText(text, MAX_COLLAPSED_MARGIN = 100) { 19 | const max_start = this.props.annotations[0] 20 | ? this.props.annotations[0].start_offset 21 | : 0; 22 | const min_end = this.props.annotations[0] 23 | ? this.props.annotations[0].end_offset 24 | : 0; 25 | 26 | var coll_annotations = this.props.annotations; 27 | if (max_start > MAX_COLLAPSED_MARGIN) { 28 | const start = text.indexOf(" ", max_start - MAX_COLLAPSED_MARGIN); 29 | if (start > 30) { 30 | coll_annotations = this.props.annotations.map((ann) => { 31 | return { 32 | ...ann, 33 | start_offset: ann.start_offset - start + 3, 34 | end_offset: ann.end_offset - start + 3, 35 | }; 36 | }); 37 | text = "..." + text.slice(start); 38 | } 39 | } 40 | 41 | if (min_end < text.length - MAX_COLLAPSED_MARGIN) { 42 | const end = text.lastIndexOf(" ", min_end + MAX_COLLAPSED_MARGIN); 43 | if (text.length - end > 30) { 44 | text = text.slice(0, end) + "..."; 45 | } 46 | } 47 | 48 | this.coll_text = text; 49 | this.coll_annotations = coll_annotations; 50 | } 51 | 52 | toggleOpen() { 53 | this.setState({ 54 | open: !this.state.open, 55 | }); 56 | } 57 | 58 | toggleIcon() { 59 | if (this.state.open) { 60 | return ; 61 | } else { 62 | return ; 63 | } 64 | } 65 | 66 | render() { 67 | var collapsedText = this.coll_text; 68 | var coll_annotations = this.coll_annotations; 69 | 70 | return ( 71 | 72 | 73 | 82 | 83 | 84 | {this.props.text !== collapsedText 85 | ? this.toggleIcon() 86 | : null} 87 | 88 | 89 | ); 90 | } 91 | } 92 | 93 | AnnotationDisplayCollapse.propTypes = { 94 | text: PropTypes.string, 95 | annotations: PropTypes.array, 96 | }; 97 | 98 | export default AnnotationDisplayCollapse; 99 | -------------------------------------------------------------------------------- /ui/src/App.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import { createMuiTheme } from '@material-ui/core/styles'; 3 | import { makeStyles } from '@material-ui/core/styles'; 4 | import { ThemeProvider } from '@material-ui/styles'; 5 | import ColorPalette from './ColorPalette'; 6 | import Main from './Main'; 7 | 8 | const windowHeight = Math.max( 9 | document.documentElement.clientHeight, 10 | window.innerHeight || 0); 11 | 12 | const useStyles = makeStyles(theme => ({ 13 | root: { 14 | width: '100%', 15 | height: windowHeight, 16 | } 17 | })); 18 | 19 | 20 | const App = () => { 21 | const theme = createMuiTheme(ColorPalette), 22 | classes = useStyles(); 23 | 24 | return ( 25 | 26 | 27 |
28 |
29 |
30 | 31 | ); 32 | }; 33 | 34 | export default App; 35 | -------------------------------------------------------------------------------- /ui/src/App.test.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import ReactDOM from 'react-dom'; 3 | import App from './App'; 4 | 5 | it('renders without crashing', () => { 6 | const div = document.createElement('div'); 7 | ReactDOM.render(, div); 8 | ReactDOM.unmountComponentAtNode(div); 9 | }); 10 | -------------------------------------------------------------------------------- /ui/src/ClassLabelsCollection.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import Button from '@material-ui/core/Button'; 3 | import ButtonGroup from '@material-ui/core/ButtonGroup'; 4 | import {connect} from "react-redux"; 5 | 6 | class ClassLabelsCollection extends React.Component { 7 | constructor(props) { 8 | super(props) 9 | this.keyHandling = this.keyHandling.bind(this); 10 | this.assignLabel = this.assignLabel.bind(this); 11 | } 12 | 13 | defaultSelections 14 | 15 | keyHandling(e) { 16 | const key = parseInt(e.key); 17 | if (key in this.props.hotKeys) { 18 | this.assignLabel(key); 19 | } 20 | } 21 | 22 | assignLabel(key) { 23 | this.props.onClick(key); 24 | } 25 | 26 | componentDidMount() { 27 | window.addEventListener("keyup", this.keyHandling); 28 | } 29 | 30 | componentWillUnmount() { 31 | window.removeEventListener("keyup", this.keyHandling); 32 | } 33 | 34 | render(){ 35 | const classes = this.props.classes; 36 | return ( 37 | 38 | { Object.entries(this.props.labelClasses).map( (labelClass, key) => 39 | 48 | ) 49 | } 50 | ) 51 | } 52 | } 53 | 54 | function mapStateToProps(state, ownProps?) { 55 | return { 56 | labelClasses: state.labelClasses.data, 57 | hotKeys: Object.values(state.labelClasses.data).map(lClass => lClass.key), 58 | annotations: state.annotations, 59 | label: state.label 60 | }; 61 | } 62 | function mapDispatchToProps(state){ 63 | return {}; 64 | } 65 | 66 | export default connect(mapStateToProps, mapDispatchToProps)(ClassLabelsCollection); -------------------------------------------------------------------------------- /ui/src/ColorPalette.js: -------------------------------------------------------------------------------- 1 | // 2 | // Labeler color theme 3 | // 4 | const ColorPalette = { 5 | props: { 6 | MuiGrid: { 7 | spacing: 3 8 | } 9 | }, 10 | 11 | "palette": { 12 | "common": { 13 | "black": "#000", 14 | "white": "#fff" 15 | }, 16 | "background": { 17 | "paper": "#fff", 18 | "default": "#fafafa" 19 | }, 20 | "primary": { 21 | "light": "rgba(150, 150, 150, 1)", 22 | "main": "rgba(50, 50, 50, 1)", 23 | "dark": "rgba(0, 0, 0, 1)", 24 | "contrastText": "#fff" 25 | }, 26 | "secondary": { 27 | "light": "rgba(107, 174, 214, 1)", 28 | "main": "rgba(33, 113, 181, 1)", 29 | "dark": "rgba(8, 81, 156, 1)", 30 | "contrastText": "#fff" 31 | }, 32 | "error": { 33 | "light": "#e57373", 34 | "main": "#f44336", 35 | "dark": "#d32f2f", 36 | "contrastText": "#fff" 37 | }, 38 | "warning": { 39 | "light": "#e57373", 40 | "main": "#f44336", 41 | "dark": "#d32f2f", 42 | "contrastText": "#fff" 43 | }, 44 | "success": { 45 | "light": "rgba(150, 150, 150, 1)", 46 | "main": "rgba(50, 50, 50, 1)", 47 | "dark": "rgba(0, 0, 0, 1)", 48 | "contrastText": "#fff" 49 | }, 50 | "info": { 51 | "light": "rgba(107, 174, 214, 1)", 52 | "main": "rgba(33, 113, 181, 1)", 53 | "dark": "rgba(8, 81, 156, 1)", 54 | "contrastText": "#fff" 55 | }, 56 | "text": { 57 | "primary": "rgba(0, 0, 0, 0.87)", 58 | "secondary": "rgba(0, 0, 0, 0.54)", 59 | "disabled": "rgba(0, 0, 0, 0.38)", 60 | "hint": "rgba(0, 0, 0, 0.38)" 61 | } 62 | } 63 | }; 64 | 65 | export default ColorPalette; -------------------------------------------------------------------------------- /ui/src/ConceptCollection.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import PropTypes from 'prop-types'; 3 | import {conceptEditors, select_concept} from './actions/concepts' 4 | import { highlight } from './actions/annotate' 5 | import {bindActionCreators} from 'redux'; 6 | import {connect} from "react-redux"; 7 | 8 | import AddIcon from '@material-ui/icons/Add'; 9 | import Box from '@material-ui/core/Box'; 10 | import Button from '@material-ui/core/Button'; 11 | import Grid from '@material-ui/core/Grid'; 12 | import MenuItem from '@material-ui/core/MenuItem'; 13 | import Select from '@material-ui/core/Select'; 14 | import Typography from '@material-ui/core/Typography'; 15 | 16 | import Concept from './Concept' 17 | 18 | import { TOKEN_VIEW, REGEX_VIEW } from './ConceptElement' 19 | 20 | 21 | import TextField from '@material-ui/core/TextField'; 22 | 23 | 24 | class ConceptCollection extends React.Component { 25 | constructor(props) { 26 | super(props); 27 | 28 | this.state = { 29 | conceptName: "", 30 | fetched: false, 31 | view: TOKEN_VIEW, //view 0 is token view, view 1 is regex view 32 | openConcept: null, 33 | }; 34 | this.handleInput = this.handleInput.bind(this); 35 | this.handleKeyPress = this.handleKeyPress.bind(this); 36 | this.handleAdd = this.handleAdd.bind(this); 37 | this.changeView = this.changeView.bind(this); 38 | this.openConcept = this.openConcept.bind(this); 39 | this.closeConcept = this.closeConcept.bind(this); 40 | 41 | } 42 | 43 | componentDidMount() { 44 | if (!this.state.fetched) { 45 | this.props.conceptEditors.fetchConcepts(); 46 | this.setState({fetched: true}); 47 | } 48 | const childRef = this.props.childRef; 49 | childRef(this); 50 | } 51 | componentWillUnmount() { 52 | const childRef = this.props.childRef; 53 | childRef(undefined); 54 | } 55 | 56 | openConcept(cname) { 57 | this.resetHighlights(); 58 | this.setState({openConcept: cname}); 59 | } 60 | 61 | closeConcept() { 62 | this.resetHighlights(); 63 | this.setState({openConcept: null}); 64 | } 65 | 66 | resetHighlights() { 67 | this.props.highlight([]); 68 | } 69 | 70 | changeView(new_view) { 71 | this.setState({...this.state, view: new_view}) 72 | } 73 | 74 | handleInput(event) { 75 | this.setState({ 76 | conceptName: event.target.value 77 | }); 78 | } 79 | 80 | handleAdd() { 81 | const newConcept = this.state.conceptName.trim() 82 | if (newConcept !== "") { 83 | this.setState({conceptName: ""}); 84 | this.props.conceptEditors.addConcept(newConcept); 85 | this.props.select_concept(newConcept); 86 | this.openConcept(newConcept); 87 | } 88 | } 89 | 90 | handleKeyPress(event){ 91 | if (event.key === 'Enter') { 92 | this.handleAdd(); 93 | } 94 | } 95 | 96 | render() { 97 | const conceptNames = Object.keys(this.props.concepts); 98 | const classes = this.props.classes; 99 | 100 | return ( 101 | 102 | 103 | Concepts 104 | 108 | 109 |
110 | 111 | { conceptNames.map( (concept) => 112 | this.openConcept(concept)} close={this.closeConcept} isOpen={this.state.openConcept===concept} mustClose={this.props.closeAll} view={this.state.view} key={concept} concept={concept} classes={classes} addAnnotations={this.props.addAnnotations}/> 113 | ) 114 | } 115 | 116 | 128 | 129 | }} 130 | /> 131 | 132 | 133 | 134 | ); 135 | } 136 | } 137 | 138 | ConceptCollection.propTypes = { 139 | addAnnotations: PropTypes.func, 140 | classes: PropTypes.object.isRequired, 141 | conceptEditors: PropTypes.object.isRequired 142 | }; 143 | 144 | function mapStateToProps(state, ownProps?) { 145 | return { concepts: state.concepts.data }; 146 | } 147 | 148 | function mapDispatchToProps(dispatch) { 149 | return { 150 | conceptEditors: bindActionCreators(conceptEditors, dispatch), 151 | select_concept: bindActionCreators(select_concept, dispatch) , 152 | highlight: bindActionCreators(highlight, dispatch) 153 | }; 154 | } 155 | 156 | export default connect(mapStateToProps, mapDispatchToProps)(ConceptCollection); 157 | -------------------------------------------------------------------------------- /ui/src/Footer.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import { makeStyles } from '@material-ui/core/styles'; 3 | import FavoriteIcon from '@material-ui/icons/Favorite'; 4 | import MegagonIcon from './MegagonIcon'; 5 | import GithubIcon from './GithubIcon'; 6 | import Toolbar from '@material-ui/core/Toolbar'; 7 | import clsx from 'clsx'; 8 | import IconButton from "@material-ui/core/IconButton"; 9 | 10 | const drawerWidth = 200; 11 | 12 | const useStyles = makeStyles(theme => ({ 13 | root: { 14 | position:'fixed', 15 | bottom:0, 16 | left:0, 17 | width:'100%', 18 | justifyContent: 'center', 19 | alignItems: 'center', 20 | background:"rgba(50, 50, 50, 1)" 21 | }, 22 | footer: { 23 | transition: theme.transitions.create(['margin', 'width'], { 24 | easing: theme.transitions.easing.sharp, 25 | duration: theme.transitions.duration.leavingScreen, 26 | }), 27 | }, 28 | footerShift: { 29 | width: `calc(100% - ${drawerWidth}px)`, 30 | marginLeft: drawerWidth, 31 | transition: theme.transitions.create(['margin', 'width'], { 32 | easing: theme.transitions.easing.easeOut, 33 | duration: theme.transitions.duration.enteringScreen, 34 | }) 35 | }, 36 | menuButton: { 37 | marginRight: theme.spacing(2), 38 | }, 39 | icon:{ 40 | color:'white' 41 | } 42 | 43 | })); 44 | 45 | 46 | function Footer(props) { 47 | 48 | const classes = useStyles(), 49 | isDrawerOpen = props.isDrawerOpen; 50 | 51 | function handleClick(url) { 52 | const win = window.open(url, '_blank'); 53 | win.focus(); 54 | } 55 | 56 | return ( 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | ); 69 | } 70 | 71 | export default Footer; 72 | -------------------------------------------------------------------------------- /ui/src/GithubIcon.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import SvgIcon from '@material-ui/core/SvgIcon'; 3 | 4 | const GithubIcon = props => ( 5 | 6 | 9 | 10 | ); 11 | 12 | export default GithubIcon; 13 | -------------------------------------------------------------------------------- /ui/src/LabelingFunctionsSuggested.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import {connect} from "react-redux"; 3 | import {bindActionCreators} from 'redux'; 4 | import PropTypes from 'prop-types'; 5 | 6 | import Checkbox from '@material-ui/core/Checkbox'; 7 | import InfoIcon from '@material-ui/icons/Info'; 8 | import Paper from '@material-ui/core/Paper'; 9 | import Table from '@material-ui/core/Table'; 10 | import TableBody from '@material-ui/core/TableBody'; 11 | import TableCell from '@material-ui/core/TableCell'; 12 | import TableHead from '@material-ui/core/TableHead'; 13 | import TableRow from '@material-ui/core/TableRow'; 14 | import Typography from '@material-ui/core/Typography'; 15 | import WarningIcon from '@material-ui/icons/Warning'; 16 | 17 | import { set_selected_LF } from './actions/labelAndSuggestLF' 18 | 19 | class LabelingFunctionsSuggested extends React.Component { 20 | constructor(props) { 21 | super(props); 22 | this.state = { 23 | all_selected: false 24 | }; 25 | 26 | } 27 | 28 | 29 | componentDidUpdate(prevProps) { 30 | if (this.state.all_selected) { 31 | for (var i = Object.values(this.props.labelingFunctions).length - 1; i >= 0; i--) { 32 | let lf = Object.values(this.props.labelingFunctions)[i]; 33 | if (lf.selected !== true) { 34 | this.setState({all_selected: false}) 35 | } 36 | } 37 | } 38 | } 39 | 40 | label(lf) { 41 | return ( 42 | Object.keys(this.props.labelClasses) 43 | .filter(c => this.props.labelClasses[c] === lf.Label)[0] 44 | ); 45 | } 46 | 47 | conditionToString(condition) { 48 | let string = condition["string"]; 49 | if (condition["case_sensitive"]) { 50 | string = ""+string+""; 51 | } 52 | if (condition.type === this.props.keyType["TOKEN"]){ 53 | return "\"" + string + "\"" 54 | } 55 | return string + " (" + condition.TYPE_ + ")"; 56 | } 57 | 58 | conditions(lf) { 59 | const conditions = lf.Conditions.map(cond => this.conditionToString(cond)); 60 | if (conditions.length > 1) { 61 | return ( 62 | conditions.join(" " + lf.CONNECTIVE_ + " ") 63 | ); 64 | } 65 | return conditions.join(''); 66 | } 67 | 68 | LFtoStrings(key, lf) { 69 | const stringsDict = { 70 | id: key, 71 | conditions: this.conditions(lf), 72 | context: lf.CONTEXT_, 73 | label: this.label(lf), 74 | order: lf.Direction.toString(), 75 | weight: lf.Weight 76 | }; 77 | return stringsDict; 78 | } 79 | 80 | selectAllLF(bool_selected) { 81 | // (de)select all LFs, depending on value of bool_selected 82 | const LF_names = Object.keys(this.props.labelingFunctions); 83 | 84 | let newLFs = {}; 85 | for (var i = LF_names.length - 1; i >= 0; i--) { 86 | let LF_key = LF_names[i]; 87 | newLFs[LF_key] = this.props.labelingFunctions[LF_key]; 88 | newLFs[LF_key]['selected'] = bool_selected; 89 | } 90 | 91 | this.setState({all_selected: bool_selected}); 92 | this.props.set_selected_LF(newLFs); 93 | } 94 | 95 | handleChange(name, event) { 96 | let updatedLF = this.props.labelingFunctions[name]; 97 | updatedLF['selected'] = !(updatedLF['selected']); 98 | const newLFs = { 99 | ...this.props.labelingFunctions, 100 | [name]: updatedLF 101 | }; 102 | this.props.set_selected_LF(newLFs); 103 | } 104 | 105 | render() { 106 | const classes = this.props.classes; 107 | 108 | var show_context = false; 109 | const LFList = Object.keys(this.props.labelingFunctions).map((lf_key) => { 110 | var lf_dict = this.LFtoStrings(lf_key, this.props.labelingFunctions[lf_key]) 111 | if (lf_dict.context) { 112 | show_context = true; 113 | } 114 | return lf_dict; 115 | }); 116 | 117 | var LF_content = 118 | 119 | 120 | 121 | this.selectAllLF(!this.state.all_selected)} 123 | checked={this.state.all_selected} 124 | /> 125 | { this.state.all_selected ? "Deselect All" : "Select All"} 126 | 127 | Conditions 128 | { show_context ? Context : null} 129 | Assign Label 130 | 131 | 132 | 133 | {LFList.map(row => ( 134 | 135 | 136 | this.handleChange(row.id, event)} 139 | checked={this.props.labelingFunctions[row.id].selected===true}/> 140 | 141 | {row.conditions} 142 | { show_context ? {row.context} : null} 143 | {/*{row.order}*/} 144 | {row.label} 145 | 146 | ))} 147 | 148 |
149 | 150 | return( 151 | 152 | 153 | Suggested Labeling Functions 154 | 155 | { this.props.no_label ? {"You must assign a label in order to generate labeling functions!"} : "" } 156 | { (this.props.no_annotations && !(this.props.no_label)) ? {"TIP: to improve function suggestions, annotate the parts of the text that guided your decision."} : "" } 157 | {LF_content} 158 | 159 | ); 160 | } 161 | } 162 | 163 | LabelingFunctionsSuggested.propTypes = { 164 | all_selected: PropTypes.bool 165 | }; 166 | 167 | function mapStateToProps(state, ownProps?) { 168 | 169 | return { 170 | labelingFunctions: state.suggestedLF, 171 | labelClasses:state.labelClasses.data, 172 | no_annotations: (state.annotations.length < 1), 173 | no_label: (state.label === null), 174 | keyType: state.gll.keyType 175 | }; 176 | } 177 | 178 | function mapDispatchToProps(dispatch) { 179 | return { 180 | set_selected_LF: bindActionCreators(set_selected_LF, dispatch) 181 | }; 182 | } 183 | 184 | export default connect(mapStateToProps, mapDispatchToProps)(LabelingFunctionsSuggested); -------------------------------------------------------------------------------- /ui/src/LeftDrawer.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import Drawer from '@material-ui/core/Drawer'; 3 | import List from '@material-ui/core/List'; 4 | import Divider from '@material-ui/core/Divider'; 5 | import IconButton from '@material-ui/core/IconButton'; 6 | import ChevronLeftIcon from '@material-ui/icons/ChevronLeft'; 7 | import ChevronRightIcon from '@material-ui/icons/ChevronRight'; 8 | import ListItem from '@material-ui/core/ListItem'; 9 | import ListItemIcon from '@material-ui/core/ListItemIcon'; 10 | import ListItemText from '@material-ui/core/ListItemText'; 11 | import { makeStyles, useTheme } from '@material-ui/core/styles'; 12 | import {default as TaskIcon} from '@material-ui/icons/Assignment' ; 13 | import {default as DashboardIcon} from '@material-ui/icons/BarChart'; 14 | import {default as LabelsIcon} from '@material-ui/icons/Label'; 15 | import {default as DatasetIcon} from '@material-ui/icons/TextFormat'; 16 | import {default as SatisfiedIcon} from '@material-ui/icons/SentimentSatisfied'; 17 | import {default as VerySatisfiedIcon} from '@material-ui/icons/SentimentVerySatisfied'; 18 | import {default as MoodIcon} from '@material-ui/icons/Mood'; 19 | 20 | 21 | const drawerWidth = 200; 22 | 23 | const useStyles = makeStyles(theme => ({ 24 | root: { 25 | display: 'flex', 26 | }, 27 | menuButton: { 28 | marginRight: theme.spacing(2), 29 | }, 30 | hide: { 31 | display: 'none', 32 | }, 33 | drawer: { 34 | width: drawerWidth, 35 | flexShrink: 0, 36 | }, 37 | drawerPaper: { 38 | width: drawerWidth, 39 | }, 40 | drawerHeader: { 41 | display: 'flex', 42 | alignItems: 'center', 43 | padding: '0 8px', 44 | ...theme.mixins.toolbar, 45 | justifyContent: 'flex-end', 46 | }, 47 | content: { 48 | flexGrow: 1, 49 | padding: theme.spacing(3), 50 | transition: theme.transitions.create('margin', { 51 | easing: theme.transitions.easing.sharp, 52 | duration: theme.transitions.duration.leavingScreen, 53 | }), 54 | marginLeft: -drawerWidth, 55 | }, 56 | contentShift: { 57 | transition: theme.transitions.create('margin', { 58 | easing: theme.transitions.easing.easeOut, 59 | duration: theme.transitions.duration.enteringScreen, 60 | }), 61 | marginLeft: 0, 62 | }, 63 | })); 64 | 65 | 66 | const LeftDrawer = (props) => { 67 | 68 | const theme = useTheme(), 69 | classes = useStyles(); 70 | 71 | 72 | 73 | return( 74 | 83 |
84 | 85 | {theme.direction === 'ltr' ? : } 86 | 87 |
88 | 89 | 90 | {['Dataset', 'Labels', 'Task Guidelines', 'Dashboard'].map((text, index) => ( 91 | 92 | {index === 0 ? : 93 | index === 1 ? : 94 | index === 2 ? : 95 | } 96 | 97 | 98 | 99 | ))} 100 | 101 | 102 | 103 | {['Other', 'Labeling', 'Tasks'].map((text, index) => ( 104 | 105 | {index === 0 ? : 106 | index === 1 ? : 107 | } 108 | 109 | 110 | 111 | ))} 112 | 113 | 114 | 115 | ); 116 | 117 | }; 118 | 119 | export default LeftDrawer; 120 | 121 | -------------------------------------------------------------------------------- /ui/src/Link.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import PropTypes from 'prop-types'; 3 | 4 | import CancelIcon from '@material-ui/icons/Cancel'; 5 | import Badge from '@material-ui/core/Badge'; 6 | import IconButton from '@material-ui/core/IconButton'; 7 | 8 | const defaultAnchor = { x: 0.5, y: 0 }; 9 | const defaultBorderColor = '#6A747E'; 10 | const defaultBorderWidth = 2; 11 | const defaultOffset = 30; 12 | const defaultOpacity = 0.25 13 | 14 | 15 | export default class Link extends React.Component { 16 | constructor(props){ 17 | super(props); 18 | 19 | this.state = { 20 | isHovering: true, 21 | width: 0, 22 | height: 0 23 | }; 24 | 25 | this.updateDimensions = this.updateDimensions.bind(this); 26 | } 27 | 28 | componentDidMount() { 29 | window.addEventListener('resize', this.updateDimensions); 30 | } 31 | 32 | componentWillUnmount() { 33 | window.removeEventListener('resize', this.updateDimensions); 34 | } 35 | 36 | findElement(id) { 37 | return document.getElementById(id); 38 | } 39 | 40 | updateDimensions() { 41 | this.setState({ width: window.innerWidth, height: window.innerHeight }); 42 | } 43 | 44 | detect() { 45 | const { from, to} = this.props; 46 | 47 | const a = this.findElement(from); 48 | const b = this.findElement(to); 49 | 50 | if (!a || !b) { 51 | console.error(`Elements with ids ${from} and ${to} were not found.`); 52 | return false; 53 | } 54 | 55 | const anchor0 = defaultAnchor; 56 | const anchor1 = defaultAnchor; 57 | 58 | const box0 = a.getBoundingClientRect(); 59 | const box1 = b.getBoundingClientRect(); 60 | 61 | let offsetX = window.pageXOffset; 62 | let offsetY = window.pageYOffset; 63 | 64 | let x0 = box0.left + box0.width * anchor0.x + offsetX; 65 | let x1 = box1.left + box1.width * anchor1.x + offsetX; 66 | const y0 = box0.top + box0.height * anchor0.y + offsetY; 67 | const y1 = box1.top + box1.height * anchor1.y + offsetY; 68 | 69 | if (Math.abs(x0-x1) < 30) { 70 | x0 = box0.left + offsetX; 71 | x1 = box1.left + box1.width + offsetX; 72 | } 73 | return { x0, y0, x1, y1 }; 74 | } 75 | 76 | handleMouseHover(newState) { 77 | this.setState({isHovering: newState}); 78 | } 79 | 80 | render() { 81 | this.props.classes.anchorOriginTopRightRectangle = { 82 | bottom: 0, 83 | right: 0, 84 | transform: 'scale(1) translate(0%, -50%)', 85 | transformOrigin: '100% 100%', 86 | '&$invisible': { 87 | transform: 'scale(0) translate(50%, 50%)', 88 | }, 89 | } 90 | 91 | const classes = this.props.classes; 92 | let offset = this.props.offset || defaultOffset; 93 | 94 | const points = this.detect(); 95 | let {x0, y0, x1, y1} = points; 96 | if (!points) { 97 | return( 98 | 99 | ) 100 | } 101 | 102 | const leftOffset = Math.min(x0, x1); 103 | const topOffset = Math.min(y0, y1) - offset; 104 | 105 | x0 -= leftOffset; 106 | x1 -= leftOffset; 107 | y0 -= topOffset; 108 | y1 -= topOffset; 109 | 110 | const width = Math.abs(x0 - x1); 111 | 112 | const positionStyle = { 113 | position: 'absolute', 114 | top: `${topOffset}px`, 115 | left: `${leftOffset}px`, 116 | width: width, 117 | height: Math.abs(y0 - y1) + offset 118 | } 119 | 120 | const color = this.props.color || defaultBorderColor; 121 | const strokeWidth = this.props.width || defaultBorderWidth; 122 | const strokeOpacity = this.props.opacity || defaultOpacity; 123 | 124 | return ( 125 | this.handleMouseHover(true)} 130 | onMouseLeave={() => this.handleMouseHover(false)} 131 | onClick={this.props.onDelete}> 132 | 133 | 134 | } 135 | invisible={this.props.onDelete ? false : true} 136 | style={positionStyle}> 137 | 138 | this.handleMouseHover(true)} 146 | onMouseLeave={() => this.handleMouseHover(false)}/> 147 | 148 | 149 | ) 150 | } 151 | } 152 | 153 | Link.propTypes = { 154 | from: PropTypes.string.isRequired, 155 | to: PropTypes.string.isRequired, 156 | width: PropTypes.number, 157 | color: PropTypes.string, 158 | onDelete: PropTypes.func, 159 | offSet: PropTypes.number, 160 | classes: PropTypes.object 161 | } -------------------------------------------------------------------------------- /ui/src/Main.js: -------------------------------------------------------------------------------- 1 | import React from "react"; 2 | import { withStyles } from "@material-ui/core/styles"; 3 | import CssBaseline from "@material-ui/core/CssBaseline"; 4 | import NavigationBar from "./NavigationBar"; 5 | import LeftDrawer from "./LeftDrawer"; 6 | import ProjectGrid from "./ProjectGrid"; 7 | import Footer from "./Footer"; 8 | import ProjectCreation from "./ProjectCreation/ProjectCreation"; 9 | 10 | import { BrowserRouter as Router, Switch, Route } from "react-router-dom"; 11 | 12 | const styles = { 13 | grow: { 14 | flexGrow: 1, 15 | }, 16 | }; 17 | 18 | class Main extends React.Component { 19 | constructor(props) { 20 | super(props); 21 | 22 | this.handleDrawerClose = this.handleDrawerClose.bind(this); 23 | this.handleDrawerOpen = this.handleDrawerOpen.bind(this); 24 | 25 | this.drawerWidth = 200; 26 | this.state = { isDrawerOpen: false }; 27 | } 28 | 29 | handleDrawerClose() { 30 | this.setState({ isDrawerOpen: false }); 31 | } 32 | 33 | handleDrawerOpen() { 34 | this.setState({ isDrawerOpen: true }); 35 | } 36 | 37 | render() { 38 | const classes = this.props.classes, 39 | isDrawerOpen = this.state.isDrawerOpen; 40 | 41 | return ( 42 | 43 |
44 | 45 | 49 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 |
65 |
66 | 67 | ); 68 | } 69 | } 70 | 71 | export default withStyles(styles)(Main); 72 | -------------------------------------------------------------------------------- /ui/src/MegagonIcon.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import SvgIcon from '@material-ui/core/SvgIcon'; 3 | 4 | const MegagonIcon = props => ( 5 | 6 | 7 | 12 | 17 | 22 | 23 | ); 24 | 25 | export default MegagonIcon; -------------------------------------------------------------------------------- /ui/src/Navigation.js: -------------------------------------------------------------------------------- 1 | import React from "react"; 2 | import { bindActionCreators } from "redux"; 3 | import { connect } from "react-redux"; 4 | 5 | // actions 6 | import getStatistics from "./actions/getStatistics"; 7 | import submitLFs, { getLFstats } from "./actions/submitLFs"; 8 | import { getText } from "./actions/getText"; 9 | import { clear_suggestions } from "./actions/labelAndSuggestLF"; 10 | import { setAsCurrentInteraction } from "./actions/interaction"; 11 | 12 | // presentational component 13 | import NavigationButtons from "./NavigationButtons"; 14 | 15 | class Navigation extends React.Component { 16 | constructor(props) { 17 | super(props); 18 | this.state = { 19 | queued_interactions: [], 20 | }; 21 | } 22 | 23 | selected_LFs(del = false) { 24 | const LF_ids = Object.keys(this.props.suggestedLF); 25 | var suggestedLF = this.props.suggestedLF; 26 | const selected_LFs = LF_ids.reduce(function (selected_LFs, id) { 27 | let LF = suggestedLF[id]; 28 | if (LF.selected) { 29 | if (del) { 30 | delete LF.selected; 31 | } 32 | selected_LFs[id] = LF; 33 | } 34 | return selected_LFs; 35 | }, {}); 36 | return selected_LFs; 37 | } 38 | 39 | clickNext() { 40 | const selected_LFs = this.selected_LFs(true); 41 | const LFS_WILL_UPDATE = Object.keys(selected_LFs).length > 0; 42 | if (LFS_WILL_UPDATE) { 43 | for (var i = selected_LFs.length - 1; i >= 0; i--) { 44 | let lf = selected_LFs[i]; 45 | delete lf.selected; 46 | } 47 | this.props.submitLFs(selected_LFs); 48 | } 49 | 50 | if (this.state.queued_interactions.length === 0) { 51 | this.props.fetchNextText(); 52 | } else { 53 | this.props.setAsCurrentInteraction( 54 | this.state.queued_interactions.pop() 55 | ); 56 | } 57 | this.props.clear_suggestions(); 58 | this.props.getStatistics(); 59 | } 60 | 61 | clickPrevious() { 62 | this.state.queued_interactions.push(this.props.index); 63 | this.props.setAsCurrentInteraction(this.props.index - 1); 64 | } 65 | 66 | render() { 67 | const selected_LFs = this.selected_LFs(); 68 | const LFS_WILL_UPDATE = Object.keys(selected_LFs).length > 0; 69 | const disableNext = LFS_WILL_UPDATE && this.props.LFLoading; 70 | const FabText = 71 | this.props.currentLabel === null && this.props.text.length !== 0 72 | ? "Skip" 73 | : "Next"; 74 | 75 | return ( 76 | 0 ? this.clickPrevious.bind(this) : null 81 | } 82 | /> 83 | ); 84 | } 85 | } 86 | 87 | function mapStateToProps(state, ownProps?) { 88 | return { 89 | LFLoading: state.selectedLF.pending, 90 | text: state.text.data, 91 | suggestedLF: state.suggestedLF, 92 | currentLabel: state.label, 93 | index: state.text.index, 94 | }; 95 | } 96 | function mapDispatchToProps(dispatch) { 97 | return { 98 | fetchNextText: bindActionCreators(getText, dispatch), 99 | getStatistics: bindActionCreators(getStatistics, dispatch), 100 | submitLFs: bindActionCreators(submitLFs, dispatch), 101 | clear_suggestions: bindActionCreators(clear_suggestions, dispatch), 102 | getLFstats: bindActionCreators(getLFstats, dispatch), 103 | setAsCurrentInteraction: bindActionCreators( 104 | setAsCurrentInteraction, 105 | dispatch 106 | ), 107 | }; 108 | } 109 | 110 | export default connect(mapStateToProps, mapDispatchToProps)(Navigation); 111 | -------------------------------------------------------------------------------- /ui/src/NavigationBar.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import {makeStyles} from '@material-ui/core/styles'; 3 | import AppBar from '@material-ui/core/AppBar'; 4 | import Toolbar from '@material-ui/core/Toolbar'; 5 | import IconButton from '@material-ui/core/IconButton'; 6 | import Typography from '@material-ui/core/Typography'; 7 | import Badge from '@material-ui/core/Badge'; 8 | import MenuItem from '@material-ui/core/MenuItem'; 9 | import Menu from '@material-ui/core/Menu'; 10 | import MegagonIcon from './MegagonIcon'; 11 | import AccountCircle from '@material-ui/icons/AccountCircle'; 12 | import MailIcon from '@material-ui/icons/Mail'; 13 | import NotificationsIcon from '@material-ui/icons/Notifications'; 14 | import MoreIcon from '@material-ui/icons/MoreVert'; 15 | import clsx from 'clsx'; 16 | import SaveButton from './SaveButton' 17 | 18 | const drawerWidth = 200; 19 | 20 | const useStyles = makeStyles(theme => ({ 21 | grow: { 22 | flexGrow: 1, 23 | }, 24 | title: { 25 | display: 'none', 26 | [theme.breakpoints.up('sm')]: { 27 | display: 'block', 28 | }, 29 | }, 30 | appBar: { 31 | transition: theme.transitions.create(['margin', 'width'], { 32 | easing: theme.transitions.easing.sharp, 33 | duration: theme.transitions.duration.leavingScreen, 34 | }), 35 | }, 36 | appBarShift: { 37 | width: `calc(100% - ${drawerWidth}px)`, 38 | marginLeft: drawerWidth, 39 | transition: theme.transitions.create(['margin', 'width'], { 40 | easing: theme.transitions.easing.easeOut, 41 | duration: theme.transitions.duration.enteringScreen, 42 | }) 43 | }, 44 | inputRoot: { 45 | color: 'inherit', 46 | }, 47 | hide: { 48 | display: 'none', 49 | }, 50 | inputInput: { 51 | padding: theme.spacing(1, 1, 1, 7), 52 | transition: theme.transitions.create('width'), 53 | width: '100%', 54 | [theme.breakpoints.up('md')]: { 55 | width: 200, 56 | }, 57 | }, 58 | sectionDesktop: { 59 | display: 'none', 60 | [theme.breakpoints.up('md')]: { 61 | display: 'flex', 62 | }, 63 | }, 64 | sectionMobile: { 65 | display: 'flex', 66 | [theme.breakpoints.up('md')]: { 67 | display: 'none', 68 | }, 69 | }, 70 | })); 71 | 72 | const NavigationBar = (props)=> { 73 | 74 | const classes = useStyles(), 75 | open = props.isDrawerOpen; 76 | 77 | const [anchorEl, setAnchorEl] = React.useState(null); 78 | const [mobileMoreAnchorEl, setMobileMoreAnchorEl] = React.useState(null); 79 | 80 | const isMenuOpen = Boolean(anchorEl); 81 | const isMobileMenuOpen = Boolean(mobileMoreAnchorEl); 82 | 83 | 84 | function handleProfileMenuOpen(event) { 85 | setAnchorEl(event.currentTarget); 86 | } 87 | 88 | function handleMobileMenuClose() { 89 | setMobileMoreAnchorEl(null); 90 | } 91 | 92 | function handleMenuClose() { 93 | setAnchorEl(null); 94 | handleMobileMenuClose(); 95 | } 96 | 97 | function handleMobileMenuOpen(event) { 98 | setMobileMoreAnchorEl(event.currentTarget); 99 | } 100 | const menuId = 'primary-search-account-menu'; 101 | const renderMenu = ( 102 | 111 | Profile 112 | My account 113 | Logout 114 | 115 | ); 116 | 117 | const mobileMenuId = 'primary-search-account-menu-mobile'; 118 | const renderMobileMenu = ( 119 | 128 | 129 | 130 | 131 | 132 | 133 | 134 |

Messages

135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 |

Notifications

143 | 144 | 145 | 146 | 147 | 148 | 154 | 155 | 156 |

Profile

157 | 158 | 159 | ); 160 | 161 | return ( 162 | 167 | 168 | 175 | 176 | 177 | 178 | Řuler: Data Programming by Demonstration for Text 179 | 180 | 181 |
182 |
183 | 191 | 192 | 193 | 194 |
195 |
196 | 203 | 204 | 205 |
206 | {renderMobileMenu} 207 | {renderMenu} 208 | 209 | 210 | ); 211 | }; 212 | 213 | export default NavigationBar; 214 | 215 | -------------------------------------------------------------------------------- /ui/src/NavigationButtons.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | 3 | // material UI 4 | import ArrowIcon from '@material-ui/icons/ArrowForward'; 5 | import ArrowBackIcon from '@material-ui/icons/ArrowBack'; 6 | import ButtonGroup from '@material-ui/core/ButtonGroup'; 7 | import Button from '@material-ui/core/Button'; 8 | 9 | export default class NavigationButtons extends React.Component { 10 | render() { 11 | return( 12 | 13 | 22 | 32 | ) 33 | } 34 | } -------------------------------------------------------------------------------- /ui/src/ProjectCreation/LabelConfig.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import {bindActionCreators} from 'redux'; 3 | import {connect} from "react-redux"; 4 | 5 | import fetchClasses, {addLabelClass} from '../actions/labelClasses'; 6 | 7 | import AddIcon from '@material-ui/icons/Add'; 8 | import Button from '@material-ui/core/Button'; 9 | import TextField from '@material-ui/core/TextField'; 10 | import Typography from '@material-ui/core/Typography'; 11 | 12 | class LabelConfig extends React.Component { 13 | 14 | constructor(props) { 15 | super(props); 16 | var new_label_key = Object.keys(props.labelClasses).length; 17 | 18 | this.state = { 19 | newLabel: "", 20 | fetched: false, 21 | new_label_key: new_label_key 22 | }; 23 | this.handleLabelInput = this.handleLabelInput.bind(this); 24 | this.handleAdd = this.handleAdd.bind(this); 25 | 26 | this.props.fetchClasses(); 27 | } 28 | 29 | handleLabelInput(event) { 30 | this.setState({ 31 | newLabel: event.target.value 32 | }); 33 | } 34 | 35 | handleAdd() { 36 | const newLabel = this.state.newLabel.trim() 37 | if (newLabel !== "") { 38 | this.props.addLabel({[newLabel]: this.state.new_label_key}); 39 | this.setState({ 40 | newLabel: "", 41 | new_label_key: this.state.new_label_key + 1 42 | }); 43 | } 44 | } 45 | 46 | handleKeyPress(event){ 47 | if (event.key === 'Enter') { 48 | this.handleAdd(); 49 | } 50 | } 51 | 52 | render() { 53 | const labels = Object.entries(this.props.labelClasses); 54 | console.log(labels) 55 | const classes = this.props.classes; 56 | return( 57 |
58 | Label Classes 59 | { labels.map( (item) => { 60 | var lname = item[0]; 61 | var value = item[1]; 62 | return( 63 |
64 | 72 |
73 | ) 74 | }) 75 | } 76 | 85 | 86 | }} 87 | /> 88 |
89 | ); 90 | } 91 | } 92 | 93 | 94 | function mapStateToProps(state, ownProps?) { 95 | return { 96 | labelClasses: state.labelClasses.data, 97 | labelClassesPending: state.labelClasses.pending, 98 | }; 99 | } 100 | 101 | function mapDispatchToProps(dispatch){ 102 | return { 103 | addLabel: bindActionCreators(addLabelClass, dispatch), 104 | fetchClasses: bindActionCreators(fetchClasses, dispatch) 105 | }; 106 | } 107 | 108 | export default connect(mapStateToProps, mapDispatchToProps)(LabelConfig); -------------------------------------------------------------------------------- /ui/src/ProjectCreation/ModelSelect.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import {bindActionCreators} from 'redux'; 3 | import {connect} from "react-redux"; 4 | 5 | import fetchClasses, {addLabelClass} from '../actions/labelClasses'; 6 | 7 | import AddIcon from '@material-ui/icons/Add'; 8 | import Button from '@material-ui/core/Button'; 9 | import FormControl from '@material-ui/core/FormControl'; 10 | import Select from '@material-ui/core/Select'; 11 | import TextField from '@material-ui/core/TextField'; 12 | import Typography from '@material-ui/core/Typography'; 13 | 14 | import { fetchModels, setSelected, createNewModel } from '../actions/model' 15 | 16 | class ModelSelect extends React.Component { 17 | constructor(props) { 18 | super(props); 19 | 20 | this.state = { 21 | newModel: "", 22 | }; 23 | this.handleSelect = this.handleSelect.bind(this); 24 | this.createModel = this.createModel.bind(this); 25 | this.handleInput = this.handleInput.bind(this); 26 | this.handleKeyPress = this.handleKeyPress.bind(this); 27 | 28 | this.props.fetchModels(); 29 | } 30 | 31 | // MODELS MANAGEMENT 32 | fetchModels() { 33 | if ( !this.props.user ) 34 | return; 35 | this.props.user.getIdToken(true).then((idToken) => { 36 | this.props.fetchModels(idToken); 37 | }); 38 | } 39 | 40 | handleSelect(event) { 41 | const { options } = event.target; 42 | const value = []; 43 | for (let i = 0, l = options.length; i < l; i += 1) { 44 | if (options[i].selected) { 45 | value.push(options[i].value); 46 | } 47 | } 48 | this.props.setSelected(value[0]); 49 | } 50 | 51 | handleInput(event){ 52 | this.setState({ 53 | newModel: event.target.value 54 | }); 55 | } 56 | 57 | createModel(){ 58 | this.props.createNewModel(this.state.newModel); 59 | } 60 | 61 | handleKeyPress(event){ 62 | if (event.key === 'Enter') { 63 | this.createModel(); 64 | } 65 | } 66 | 67 | render() { 68 | const model_names = this.props.models; 69 | const { classes } = this.props; 70 | 71 | return( 72 |
73 | 74 | Load Model 75 |
76 | 77 | 86 | 87 |
88 | -OR- 89 | Create new model: 90 |
91 | Model Name 92 | 102 | 103 | }} 104 | /> 105 | 106 |
107 |
108 | ) 109 | } 110 | } 111 | 112 | function mapStateToProps(state, ownProps?) { 113 | return { 114 | models: state.models.data, 115 | modelsPending: state.models.pending, 116 | }; 117 | } 118 | 119 | function mapDispatchToProps(dispatch){ 120 | return { 121 | addLabel: bindActionCreators(addLabelClass, dispatch), 122 | fetchClasses: bindActionCreators(fetchClasses, dispatch), 123 | fetchModels: bindActionCreators(fetchModels, dispatch), 124 | setSelected: bindActionCreators(setSelected, dispatch), 125 | createNewModel: bindActionCreators(createNewModel, dispatch) 126 | }; 127 | } 128 | 129 | export default connect(mapStateToProps, mapDispatchToProps)(ModelSelect); -------------------------------------------------------------------------------- /ui/src/RichTextUtils.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | 3 | import { styled, withStyles } from '@material-ui/core/styles'; 4 | import Badge from '@material-ui/core/Badge'; 5 | import Box from '@material-ui/core/Box'; 6 | import Tooltip from '@material-ui/core/Tooltip'; 7 | 8 | export const InlineBox = styled(Box)({ 9 | width: "fit-content", 10 | display: "inline" 11 | }); 12 | 13 | const StyledBadge = withStyles((theme) => ({ 14 | badge: { 15 | right: "50%", 16 | top: 10, 17 | borderRadius: "50%", 18 | //background: `${theme.palette.background.paper}`, 19 | //padding: '0 4px', 20 | }, 21 | }))(Badge); 22 | 23 | const InlineBadge = styled(StyledBadge)({ 24 | width: "fit-content", 25 | display: "inline" 26 | }); 27 | 28 | export function NERSpan(text, NER, start_offset) { 29 | return( 30 | 34 | 35 | {NER.label}} //should be NER.label but that breaks the annotation... 41 | >{text} 42 | 43 | 44 | ) 45 | } 46 | 47 | export function MatchedSpan(text, matched_span_data, start_offset) { 48 | return( 49 | 53 | {text} 54 | 55 | ) 56 | } -------------------------------------------------------------------------------- /ui/src/SaveButton.js: -------------------------------------------------------------------------------- 1 | import IconButton from "@material-ui/core/IconButton"; 2 | import SaveIcon from '@material-ui/icons/Save'; 3 | 4 | import React from "react"; 5 | import { bindActionCreators } from "redux"; 6 | import { connect } from "react-redux"; 7 | import { saveModel } from "./actions/save"; 8 | 9 | const SaveButton = (props) => { 10 | return ( 11 | 12 | 13 | 14 | ); 15 | }; 16 | 17 | function mapStateToProps(state, ownProps?) { 18 | return {}; 19 | } 20 | 21 | function mapDispatchToProps(dispatch) { 22 | return { 23 | save: bindActionCreators(saveModel, dispatch), 24 | }; 25 | } 26 | 27 | export default connect(mapStateToProps, mapDispatchToProps)(SaveButton); 28 | -------------------------------------------------------------------------------- /ui/src/SelectedSpan.js: -------------------------------------------------------------------------------- 1 | import React from "react"; 2 | import PropTypes from "prop-types"; 3 | import { connect } from "react-redux"; 4 | 5 | import CancelIcon from "@material-ui/icons/Cancel"; 6 | import IconButton from "@material-ui/core/IconButton"; 7 | import LinkIcon from "@material-ui/icons/Link"; 8 | import LinkOffIcon from "@material-ui/icons/LinkOff"; 9 | import Typography from "@material-ui/core/Typography"; 10 | 11 | import { InlineBox } from "./RichTextUtils"; 12 | import { DIR_LINK, UNDIR_LINK } from "./AnnotationBuilder"; 13 | 14 | import { styled } from "@material-ui/core/styles"; 15 | import Badge from "@material-ui/core/Badge"; 16 | 17 | const DeleteBadge = styled(Badge)({ 18 | width: "fit-content", 19 | display: "inline", 20 | left: 10, 21 | }); 22 | 23 | const LinkBadge = styled(Badge)({ 24 | width: "fit-content", 25 | display: "inline", 26 | //top: 5 27 | }); 28 | 29 | class SelectedSpan extends React.Component { 30 | constructor(props) { 31 | super(props); 32 | 33 | this.handleClick = this.handleClick.bind(this); 34 | this.handleMouseHover = this.handleMouseHover.bind(this); 35 | this.state = { 36 | isHovering: false, 37 | }; 38 | } 39 | 40 | handleMouseHover(newState) { 41 | this.setState({ isHovering: newState }); 42 | } 43 | 44 | delayMouseLeave() { 45 | setTimeout( 46 | function () { 47 | this.setState({ isHovering: false }); 48 | }.bind(this), 49 | 1000 50 | ); 51 | } 52 | 53 | handleClick() { 54 | this.props.annotate(); 55 | } 56 | 57 | render() { 58 | const classes = this.props.classes; 59 | const linkVisible = 60 | this.state.isHovering || 61 | this.props.selectedLink.type === UNDIR_LINK; 62 | let style = this.props.style; 63 | 64 | const text = this.props.text; 65 | 66 | const innerSpan = ( 67 | this.handleMouseHover(true)} 72 | onMouseLeave={() => this.handleMouseHover(false)} 73 | onClick={ 74 | "clickSegment" in this.props 75 | ? this.props.clickSegment 76 | : () => {} 77 | } 78 | > 79 | 86 | {text} 87 | 88 | 89 | ); 90 | 91 | if (this.props.clickLinkButton && this.props.onDelete) { 92 | return ( 93 | <> 94 | this.handleMouseHover(true)} 101 | //onMouseLeave={() => this.handleMouseHover(false)} 102 | onClick={this.props.clickLinkButton} 103 | > 104 | {this.props.linked ? ( 105 | 106 | ) : ( 107 | 108 | )} 109 | 110 | } 111 | > 112 | {""} 113 | 114 | {innerSpan} 115 | this.handleMouseHover(true)} 121 | //onMouseLeave={() => this.handleMouseHover(false)} 122 | onClick={this.props.onDelete} 123 | > 124 | 125 | 126 | } 127 | > 128 | {""} 129 | 130 | 131 | ); 132 | } else { 133 | return innerSpan; 134 | } 135 | } 136 | } 137 | 138 | SelectedSpan.propTypes = { 139 | annotate: PropTypes.func, 140 | clickLinkButton: PropTypes.func, 141 | onDelete: PropTypes.func, 142 | classes: PropTypes.object, 143 | clickSegment: PropTypes.func, 144 | }; 145 | 146 | function mapStateToProps(state, ownProps?) { 147 | return { selectedLink: state.selectedLink }; 148 | } 149 | function mapDispatchToProps(dispatch) { 150 | return {}; 151 | } 152 | 153 | export default connect(mapStateToProps, mapDispatchToProps)(SelectedSpan); 154 | -------------------------------------------------------------------------------- /ui/src/SortingTableUtils.js: -------------------------------------------------------------------------------- 1 | 2 | export function desc(a, b, orderBy) { 3 | if (b[orderBy] < a[orderBy]) { 4 | return -1; 5 | } 6 | if (b[orderBy] > a[orderBy]) { 7 | return 1; 8 | } 9 | return 0; 10 | } 11 | 12 | export function stableSort(array, cmp) { 13 | const stabilizedThis = array.map((el, index) => [el, index]); 14 | stabilizedThis.sort((a, b) => { 15 | const order = cmp(a[0], b[0]); 16 | if (order !== 0) return order; 17 | return a[1] - b[1]; 18 | }); 19 | return stabilizedThis.map(el => el[0]); 20 | } 21 | 22 | export function getSorting(order, orderBy) { 23 | return order === 'desc' ? (a, b) => desc(a, b, orderBy) : (a, b) => -desc(a, b, orderBy); 24 | } 25 | 26 | export function style(number) { 27 | if (isNaN(number)) { 28 | return null; 29 | } else if (number <= 1.0) { 30 | return parseFloat(Math.round(number * 100) / 100).toFixed(3); 31 | } else { 32 | return number; 33 | } 34 | } -------------------------------------------------------------------------------- /ui/src/Span.js: -------------------------------------------------------------------------------- 1 | let Span = function(startOffset, endOffset, text, label=null) { 2 | const span = { 3 | id: startOffset, 4 | label: label, 5 | start_offset: startOffset, 6 | end_offset: endOffset, 7 | text: text.slice(startOffset, endOffset), 8 | link: null 9 | }; 10 | return span; 11 | } 12 | 13 | export default Span; -------------------------------------------------------------------------------- /ui/src/StatisticsLRPane.js: -------------------------------------------------------------------------------- 1 | import React from "react"; 2 | import { connect } from "react-redux"; 3 | import { bindActionCreators } from "redux"; 4 | 5 | import ArrowDropUpIcon from "@material-ui/icons/ArrowDropUp"; 6 | import ArrowDropDownIcon from "@material-ui/icons/ArrowDropDown"; 7 | import Divider from "@material-ui/core/Divider"; 8 | import LinearProgress from "@material-ui/core/LinearProgress"; 9 | import Paper from "@material-ui/core/Paper"; 10 | import RefreshIcon from "@material-ui/icons/Refresh"; 11 | import Table from "@material-ui/core/Table"; 12 | import TableBody from "@material-ui/core/TableBody"; 13 | import TableCell from "@material-ui/core/TableCell"; 14 | import TableRow from "@material-ui/core/TableRow"; 15 | import Typography from "@material-ui/core/Typography"; 16 | 17 | import { getLRStatistics } from "./actions/getStatistics"; 18 | import { style } from "./SortingTableUtils"; 19 | 20 | class LRStatisticsPane extends React.Component { 21 | componentDidUpdate(prevProps, prevState) { 22 | if ( 23 | prevProps.statistics !== this.props.statistics && 24 | Object.keys(prevProps.statistics).length > 0 25 | ) { 26 | this.setState({ prevStats: prevProps.statistics }); 27 | } 28 | } 29 | 30 | statDelta(key) { 31 | if (this.state && "prevStats" in this.state) { 32 | const delta = this.props.statistics[key] - this.state.prevStats[key]; 33 | let cellContent = delta; 34 | if (delta > 0.0) { 35 | cellContent = ( 36 | 37 | {style(delta)}{" "} 38 | 39 | ); 40 | } else if (delta < -0.0) { 41 | cellContent = ( 42 | 43 | {style(delta)} 44 | 45 | ); 46 | } 47 | return {cellContent}; 48 | } 49 | } 50 | 51 | render() { 52 | const classes = this.props.classes; 53 | const prevStats = this.state && "prevStats" in this.state; 54 | return ( 55 | 56 | 57 | 61 | Trained Model Statistics 62 | 63 | 64 | Train a logistic regression model with bag-of-words features on your 65 | training set. 66 | 67 | 68 | {this.props.pending ? : } 69 | 70 | 76 | 77 | {["accuracy", "micro_f1"].map((key) => { 78 | if (key in this.props.statistics) { 79 | return ( 80 | 81 | {key} 82 | 83 | {style(this.props.statistics[key])} 84 | 85 | {this.statDelta(key)} 86 | 87 | ); 88 | } else { 89 | return null; 90 | } 91 | })} 92 | 93 |
94 |
95 | 96 | Class-Specific Statistics 97 | 98 | 99 | 100 | 101 | 102 | Class0 103 | 104 | 105 | {prevStats ? : null} 106 | 107 | Class1 108 | 109 | 110 | 111 | {["precision", "recall"].map((key) => { 112 | if (key + "_0" in this.props.statistics) { 113 | return ( 114 | 115 | {key + "_0"} 116 | 117 | {style(this.props.statistics[key + "_0"])} 118 | 119 | {this.statDelta(key + "_0")} 120 | {key + "_1"} 121 | 122 | {style(this.props.statistics[key + "_1"])} 123 | 124 | {this.statDelta(key + "_1")} 125 | 126 | ); 127 | } 128 | return null; 129 | })} 130 | 131 |
132 | 133 | ); 134 | } 135 | } 136 | 137 | function mapStateToProps(state, ownProps?) { 138 | return { 139 | statistics: state.statistics_LRmodel.data, 140 | pending: state.statistics_LRmodel.pending, 141 | }; 142 | } 143 | function mapDispatchToProps(dispatch) { 144 | return { 145 | getLRStatistics: bindActionCreators(getLRStatistics, dispatch), 146 | }; 147 | } 148 | export default connect(mapStateToProps, mapDispatchToProps)(LRStatisticsPane); 149 | -------------------------------------------------------------------------------- /ui/src/StatisticsPane.js: -------------------------------------------------------------------------------- 1 | import React from "react"; 2 | import { connect } from "react-redux"; 3 | 4 | import ArrowDropUpIcon from "@material-ui/icons/ArrowDropUp"; 5 | import ArrowDropDownIcon from "@material-ui/icons/ArrowDropDown"; 6 | import Divider from "@material-ui/core/Divider"; 7 | import LinearProgress from "@material-ui/core/LinearProgress"; 8 | import Paper from "@material-ui/core/Paper"; 9 | import Table from "@material-ui/core/Table"; 10 | import TableBody from "@material-ui/core/TableBody"; 11 | import TableCell from "@material-ui/core/TableCell"; 12 | import TableRow from "@material-ui/core/TableRow"; 13 | import Typography from "@material-ui/core/Typography"; 14 | 15 | import { style } from "./SortingTableUtils"; 16 | 17 | class StatisticsPane extends React.Component { 18 | componentDidUpdate(prevProps, prevState) { 19 | if (Object.keys(prevProps.statistics).length > 0) { 20 | if (!this.statsSame(prevProps.statistics, this.props.statistics)) { 21 | this.setState({ prevStats: prevProps.statistics }); 22 | } 23 | } 24 | } 25 | 26 | statsSame(oldStats, newStats) { 27 | const stat_list = Object.keys(oldStats); 28 | for (var i = stat_list.length - 1; i >= 0; i--) { 29 | let stat_key = stat_list[i]; 30 | if (oldStats[stat_key] !== newStats[stat_key]) { 31 | return false; 32 | } 33 | } 34 | return true; 35 | } 36 | 37 | statDelta(key) { 38 | if (this.state && "prevStats" in this.state) { 39 | const delta = this.props.statistics[key] - this.state.prevStats[key]; 40 | let cellContent = delta; 41 | if (delta > 0) { 42 | cellContent = ( 43 | 44 | {style(delta)}{" "} 45 | 46 | ); 47 | } else if (delta < 0) { 48 | cellContent = ( 49 | 50 | {style(delta)} 51 | 52 | ); 53 | } 54 | return {cellContent}; 55 | } 56 | } 57 | 58 | render() { 59 | const classes = this.props.classes; 60 | return ( 61 | 62 | 63 | Labeling Statistics 64 | 65 | 66 | Your labeling functions are agreggated by snorkel's labeling model 67 | then evaluated on the labeled development set. More information about 68 | these metrics is available via{" "} 69 | 70 | Scikit-Learn 71 | 72 | . 73 | 74 | 75 | {this.props.pending ? : } 76 | 77 | 83 | 84 | {Object.keys(this.props.statistics).map((key) => ( 85 | 86 | {key} 87 | 88 | {style(this.props.statistics[key])} 89 | 90 | {this.statDelta(key)} 91 | 92 | ))} 93 | 94 |
95 | 96 | ); 97 | } 98 | } 99 | 100 | function mapStateToProps(state, ownProps?) { 101 | return { 102 | statistics: state.statistics.data, 103 | pending: state.statistics.pending, 104 | }; 105 | } 106 | function mapDispatchToProps(dispatch) { 107 | return {}; 108 | } 109 | export default connect(mapStateToProps, mapDispatchToProps)(StatisticsPane); 110 | -------------------------------------------------------------------------------- /ui/src/actions/ConceptStyler.js: -------------------------------------------------------------------------------- 1 | class ConceptStyler{ 2 | 3 | constructor() { 4 | this.colorIndex = 0; 5 | this.colorAssignments = {}; 6 | this.hotKeys = {}; 7 | this.availableLetters = "abcdefghijklmnopqrstuvwxyz-=[];',."; 8 | } 9 | 10 | hotkey(conceptString) { 11 | if (conceptString in this.hotKeys) { 12 | return this.hotKeys[conceptString]; 13 | } 14 | 15 | let letter = conceptString[0]; 16 | if (this.availableLetters.indexOf(letter) === -1) { 17 | letter = this.availableLetters[0]; 18 | } 19 | this.availableLetters = this.availableLetters.replace(letter, ""); 20 | this.hotKeys[conceptString] = letter; 21 | return letter; 22 | } 23 | 24 | color(conceptString) { 25 | return this.nextColor(conceptString); 26 | } 27 | 28 | nextColor(string){ 29 | if (string in this.colorAssignments){ 30 | return this.colorAssignments[string]; 31 | } 32 | const colorPalette = ["#2CA02C", "#E377C2", "#17BECF", "#8C564B", "#D62728", "#BCBD22", "#9467BD", "#FF7F0E", "#1F77B4"]; 33 | let color = colorPalette[this.colorIndex]; 34 | this.colorAssignments[string] = color; 35 | this.colorIndex = (this.colorIndex + 1) % colorPalette.length; 36 | return color 37 | } 38 | 39 | stringToColor(string) { 40 | let hash = 0; 41 | let i; 42 | 43 | /* eslint-disable no-bitwise */ 44 | for (i = 0; i < string.length; i += 1) { 45 | hash = string.charCodeAt(i) + ((hash << 5) - hash); 46 | } 47 | 48 | let colour = '#'; 49 | 50 | for (i = 0; i < 3; i += 1) { 51 | const value = (hash >> (i * 8)) & 0xff; 52 | colour += `00${value.toString(16)}`.substr(-2); 53 | } 54 | /* eslint-enable no-bitwise */ 55 | 56 | return colour; 57 | } 58 | } 59 | 60 | let conceptStyler = new ConceptStyler(); 61 | export default conceptStyler; -------------------------------------------------------------------------------- /ui/src/actions/annotate.js: -------------------------------------------------------------------------------- 1 | export const ANNOTATE = "ANNOTATE"; 2 | export const HIGHLIGHT = "HIGHLIGHT"; 3 | export const HIGHLIGHT_ERROR = "HIGHLIGHT_ERROR" 4 | export const SELECT_LINK = "SELECT_LINK"; 5 | export const ADD_NER = "ADD_NER"; 6 | 7 | export function annotate(data) { 8 | /* data is an array of annotations, for example 9 | [{ 10 | id: 11, 11 | label: 0, 12 | start_offset: 11, 13 | end_offset: 17, 14 | text: 'Russia', 15 | link: null 16 | }] 17 | ] */ 18 | return dispatch => { 19 | dispatch({ 20 | type: ANNOTATE, 21 | data 22 | }) 23 | } 24 | } 25 | 26 | export function highlight(data) { 27 | /* data is an array of annotations, for example 28 | [{ 29 | id: 11, 30 | label: 0, 31 | start_offset: 11, 32 | end_offset: 17, 33 | text: 'Russia', 34 | link: null 35 | }] 36 | ] */ 37 | return dispatch => { 38 | dispatch({ 39 | type: HIGHLIGHT, 40 | data 41 | }) 42 | } 43 | } 44 | 45 | export function NER(data) { 46 | return dispatch => { 47 | dispatch({ 48 | type: ADD_NER, 49 | data 50 | }) 51 | } 52 | } 53 | 54 | export function highlight_regex(regex_str, text, case_sensitive, idx) { 55 | var data = []; 56 | var matches = []; 57 | try { 58 | var regex = null; 59 | if (!case_sensitive) { 60 | regex = new RegExp(regex_str, 'ig'); 61 | } else { 62 | regex = new RegExp(regex_str, 'g'); 63 | } 64 | matches = text.matchAll(regex); 65 | } 66 | catch (err) { 67 | return dispatch => dispatch({ 68 | type: HIGHLIGHT_ERROR, error: err.toString(), idx: idx}); 69 | } 70 | for (const match of matches) { 71 | data.push({ 72 | id: `${match.index}_pending`, // 'pending' because this annotation will not necessarily be submitted 73 | label: 'pending', 74 | start_offset: match.index, 75 | end_offset: match.index + match[0].length, 76 | link: null, 77 | }); 78 | } 79 | return highlight(data); 80 | } 81 | 82 | export function isToken(string) { 83 | if (string.startsWith("(?:(?<=\\W)|(?<=^))(")) { 84 | string = string.slice("(?:(?<=\\W)|(?<=^))(".length); 85 | if (string.endsWith(")(?=\\W|$)")) { 86 | string = string.slice(0, string.length - ")(?=\\W|$)".length); 87 | return string; 88 | } 89 | } return false; 90 | } 91 | 92 | export function TokenToRegex(token) { 93 | const regex = "(?:(?<=\\W)|(?<=^))(" + token + ")(?=\\W|$)"; 94 | return regex; 95 | } 96 | 97 | export function highlight_string(string, text, case_sensitive=false) { 98 | if (!case_sensitive) { 99 | text = text.toLowerCase(); 100 | } 101 | const regex = TokenToRegex(string, case_sensitive); 102 | 103 | return highlight_regex(regex, text) 104 | } 105 | 106 | export function select_link(data){ 107 | /* EXAMPLE DATA 108 | { 109 | type: 'Undirected Link', 110 | segment: { 111 | id: 11, 112 | label: 0, 113 | start_offset: 11, 114 | end_offset: 15, 115 | text: 'Ever', 116 | link: null 117 | } 118 | } 119 | */ 120 | return dispatch => { 121 | dispatch({ 122 | type: SELECT_LINK, 123 | data 124 | }) 125 | } 126 | } -------------------------------------------------------------------------------- /ui/src/actions/concepts.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | import conceptStyler from './ConceptStyler' 3 | 4 | export const GET_CONCEPTS_PENDING = "GET_CONCEPTS_PENDING"; 5 | export const UPDATE_CONCEPT_PENDING = "UPDATE_CONCEPT_PENDING"; 6 | export const GET_CONCEPTS_SUCCESS = "GET_CONCEPTS_SUCCESS"; 7 | export const GET_CONCEPTS_ERROR = "GET_CONCEPTS_ERROR"; 8 | export const SELECT_CONCEPT = 'SELECT_CONCEPT' 9 | 10 | const api = process.env.REACT_APP_SERVER; 11 | 12 | function getConceptsPending() { 13 | return { 14 | type: GET_CONCEPTS_PENDING, 15 | } 16 | } 17 | 18 | function updateConceptPending(conceptName) { 19 | return { 20 | type: UPDATE_CONCEPT_PENDING, 21 | conceptName: conceptName 22 | } 23 | } 24 | 25 | function getConceptsSuccess(data) { 26 | return { 27 | type: GET_CONCEPTS_SUCCESS, 28 | data: data 29 | } 30 | } 31 | 32 | function getConceptsError(error) { 33 | return { 34 | type: GET_CONCEPTS_ERROR, 35 | error: error 36 | } 37 | } 38 | 39 | function fetchConcepts() { 40 | return dispatch => { 41 | dispatch(getConceptsPending()); 42 | axios.get(`${api}/concept`) 43 | .then(response => { 44 | if(response.error) { 45 | throw(response.error); 46 | } 47 | const conceptNames = Object.keys(response.data); 48 | let data = {}; 49 | for (let i=0; i { 62 | dispatch(getConceptsError(error)); 63 | }) 64 | } 65 | } 66 | 67 | function addConcept(conceptName){ 68 | const data = { 69 | name: conceptName, 70 | tokens: [] 71 | }; 72 | 73 | return dispatch => { 74 | dispatch(getConceptsPending()); 75 | axios.post(`${api}/concept`, data) 76 | .then(response => { 77 | if(response.error) { 78 | throw(response.error); 79 | } 80 | dispatch(fetchConcepts()) 81 | }); 82 | } 83 | } 84 | 85 | function deleteConcept(conceptName){ 86 | return dispatch => { 87 | dispatch(updateConceptPending(conceptName)); 88 | axios.delete(`${api}/concept/${conceptName}`) 89 | .then(response => { 90 | if(response.error) { 91 | throw(response.error); 92 | } 93 | dispatch(fetchConcepts()) 94 | }); 95 | } 96 | } 97 | 98 | // add string to concept tokens list 99 | function updateConcept(conceptName, data){ 100 | return dispatch => { 101 | dispatch(updateConceptPending(conceptName)); 102 | axios.put(`${api}/concept/${conceptName}`, data) 103 | .then(response => { 104 | if(response.error) { 105 | throw(response.error); 106 | } 107 | dispatch(fetchConcepts()) 108 | }); 109 | } 110 | } 111 | 112 | // select a concept for annotating spans 113 | export function select_concept(data){ 114 | return dispatch => dispatch({type: SELECT_CONCEPT, data: data}) 115 | } 116 | 117 | export const conceptEditors = { 118 | addConcept, 119 | deleteConcept, 120 | fetchConcepts, 121 | updateConcept 122 | } -------------------------------------------------------------------------------- /ui/src/actions/connectivesAndKeyTypes.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | export const KEYTYPE = "KEYTYPE" 4 | export const CONNECTIVE = "CONNECTIVE" 5 | const api = process.env.REACT_APP_SERVER; 6 | 7 | export function getKeyTypes() { 8 | return dispatch => { 9 | axios.get(`${api}/keytype`) 10 | .then(response => { 11 | if(response.error) { 12 | throw(response.error); 13 | } 14 | dispatch({type: KEYTYPE, data: response.data}); 15 | }) 16 | } 17 | } 18 | 19 | export function getConnectives() { 20 | return dispatch => { 21 | axios.get(`${api}/connective`) 22 | .then(response => { 23 | if(response.error) { 24 | throw(response.error); 25 | } 26 | dispatch({type: CONNECTIVE, data: response.data}); 27 | }) 28 | } 29 | } -------------------------------------------------------------------------------- /ui/src/actions/datasets.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | export const SUCCESS = "DATASET_UPLOAD_SUCCESS" 4 | export const PENDING = "DATASET_UPLOAD_PENDING" 5 | export const ERROR = "DATASET_UPLOAD_ERROR" 6 | export const SELECT = "DATASET_SELECT" 7 | 8 | const api = process.env.REACT_APP_SERVER; 9 | 10 | 11 | function UploadError(error) { 12 | return { 13 | type: ERROR, 14 | error: error 15 | } 16 | } 17 | 18 | function selectDataset(data) { 19 | return { 20 | type:SELECT, 21 | data: data 22 | } 23 | } 24 | 25 | function UploadPending() { 26 | return { 27 | type: PENDING 28 | } 29 | } 30 | 31 | function UploadSuccess(data) { 32 | return { 33 | type: SUCCESS, 34 | data: data 35 | } 36 | } 37 | 38 | export function fetchDatasets(idToken) { 39 | return dispatch => { 40 | axios.get(`${api}/datasets`, 41 | { 42 | headers: { 43 | 'Authorization': 'Bearer ' + idToken 44 | } 45 | } 46 | ) 47 | .then(response => { 48 | if(response.error) { 49 | dispatch(UploadError()); 50 | throw(response.error); 51 | } 52 | dispatch(UploadSuccess(response.data)); 53 | return response.data; 54 | }) 55 | .catch(error => { 56 | dispatch(UploadError(error)); 57 | }) 58 | } 59 | 60 | } 61 | 62 | export function setSelected(dataset_uuid, idToken=0) { 63 | return dispatch => { 64 | axios.post(`${api}/datasets`, 65 | {dataset_uuid: dataset_uuid[0]}, 66 | { 67 | headers: { 68 | 'Authorization': 'Bearer ' + idToken 69 | } 70 | } 71 | ) 72 | .then(response => { 73 | if (response.error) { 74 | throw(response.error); 75 | } 76 | dispatch(selectDataset(dataset_uuid)); 77 | }) 78 | } 79 | } 80 | 81 | export default function uploadDataset(formData, dataset_uuid=0, idToken=0) { 82 | return dispatch => { 83 | dispatch(UploadPending()); 84 | console.log(formData); 85 | axios.post(`${api}/datasets/${dataset_uuid}`, 86 | formData, 87 | { 88 | headers: { 89 | 'Content-Type': 'multipart/form-data', 90 | 'Authorization': 'Bearer ' + idToken 91 | } 92 | } 93 | ) 94 | .then(response => { 95 | if(response.error) { 96 | dispatch(UploadError()); 97 | throw(response.error); 98 | } 99 | dispatch(UploadSuccess(response.data)); 100 | setSelected(dataset_uuid); 101 | return response.data; 102 | }) 103 | .catch(error => { 104 | dispatch(UploadError(error)); 105 | }) 106 | } 107 | } -------------------------------------------------------------------------------- /ui/src/actions/getStatistics.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | export const GET_STATS_PENDING = "GET_STATS_PENDING"; 4 | export const GET_STATS_SUCCESS = "GET_STATS_SUCCESS"; 5 | export const GET_STATS_ERROR = "GET_STATS_ERROR"; 6 | 7 | const api = process.env.REACT_APP_SERVER; 8 | 9 | export function getStatsPending() { 10 | return { 11 | type: GET_STATS_PENDING 12 | } 13 | } 14 | 15 | function getStatsSuccess(data) { 16 | return { 17 | type: GET_STATS_SUCCESS, 18 | data: data 19 | } 20 | } 21 | 22 | function getStatsError(error) { 23 | return { 24 | type: GET_STATS_ERROR, 25 | error: error 26 | } 27 | } 28 | 29 | function getStatistics() { 30 | return dispatch => { 31 | dispatch(getStatsPending()); 32 | axios.get(`${api}/statistics`) 33 | .then(response => { 34 | if(response.error) { 35 | throw(response.error); 36 | } 37 | dispatch(getStatsSuccess(response.data)); 38 | return response.data; 39 | }) 40 | .catch(error => { 41 | dispatch(getStatsError(error)); 42 | }) 43 | } 44 | } 45 | 46 | export default getStatistics; 47 | 48 | 49 | export const GET_LRSTATS_PENDING = "GET_LRSTATS_PENDING"; 50 | export const GET_LRSTATS_SUCCESS = "GET_LRSTATS_SUCCESS"; 51 | export const GET_LRSTATS_ERROR = "GET_LRSTATS_ERROR"; 52 | 53 | function getLRStatsPending() { 54 | return { 55 | type: GET_LRSTATS_PENDING 56 | } 57 | } 58 | 59 | function getLRStatsSuccess(data) { 60 | return { 61 | type: GET_LRSTATS_SUCCESS, 62 | data: data 63 | } 64 | } 65 | 66 | function getLRStatsError(error) { 67 | return { 68 | type: GET_LRSTATS_ERROR, 69 | error: error 70 | } 71 | } 72 | 73 | export function getLRStatistics() { 74 | return dispatch => { 75 | dispatch(getLRStatsPending()); 76 | axios.get(`${api}/lr_statistics`) 77 | .then(response => { 78 | if(response.error) { 79 | throw(response.error); 80 | } 81 | dispatch(getLRStatsSuccess(response.data)); 82 | return response.data; 83 | }) 84 | .catch(error => { 85 | dispatch(getLRStatsError(error)); 86 | }) 87 | } 88 | } -------------------------------------------------------------------------------- /ui/src/actions/getText.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | import { annotate, select_link, NER } from './annotate' 3 | import { reset_label, label } from "./labelAndSuggestLF"; 4 | 5 | export const GET_TEXT_PENDING = "GET_TEXT_PENDING"; 6 | export const GET_TEXT_SUCCESS = 'GET_TEXT_SUCCESS'; 7 | export const GET_TEXT_ERROR = "GET_TEXT_ERROR"; 8 | 9 | const api = process.env.REACT_APP_SERVER; 10 | 11 | export function getTextPending() { 12 | return { 13 | type: GET_TEXT_PENDING, 14 | } 15 | } 16 | 17 | function newText(data, index){ 18 | return { 19 | type: GET_TEXT_SUCCESS, 20 | data, 21 | index 22 | } 23 | } 24 | 25 | function getTextError(error) { 26 | return { 27 | type: GET_TEXT_ERROR, 28 | error 29 | } 30 | } 31 | 32 | export function getText(){ 33 | return dispatch => { 34 | dispatch(getTextPending()); 35 | axios.get(`${api}/interaction`) 36 | .then(response => { 37 | if (response.error) { 38 | throw(response.error); 39 | } 40 | setInteraction(response, dispatch); 41 | return response.data.text 42 | }) 43 | .catch(error => { 44 | dispatch(getTextError(error)); 45 | }) 46 | } 47 | } 48 | 49 | // Set the new text, and reset all state related to the previous text 50 | export function setInteraction(response, dispatch){ 51 | //change the text 52 | dispatch(newText(response.data.text, response.data.index)); 53 | 54 | //reset annotations 55 | let annotations = []; 56 | if ("annotations" in response.data) { 57 | annotations = response.data.annotations; 58 | } 59 | dispatch(annotate(annotations)); 60 | let ners = []; 61 | if ("NER" in response.data) { 62 | ners = response.data.NER; 63 | } 64 | dispatch(NER(ners)); 65 | 66 | //reset selected span to link 67 | dispatch(select_link({type: null})); 68 | 69 | if ("label" in response.data) { 70 | dispatch(label(response.data)); 71 | } else { 72 | //reset selected label 73 | dispatch(reset_label()); 74 | } 75 | 76 | } -------------------------------------------------------------------------------- /ui/src/actions/interaction.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | import {setInteraction, getTextPending} from './getText' 3 | 4 | export const DELETE_INTERACTION = "DELETE_INTERACTION"; 5 | export const GET_INTERACTION_SUCCESS = "GET_INTERACTION_SUCCESS" 6 | export const GET_INTERACTION_PENDING = "GET_INTERACTION_PENDING" 7 | export const GET_INTERACTION_ERROR = "GET_INTERACTION_ERROR" 8 | 9 | const api = process.env.REACT_APP_SERVER; 10 | 11 | export function deleteInteraction(index) { 12 | return { 13 | type: DELETE_INTERACTION, 14 | index: index 15 | } 16 | } 17 | 18 | function getInteractionError() { 19 | return { 20 | type: GET_INTERACTION_ERROR 21 | } 22 | } 23 | 24 | function getInteractionPending() { 25 | return { 26 | type: GET_INTERACTION_PENDING 27 | } 28 | } 29 | 30 | function getInteractionSuccess(data) { 31 | return { 32 | type: GET_INTERACTION_SUCCESS, 33 | data: data 34 | } 35 | } 36 | 37 | export function getInteraction(index) { 38 | return dispatch => { 39 | dispatch(getInteractionPending()); 40 | axios.get(`${api}/interaction/${index}`) 41 | .then(response => { 42 | if(response.error) { 43 | dispatch(getInteractionError()); 44 | throw(response.error); 45 | } 46 | dispatch(getInteractionSuccess(response.data)); 47 | return response.data; 48 | }) 49 | .catch(error => { 50 | dispatch(getInteractionError(error)); 51 | }) 52 | } 53 | } 54 | 55 | export function setAsCurrentInteraction(index) { 56 | return dispatch => { 57 | dispatch(getTextPending()) 58 | axios.get(`${api}/interaction/${index}`) 59 | .then(response => { 60 | if(response.error) { 61 | dispatch(getInteractionError()); 62 | throw(response.error); 63 | } 64 | setInteraction(response, dispatch); 65 | dispatch(getInteractionSuccess(response.data)); 66 | return response.data; 67 | }) 68 | } 69 | } -------------------------------------------------------------------------------- /ui/src/actions/labelAndSuggestLF.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | export const LABEL = 'LABEL' 4 | export const NEW_LF = 'NEW_LF' 5 | 6 | const api = process.env.REACT_APP_SERVER; 7 | 8 | export function reset_label(){ 9 | return dispatch => { 10 | dispatch({type: LABEL, data: {label: null}}); 11 | } 12 | } 13 | 14 | export function label(data){ 15 | return dispatch => { 16 | dispatch({type: LABEL, data: data}); 17 | axios.post(`${api}/interaction`, data) 18 | .then( response => { 19 | dispatch({type: NEW_LF, data: response.data}) 20 | }) 21 | } 22 | } 23 | 24 | export function set_selected_LF(data){ 25 | return dispatch => dispatch({type: NEW_LF, data: data}); 26 | } 27 | 28 | export function clear_suggestions(){ 29 | return dispatch => dispatch({type: NEW_LF, data: {}}) 30 | } -------------------------------------------------------------------------------- /ui/src/actions/labelClasses.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | export const GET_CLASSES_SUCCESS="GET_CLASSES_SUCCESS"; 4 | export const GET_CLASSES_PENDING="GET_CLASSES_PENDING"; 5 | export const GET_CLASSES_ERROR="GET_CLASSES_ERROR"; 6 | export const ADD_CLASS_SUCCESS="ADD_CLASS_SUCCESS"; 7 | 8 | const api = process.env.REACT_APP_SERVER; 9 | 10 | function pending() { 11 | return { 12 | type: GET_CLASSES_PENDING 13 | } 14 | } 15 | 16 | function getClassesSuccess(data) { 17 | return { 18 | type: GET_CLASSES_SUCCESS, 19 | data: data 20 | } 21 | } 22 | 23 | function raiseError(error) { 24 | return { 25 | type: GET_CLASSES_ERROR, 26 | error: error 27 | } 28 | } 29 | 30 | function addClassSuccess(data) { 31 | return { 32 | type: ADD_CLASS_SUCCESS, 33 | data: data 34 | } 35 | } 36 | 37 | function dataFromResponse(response_data) { 38 | return response_data; 39 | /*return Object.keys(response_data).map(k => { 40 | return { 41 | name: k, 42 | key: response_data[k] 43 | } 44 | })*/ 45 | } 46 | 47 | export function submitLabels(labelClasses) { 48 | return dispatch => { 49 | dispatch(pending()); 50 | axios.post(`${api}/label`, 51 | { 52 | labels: labelClasses, 53 | } 54 | ) 55 | .then(response => { 56 | if (response.error) { 57 | throw(response.error); 58 | } 59 | const data = dataFromResponse(response.data); 60 | dispatch(getClassesSuccess(data)); 61 | }) 62 | .catch(error => { 63 | dispatch(raiseError(error)); 64 | }) 65 | } 66 | } 67 | 68 | export function addLabelClass(labelClassObj) { 69 | return dispatch => { 70 | dispatch(addClassSuccess(labelClassObj)); 71 | } 72 | } 73 | 74 | function fetchClasses() { 75 | return dispatch => { 76 | dispatch(pending()); 77 | axios.get(`${api}/label`) 78 | .then(response => { 79 | if(response.error) { 80 | throw(response.error); 81 | } 82 | var data = dataFromResponse(response.data); 83 | if (data.labels) { 84 | data = data.labels; 85 | } 86 | console.log(data); 87 | dispatch(getClassesSuccess(data)); 88 | }) 89 | .catch(error => { 90 | dispatch(raiseError(error)); 91 | }) 92 | } 93 | } 94 | 95 | export default fetchClasses; -------------------------------------------------------------------------------- /ui/src/actions/loadingBar.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | const api = process.env.REACT_APP_SERVER; 4 | 5 | function updateLoadingBar(value) { 6 | return { 7 | type: "LOADING_BAR", 8 | data: value 9 | } 10 | } 11 | 12 | function setThread(thread) { 13 | return { 14 | type: "SET_LAUNCH_THREAD", 15 | data: thread 16 | } 17 | } 18 | 19 | export function launchStatus(dataset_name=0){ 20 | return dispatch => { 21 | console.log("getting launch status"); 22 | axios.get(`${api}/datasets/${dataset_name}/status`) 23 | .then(response => { 24 | console.log(response); 25 | dispatch(updateLoadingBar(response.data)); 26 | }) 27 | } 28 | } 29 | 30 | export default function launch(){ 31 | return dispatch => { 32 | dispatch(updateLoadingBar(0)); 33 | dispatch(setThread(0)); 34 | axios.post(`${api}/launch`, {}) 35 | .then(response => { 36 | dispatch(setThread(response.data)); 37 | }); 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /ui/src/actions/model.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | export const SUCCESS = "GET_MODELS_SUCCESS" 4 | export const PENDING = "GET_MODELS_PENDING" 5 | export const ERROR = "GET_MODELS_ERROR" 6 | export const SELECT = "MODEL_SELECT" 7 | 8 | const api = process.env.REACT_APP_SERVER; 9 | 10 | function UploadError(error) { 11 | return { 12 | type: ERROR, 13 | error: error 14 | } 15 | } 16 | 17 | function selectModel(data) { 18 | return { 19 | type:SELECT, 20 | data: data 21 | } 22 | } 23 | 24 | function UploadPending() { 25 | return { 26 | type: PENDING 27 | } 28 | } 29 | 30 | function UploadSuccess(data) { 31 | return { 32 | type: SUCCESS, 33 | data: data 34 | } 35 | } 36 | 37 | export function fetchModels(idToken) { 38 | return dispatch => { 39 | dispatch(UploadPending()); 40 | axios.get(`${api}/models`, 41 | { 42 | headers: { 43 | 'Authorization': 'Bearer ' + idToken 44 | } 45 | } 46 | ) 47 | .then(response => { 48 | if(response.error) { 49 | dispatch(UploadError()); 50 | throw(response.error); 51 | } 52 | dispatch(UploadSuccess(response.data)); 53 | return response.data; 54 | }) 55 | .catch(error => { 56 | dispatch(UploadError(error)); 57 | }) 58 | } 59 | } 60 | 61 | export function setSelected(model_name, idToken=0) { 62 | 63 | return dispatch => { 64 | axios.post(`${api}/models`, 65 | {model_name: model_name}, 66 | { 67 | headers: { 68 | 'Authorization': 'Bearer ' + idToken 69 | } 70 | } 71 | ) 72 | .then(response => { 73 | if (response.error) { 74 | throw(response.error); 75 | } 76 | dispatch(selectModel(model_name)); 77 | }) 78 | } 79 | } 80 | 81 | export function createNewModel(model_name=0, formData={}, idToken=0) { 82 | return dispatch => { 83 | dispatch(UploadPending()); 84 | console.log(formData); 85 | axios.post(`${api}/models/${model_name}`, 86 | formData, 87 | { 88 | headers: { 89 | 'Content-Type': 'multipart/form-data', 90 | 'Authorization': 'Bearer ' + idToken 91 | } 92 | } 93 | ) 94 | .then(response => { 95 | if(response.error) { 96 | dispatch(UploadError()); 97 | throw(response.error); 98 | } 99 | dispatch(UploadSuccess(response.data)); 100 | dispatch(selectModel(model_name)); 101 | return response.data; 102 | }) 103 | .catch(error => { 104 | dispatch(UploadError(error)); 105 | }) 106 | } 107 | } -------------------------------------------------------------------------------- /ui/src/actions/save.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | 3 | export const SAVE_ERROR = "SAVE_ERROR" 4 | export const SAVE_PENDING = "SAVE_PENDING" 5 | export const SAVE_SUCCESS = "SAVE_SUCCESS" 6 | 7 | const api = process.env.REACT_APP_SERVER; 8 | 9 | function saveSuccess(data) { 10 | return { 11 | type: SAVE_SUCCESS, 12 | data: data, 13 | } 14 | } 15 | 16 | export function savePending() { 17 | return { 18 | type: SAVE_PENDING 19 | } 20 | } 21 | 22 | function saveError(error) { 23 | return { 24 | type: SAVE_ERROR, 25 | error: error 26 | } 27 | } 28 | 29 | export function saveModel() { 30 | return dispatch => { 31 | axios.post(`${api}/save`) 32 | .then(response => {}); 33 | } 34 | } 35 | 36 | export function uploadModel() { 37 | return dispatch => { 38 | dispatch(savePending()); 39 | var today = new Date(); 40 | var data = {"dirname": (today.getMonth()+1)+'-'+today.getDate()+ "_" + today.getHours() + "-" + today.getMinutes() + "-" + today.getSeconds()} 41 | axios.post(`${api}/save`, data) 42 | .then(response => { 43 | if(response.error) { 44 | throw(response.error); 45 | } 46 | dispatch(saveSuccess(response.data)); 47 | return response.data; 48 | }) 49 | .catch(error => { 50 | dispatch(saveError(error)); 51 | }) 52 | } 53 | } -------------------------------------------------------------------------------- /ui/src/actions/submitLFs.js: -------------------------------------------------------------------------------- 1 | import axios from 'axios'; 2 | import getStatistics from './getStatistics' 3 | 4 | 5 | export const SUBMIT_LF_PENDING = "SUBMIT_LF_PENDING"; 6 | export const ONE_LF_PENDING = "ONE_LF_PENDING"; 7 | export const SUBMIT_LF_SUCCESS = "SUBMIT_LF_SUCCESS"; 8 | export const SUBMIT_LF_ERROR = "SUBMIT_LF_ERROR"; 9 | export const LF_STATS = "LF_STATS"; 10 | export const LF_STATS_ERROR = "LF_STATS_ERROR"; 11 | export const LF_LABEL_EXAMPLES = "LF_LABEL_EXAMPLES"; 12 | 13 | const api = process.env.REACT_APP_SERVER; 14 | 15 | function allLFPending(data) { 16 | if (data) { 17 | return { 18 | type: SUBMIT_LF_PENDING, 19 | data: data 20 | } 21 | } return { type: SUBMIT_LF_PENDING } 22 | 23 | } 24 | 25 | function oneLFPending(lf_id) { 26 | return { 27 | type: ONE_LF_PENDING, 28 | lf_id: lf_id 29 | } 30 | } 31 | 32 | function submitLFSuccess(data) { 33 | return { 34 | type: SUBMIT_LF_SUCCESS, 35 | data: data 36 | } 37 | } 38 | 39 | function submitLFError(error) { 40 | return { 41 | type: SUBMIT_LF_ERROR, 42 | error: error 43 | } 44 | } 45 | 46 | function lfStats(data) { 47 | return { 48 | type: LF_STATS, 49 | data: data 50 | } 51 | } 52 | 53 | function lfStatsError(error) { 54 | return { 55 | type: LF_STATS_ERROR, 56 | error: error 57 | } 58 | } 59 | 60 | export function deleteLF(lf_ids) { 61 | return dispatch => { 62 | dispatch(allLFPending()); 63 | axios.put(`${api}/labelingfunctions`, lf_ids) 64 | .then(response => { 65 | dispatch(getLFstats()); 66 | }) 67 | } 68 | } 69 | 70 | function submitLFs(data) { 71 | return dispatch => { 72 | dispatch(allLFPending(data)); 73 | axios.put(`${api}/interaction`, data) 74 | .then(response => { 75 | if(response.error) { 76 | throw(response.error); 77 | } 78 | dispatch(submitLFSuccess(response.data)); 79 | dispatch(getStatistics()); 80 | return response.data; 81 | }) 82 | .catch(error => { 83 | dispatch(submitLFError(error)); 84 | }) 85 | } 86 | } 87 | 88 | export function getLFstats() { 89 | return dispatch => { 90 | dispatch(allLFPending({})); 91 | axios.get(`${api}/labelingfunctions`) 92 | .then(response => { 93 | if(response.error) { 94 | throw(response.error); 95 | } 96 | dispatch(lfStats(response.data)); 97 | dispatch(getStatistics()); 98 | return response.data; 99 | }) 100 | .catch(error => { 101 | dispatch(lfStatsError(error)); 102 | }) 103 | } 104 | } 105 | 106 | export function getLFexamples(lf_id) { 107 | if (lf_id === null) { 108 | return dispatch => { dispatch({ 109 | type: LF_LABEL_EXAMPLES, 110 | data: {} 111 | })}; 112 | } 113 | return dispatch => { 114 | dispatch(oneLFPending(lf_id)); 115 | axios.get(`${api}/labelingfunctions/${lf_id}`) 116 | .then(response => { 117 | if(response.error) { 118 | throw(response.error); 119 | } 120 | dispatch({ 121 | type: LF_LABEL_EXAMPLES, 122 | data: response.data 123 | }); 124 | return response.data; 125 | }) 126 | .catch(error => { 127 | dispatch(lfStatsError(error)); 128 | }) 129 | } 130 | } 131 | 132 | export default submitLFs; -------------------------------------------------------------------------------- /ui/src/errorSnackbar.js: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import PropTypes from 'prop-types'; 3 | import clsx from 'clsx'; 4 | import CheckCircleIcon from '@material-ui/icons/CheckCircle'; 5 | import ErrorIcon from '@material-ui/icons/Error'; 6 | import InfoIcon from '@material-ui/icons/Info'; 7 | import CloseIcon from '@material-ui/icons/Close'; 8 | import { amber, green } from '@material-ui/core/colors'; 9 | import IconButton from '@material-ui/core/IconButton'; 10 | import Snackbar from '@material-ui/core/Snackbar'; 11 | import SnackbarContent from '@material-ui/core/SnackbarContent'; 12 | import WarningIcon from '@material-ui/icons/Warning'; 13 | import { makeStyles } from '@material-ui/core/styles'; 14 | 15 | const variantIcon = { 16 | success: CheckCircleIcon, 17 | warning: WarningIcon, 18 | error: ErrorIcon, 19 | info: InfoIcon, 20 | }; 21 | 22 | const useStyles1 = makeStyles(theme => ({ 23 | success: { 24 | backgroundColor: green[600], 25 | }, 26 | error: { 27 | backgroundColor: theme.palette.error.dark, 28 | }, 29 | info: { 30 | backgroundColor: theme.palette.primary.main, 31 | }, 32 | warning: { 33 | backgroundColor: amber[700], 34 | }, 35 | icon: { 36 | fontSize: 20, 37 | }, 38 | iconVariant: { 39 | opacity: 0.9, 40 | marginRight: theme.spacing(1), 41 | }, 42 | message: { 43 | display: 'flex', 44 | alignItems: 'center', 45 | }, 46 | })); 47 | 48 | function MySnackbarContentWrapper(props) { 49 | const classes = useStyles1(); 50 | const { className, message, onClose, variant, ...other } = props; 51 | const Icon = variantIcon[variant]; 52 | 53 | return ( 54 | 59 | 60 | {message} 61 | 62 | } 63 | action={[ 64 | 65 | 66 | , 67 | ]} 68 | {...other} 69 | /> 70 | ); 71 | } 72 | 73 | MySnackbarContentWrapper.propTypes = { 74 | className: PropTypes.string, 75 | message: PropTypes.string, 76 | onClose: PropTypes.func, 77 | variant: PropTypes.oneOf(['error', 'info', 'success', 'warning']).isRequired, 78 | }; 79 | 80 | export default function ErrorSnackbar(props) { 81 | let {open, setOpen} = props; 82 | 83 | const handleClose = (event, reason) => { 84 | if (reason === 'clickaway') { 85 | return; 86 | } 87 | 88 | setOpen(false); 89 | }; 90 | 91 | return ( 92 |
93 | 102 | 107 | 108 |
109 | ); 110 | } -------------------------------------------------------------------------------- /ui/src/index.js: -------------------------------------------------------------------------------- 1 | import React from "react"; 2 | import ReactDOM from "react-dom"; 3 | import { Provider } from "react-redux"; 4 | import configureStore from "./store"; 5 | //import './index.css'; 6 | import App from "./App"; 7 | import * as serviceWorker from "./serviceWorker"; 8 | 9 | // const store = configureStore(); 10 | 11 | ReactDOM.render( 12 | 13 | 14 | , 15 | document.getElementById("root") 16 | ); 17 | 18 | // If you want your app to work offline and load faster, you can change 19 | // unregister() to register() below. Note this comes with some pitfalls. 20 | // Learn more about service workers: https://bit.ly/CRA-PWA 21 | serviceWorker.unregister(); 22 | -------------------------------------------------------------------------------- /ui/src/reducers.js: -------------------------------------------------------------------------------- 1 | import {combineReducers} from 'redux'; 2 | 3 | import annotations, { selectedLink } from './annotationsReducer' 4 | import concepts, { selectedConcept } from './conceptsReducer' 5 | import connectivesKeyTypes from './ConnectivesKeyTypesReducer' 6 | import interactionHistory from './interactionHistoryReducer' 7 | import { label, suggestedLF } from './labelAndSuggestLFReducer' 8 | import labelClasses from './labelClassesReducer' 9 | import labelExamples from './labelExampleReducer' 10 | import selectedLF from './selectedLFReducer' 11 | import statistics from './statisticsReducer' 12 | import text from './textReducer' 13 | 14 | const rootReducer = combineReducers({ 15 | annotations, 16 | concepts, 17 | gll: connectivesKeyTypes, 18 | interactionHistory, 19 | label, 20 | labelClasses, 21 | labelExamples, 22 | selectedConcept, 23 | selectedLF, 24 | selectedLink, 25 | suggestedLF, 26 | statistics, 27 | text 28 | }); 29 | 30 | export default rootReducer; 31 | 32 | -------------------------------------------------------------------------------- /ui/src/reducers/ConnectivesKeyTypesReducer.js: -------------------------------------------------------------------------------- 1 | import { CONNECTIVE, KEYTYPE } from '../actions/connectivesAndKeyTypes' 2 | 3 | const initialState = { 4 | fetched_conn: false, 5 | fetched_keytype: false, 6 | connective: {}, 7 | keyType: {}, 8 | } 9 | 10 | export default function connectivesKeyTypesReducer(state=initialState, action) { 11 | switch (action.type) { 12 | case CONNECTIVE: 13 | return {...state, connective: action.data, fetched_conn: true} 14 | case KEYTYPE: 15 | return {...state, keyType: action.data, fetched_keytype: true} 16 | default: 17 | return state 18 | } 19 | } -------------------------------------------------------------------------------- /ui/src/reducers/annotationsReducer.js: -------------------------------------------------------------------------------- 1 | import { ANNOTATE, HIGHLIGHT, HIGHLIGHT_ERROR, SELECT_LINK, ADD_NER } from '../actions/annotate'; 2 | 3 | function annotations(state=[], action ){ 4 | switch (action.type) { 5 | case ANNOTATE: 6 | return action.data 7 | default: 8 | return state 9 | } 10 | } 11 | 12 | export default annotations; 13 | 14 | export function highlights(state={data: [], error: {}}, action ){ 15 | switch (action.type) { 16 | case HIGHLIGHT: 17 | return { 18 | ...state, 19 | data: action.data, 20 | error: {}, 21 | idx: null 22 | } 23 | case HIGHLIGHT_ERROR: 24 | var newErr = state.error; 25 | newErr[action.idx] = action.error; 26 | return { 27 | ...state, 28 | error: newErr 29 | } 30 | default: 31 | return state 32 | } 33 | } 34 | 35 | export function ners(state=[], action) { 36 | switch (action.type) { 37 | case ADD_NER: 38 | return action.data 39 | default: 40 | return state 41 | } 42 | } 43 | 44 | export function selectedLink(state={type: null}, action) { 45 | switch (action.type) { 46 | case SELECT_LINK: 47 | return action.data 48 | default: 49 | return state 50 | } 51 | } -------------------------------------------------------------------------------- /ui/src/reducers/conceptsReducer.js: -------------------------------------------------------------------------------- 1 | 2 | import { 3 | GET_CONCEPTS_PENDING, 4 | UPDATE_CONCEPT_PENDING, 5 | GET_CONCEPTS_SUCCESS, 6 | GET_CONCEPTS_ERROR, 7 | SELECT_CONCEPT 8 | } from '../actions/concepts' 9 | 10 | const initialState = { 11 | pending: false, 12 | data: {}, 13 | error: null 14 | } 15 | 16 | function concepts (state = initialState, action ){ 17 | switch (action.type) { 18 | case GET_CONCEPTS_PENDING: 19 | return { 20 | ...state, 21 | data: {...state.data, ...action.data}, 22 | pending: true 23 | } 24 | case UPDATE_CONCEPT_PENDING: 25 | const conceptName = action.conceptName; 26 | state.data[conceptName].pending = true; 27 | return { 28 | ...state, 29 | data: state.data, 30 | pending: false 31 | } 32 | case GET_CONCEPTS_SUCCESS: 33 | return { 34 | ...state, 35 | data: action.data, 36 | pending: false, 37 | } 38 | case GET_CONCEPTS_ERROR: 39 | return { 40 | ...state, 41 | pending: false, 42 | error: action.error 43 | } 44 | default: 45 | return state 46 | } 47 | } 48 | 49 | export default concepts; 50 | 51 | export function selectedConcept(state=null, action) { 52 | switch (action.type) { 53 | case SELECT_CONCEPT: 54 | return action.data 55 | //{...state, ...action.data} 56 | default: 57 | return state 58 | } 59 | } 60 | -------------------------------------------------------------------------------- /ui/src/reducers/datasetsReducer.js: -------------------------------------------------------------------------------- 1 | import { SUCCESS, ERROR, SELECT } from '../actions/datasets' 2 | 3 | const initialState = { 4 | error: null, 5 | data: [], 6 | selected: undefined 7 | } 8 | 9 | export default function datasetsReducer(state=initialState, action) { 10 | switch(action.type) { 11 | case SUCCESS: 12 | return {...state, 13 | data: action.data} 14 | case ERROR: 15 | return { 16 | ...state, 17 | error: action.error} 18 | case SELECT: 19 | return { 20 | ...state, 21 | selected: action.data 22 | } 23 | default: 24 | return state; 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /ui/src/reducers/interactionHistoryReducer.js: -------------------------------------------------------------------------------- 1 | 2 | import { 3 | GET_INTERACTION_SUCCESS, 4 | GET_INTERACTION_PENDING, 5 | GET_INTERACTION_ERROR 6 | } from '../actions/interaction' 7 | 8 | const initialState = { 9 | pending: false, 10 | data: {}, 11 | error: null 12 | } 13 | 14 | function interactionHistory(state = initialState, action ){ 15 | switch (action.type) { 16 | case GET_INTERACTION_PENDING: 17 | return { 18 | ...state, 19 | data: {}, 20 | pending: true 21 | } 22 | case GET_INTERACTION_SUCCESS: 23 | return { 24 | ...state, 25 | data: action.data, 26 | pending: false, 27 | } 28 | case GET_INTERACTION_ERROR: 29 | return { 30 | ...state, 31 | pending: false, 32 | error: action.error 33 | } 34 | default: 35 | return state 36 | } 37 | } 38 | 39 | export default interactionHistory; -------------------------------------------------------------------------------- /ui/src/reducers/labelAndSuggestLFReducer.js: -------------------------------------------------------------------------------- 1 | import { 2 | NEW_LF, 3 | LABEL, 4 | } from '../actions/labelAndSuggestLF'; 5 | 6 | 7 | export function label (state = null, action ) { 8 | switch (action.type) { 9 | case LABEL: 10 | return action.data.label 11 | default: 12 | return state 13 | } 14 | } 15 | 16 | export function suggestedLF (state = {}, action) { 17 | switch (action.type) { 18 | case NEW_LF: 19 | /* Keep already selected LFs */ 20 | const LF_names = Object.keys(state); 21 | var already_selected_lfs = {}; 22 | for (var i = LF_names.length - 1; i >= 0; i--) { 23 | let lf_id = LF_names[i]; 24 | let lf = state[lf_id]; 25 | if (lf.selected) { 26 | already_selected_lfs[lf_id] = lf 27 | } 28 | } 29 | return { 30 | ...action.data, 31 | ...already_selected_lfs 32 | }; 33 | default: 34 | return state; 35 | } 36 | } -------------------------------------------------------------------------------- /ui/src/reducers/labelClassesReducer.js: -------------------------------------------------------------------------------- 1 | import { 2 | GET_CLASSES_SUCCESS, 3 | GET_CLASSES_PENDING, 4 | GET_CLASSES_ERROR, 5 | ADD_CLASS_SUCCESS 6 | } from '../actions/labelClasses' 7 | 8 | const initialState = { 9 | pending: true, 10 | data: {}, 11 | error: null 12 | } 13 | 14 | function labelClassesReducer(state=initialState, action){ 15 | switch (action.type) { 16 | case GET_CLASSES_SUCCESS: 17 | return { 18 | ...state, 19 | data: action.data, 20 | pending: false 21 | } 22 | case GET_CLASSES_ERROR: 23 | return { 24 | ...state, 25 | error: action.error, 26 | pending: false 27 | } 28 | case GET_CLASSES_PENDING: 29 | return { 30 | ...state, 31 | pending: true 32 | } 33 | case ADD_CLASS_SUCCESS: 34 | return { 35 | ...state, 36 | data: {...state.data, ...action.data}, 37 | pending: false 38 | } 39 | default: 40 | return state; 41 | } 42 | } 43 | 44 | export default labelClassesReducer; -------------------------------------------------------------------------------- /ui/src/reducers/labelExampleReducer.js: -------------------------------------------------------------------------------- 1 | import { LF_LABEL_EXAMPLES } from '../actions/submitLFs' 2 | 3 | const initialState = { 4 | } 5 | 6 | export default function labelExamples(state=initialState, action) { 7 | switch(action.type) { 8 | case LF_LABEL_EXAMPLES: 9 | return action.data 10 | default: 11 | return state; 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /ui/src/reducers/loadingBarReducer.js: -------------------------------------------------------------------------------- 1 | export default function loadingBarReducer(state={progress: 0, thread: null}, action) { 2 | switch (action.type) { 3 | case "LOADING_BAR": 4 | if (action.data < 1.0) { 5 | return { 6 | ...state, 7 | progress: action.data, 8 | } 9 | } else { 10 | return { 11 | ...state, 12 | progress: action.data, 13 | thread: null 14 | } 15 | } 16 | case "SET_LAUNCH_THREAD": 17 | return { 18 | ...state, 19 | thread: action.data 20 | } 21 | default: 22 | return state 23 | } 24 | } -------------------------------------------------------------------------------- /ui/src/reducers/modelsReducer.js: -------------------------------------------------------------------------------- 1 | import { SUCCESS, ERROR, SELECT } from '../actions/model' 2 | 3 | const initialState = { 4 | error: null, 5 | data: [], 6 | selected: undefined 7 | } 8 | 9 | export default function modelsReducer(state=initialState, action) { 10 | switch(action.type) { 11 | case SUCCESS: 12 | return {...state, 13 | data: action.data} 14 | case ERROR: 15 | return { 16 | ...state, 17 | error: action.error} 18 | case SELECT: 19 | return { 20 | ...state, 21 | selected: action.data 22 | } 23 | default: 24 | return state; 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /ui/src/reducers/reducers.js: -------------------------------------------------------------------------------- 1 | import {combineReducers} from 'redux'; 2 | 3 | // custom reducers 4 | import annotations, { highlights, selectedLink, ners } from './annotationsReducer' 5 | import concepts, { selectedConcept } from './conceptsReducer' 6 | import connectivesKeyTypes from './ConnectivesKeyTypesReducer' 7 | import datasets from './datasetsReducer' 8 | import { label, suggestedLF } from './labelAndSuggestLFReducer' 9 | import labelClasses from './labelClassesReducer' 10 | import labelExamples from './labelExampleReducer' 11 | import loadingBarReducer from './loadingBarReducer' 12 | import modelFile from './saveFileReducer' 13 | import models from './modelsReducer' 14 | import selectedLF from './selectedLFReducer' 15 | import statistics, {statistics_LRmodel} from './statisticsReducer' 16 | import text from './textReducer' 17 | 18 | const rootReducer = combineReducers({ 19 | annotations, 20 | concepts, 21 | datasets, 22 | gll: connectivesKeyTypes, 23 | highlights, 24 | label, 25 | labelClasses, 26 | labelExamples, 27 | launchProgress: loadingBarReducer, 28 | modelFile, 29 | models, 30 | ners, 31 | selectedConcept, 32 | selectedLF, 33 | selectedLink, 34 | suggestedLF, 35 | statistics, 36 | statistics_LRmodel, 37 | text 38 | }); 39 | 40 | export default rootReducer; 41 | 42 | -------------------------------------------------------------------------------- /ui/src/reducers/saveFileReducer.js: -------------------------------------------------------------------------------- 1 | import { SAVE_SUCCESS, SAVE_PENDING, SAVE_ERROR} from '../actions/save' 2 | 3 | export default function projectReducer(state = {}, action) 4 | { 5 | switch (action.type) { 6 | case SAVE_ERROR : 7 | return {...state, 8 | error: action.error}; 9 | default: 10 | return state; 11 | } 12 | return state; 13 | 14 | } -------------------------------------------------------------------------------- /ui/src/reducers/selectedLFReducer.js: -------------------------------------------------------------------------------- 1 | import { 2 | SUBMIT_LF_PENDING, 3 | SUBMIT_LF_SUCCESS, 4 | SUBMIT_LF_ERROR, 5 | ONE_LF_PENDING, 6 | LF_STATS, 7 | LF_STATS_ERROR 8 | } from '../actions/submitLFs' 9 | 10 | const initialState = { 11 | pending: false, 12 | data: {}, 13 | error: null 14 | } 15 | 16 | function updateData(old_data, new_data) { 17 | old_data = {...old_data, ...new_data} 18 | return Object.keys(old_data).reduce(function(updated_data, key) { 19 | updated_data[key] = old_data[key]; 20 | return updated_data; 21 | }, {}); 22 | } 23 | 24 | export default function selectedLFReducer(state=initialState, action) { 25 | switch (action.type) { 26 | case SUBMIT_LF_PENDING: 27 | return { 28 | ...state, 29 | data: {...state.data, ...action.data}, 30 | pending: true 31 | } 32 | case ONE_LF_PENDING: 33 | return state // TODO tell redux it's loading 34 | case SUBMIT_LF_SUCCESS: 35 | return { 36 | ...state, 37 | data: updateData(state.data, action.data), 38 | pending: false, 39 | stats: false 40 | } 41 | case LF_STATS: 42 | return { 43 | data: action.data, 44 | pending: false, 45 | stats: true 46 | } 47 | case LF_STATS_ERROR: 48 | return { 49 | ...state, 50 | pending: false, 51 | error: action.error 52 | } 53 | case SUBMIT_LF_ERROR: 54 | return { 55 | ...state, 56 | pending: false, 57 | error: action.error 58 | } 59 | default: 60 | return state; 61 | } 62 | } -------------------------------------------------------------------------------- /ui/src/reducers/statisticsReducer.js: -------------------------------------------------------------------------------- 1 | import { GET_STATS_PENDING, GET_STATS_SUCCESS, GET_STATS_ERROR } from '../actions/getStatistics' 2 | import { GET_LRSTATS_PENDING, GET_LRSTATS_SUCCESS, GET_LRSTATS_ERROR } from '../actions/getStatistics' 3 | 4 | 5 | const initialState = { 6 | pending: false, 7 | data: {}, 8 | error: null 9 | } 10 | 11 | export default function statisticsReducer(state=initialState, action) { 12 | switch (action.type) { 13 | case GET_STATS_PENDING: 14 | return { 15 | ...state, 16 | pending: true 17 | } 18 | case GET_STATS_SUCCESS: 19 | return { 20 | ...state, 21 | data: action.data, 22 | pending: false 23 | } 24 | case GET_STATS_ERROR: 25 | return { 26 | ...state, 27 | pending: false, 28 | error: action.error 29 | } 30 | default: 31 | return state; 32 | } 33 | } 34 | 35 | 36 | export function statistics_LRmodel(state=initialState, action) { 37 | switch (action.type) { 38 | case GET_LRSTATS_PENDING: 39 | return { 40 | ...state, 41 | pending: true 42 | } 43 | case GET_LRSTATS_SUCCESS: 44 | return { 45 | ...state, 46 | data: action.data, 47 | pending: false 48 | } 49 | case GET_LRSTATS_ERROR: 50 | return { 51 | ...state, 52 | pending: false, 53 | error: action.error 54 | } 55 | default: 56 | return state; 57 | } 58 | } -------------------------------------------------------------------------------- /ui/src/reducers/textReducer.js: -------------------------------------------------------------------------------- 1 | import { 2 | GET_TEXT_PENDING, 3 | GET_TEXT_SUCCESS, 4 | GET_TEXT_ERROR 5 | } from '../actions/getText'; 6 | 7 | const initialState = { 8 | pending: false, 9 | data: "", 10 | error: null 11 | } 12 | 13 | function text (state =initialState, action ){ 14 | switch (action.type) { 15 | case GET_TEXT_PENDING: 16 | return { 17 | ...state, 18 | pending: true 19 | } 20 | case GET_TEXT_SUCCESS: 21 | return { 22 | ...state, 23 | data: action.data, 24 | index: action.index, 25 | pending: false 26 | } 27 | case GET_TEXT_ERROR: 28 | return { 29 | ...state, 30 | pending: false, 31 | error: action.error 32 | } 33 | default: 34 | return state; 35 | } 36 | } 37 | 38 | export default text; -------------------------------------------------------------------------------- /ui/src/serviceWorker.js: -------------------------------------------------------------------------------- 1 | // This optional code is used to register a service worker. 2 | // register() is not called by default. 3 | 4 | // This lets the app load faster on subsequent visits in production, and gives 5 | // it offline capabilities. However, it also means that developers (and users) 6 | // will only see deployed updates on subsequent visits to a page, after all the 7 | // existing tabs open on the page have been closed, since previously cached 8 | // resources are updated in the background. 9 | 10 | // To learn more about the benefits of this model and instructions on how to 11 | // opt-in, read https://bit.ly/CRA-PWA 12 | 13 | const isLocalhost = Boolean( 14 | window.location.hostname === 'localhost' || 15 | // [::1] is the IPv6 localhost address. 16 | window.location.hostname === '[::1]' || 17 | // 127.0.0.1/8 is considered localhost for IPv4. 18 | window.location.hostname.match( 19 | /^127(?:\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}$/ 20 | ) 21 | ); 22 | 23 | export function register(config) { 24 | if (process.env.NODE_ENV === 'production' && 'serviceWorker' in navigator) { 25 | // The URL constructor is available in all browsers that support SW. 26 | const publicUrl = new URL(process.env.PUBLIC_URL, window.location.href); 27 | if (publicUrl.origin !== window.location.origin) { 28 | // Our service worker won't work if PUBLIC_URL is on a different origin 29 | // from what our page is served on. This might happen if a CDN is used to 30 | // serve assets; see https://github.com/facebook/create-react-app/issues/2374 31 | return; 32 | } 33 | 34 | window.addEventListener('load', () => { 35 | const swUrl = `${process.env.PUBLIC_URL}/service-worker.js`; 36 | 37 | if (isLocalhost) { 38 | // This is running on localhost. Let's check if a service worker still exists or not. 39 | checkValidServiceWorker(swUrl, config); 40 | 41 | // Add some additional logging to localhost, pointing developers to the 42 | // service worker/PWA documentation. 43 | navigator.serviceWorker.ready.then(() => { 44 | console.log( 45 | 'This web app is being served cache-first by a service ' + 46 | 'worker. To learn more, visit https://bit.ly/CRA-PWA' 47 | ); 48 | }); 49 | } else { 50 | // Is not localhost. Just register service worker 51 | registerValidSW(swUrl, config); 52 | } 53 | }); 54 | } 55 | } 56 | 57 | function registerValidSW(swUrl, config) { 58 | navigator.serviceWorker 59 | .register(swUrl) 60 | .then(registration => { 61 | registration.onupdatefound = () => { 62 | const installingWorker = registration.installing; 63 | if (installingWorker == null) { 64 | return; 65 | } 66 | installingWorker.onstatechange = () => { 67 | if (installingWorker.state === 'installed') { 68 | if (navigator.serviceWorker.controller) { 69 | // At this point, the updated precached content has been fetched, 70 | // but the previous service worker will still serve the older 71 | // content until all client tabs are closed. 72 | console.log( 73 | 'New content is available and will be used when all ' + 74 | 'tabs for this page are closed. See https://bit.ly/CRA-PWA.' 75 | ); 76 | 77 | // Execute callback 78 | if (config && config.onUpdate) { 79 | config.onUpdate(registration); 80 | } 81 | } else { 82 | // At this point, everything has been precached. 83 | // It's the perfect time to display a 84 | // "Content is cached for offline use." message. 85 | console.log('Content is cached for offline use.'); 86 | 87 | // Execute callback 88 | if (config && config.onSuccess) { 89 | config.onSuccess(registration); 90 | } 91 | } 92 | } 93 | }; 94 | }; 95 | }) 96 | .catch(error => { 97 | console.error('Error during service worker registration:', error); 98 | }); 99 | } 100 | 101 | function checkValidServiceWorker(swUrl, config) { 102 | // Check if the service worker can be found. If it can't reload the page. 103 | fetch(swUrl) 104 | .then(response => { 105 | // Ensure service worker exists, and that we really are getting a JS file. 106 | const contentType = response.headers.get('content-type'); 107 | if ( 108 | response.status === 404 || 109 | (contentType != null && contentType.indexOf('javascript') === -1) 110 | ) { 111 | // No service worker found. Probably a different app. Reload the page. 112 | navigator.serviceWorker.ready.then(registration => { 113 | registration.unregister().then(() => { 114 | window.location.reload(); 115 | }); 116 | }); 117 | } else { 118 | // Service worker found. Proceed as normal. 119 | registerValidSW(swUrl, config); 120 | } 121 | }) 122 | .catch(() => { 123 | console.log( 124 | 'No internet connection found. App is running in offline mode.' 125 | ); 126 | }); 127 | } 128 | 129 | export function unregister() { 130 | if ('serviceWorker' in navigator) { 131 | navigator.serviceWorker.ready.then(registration => { 132 | registration.unregister(); 133 | }); 134 | } 135 | } 136 | -------------------------------------------------------------------------------- /ui/src/store.js: -------------------------------------------------------------------------------- 1 | import { createStore, applyMiddleware, compose } from 'redux'; 2 | import thunk from 'redux-thunk'; 3 | import rootReducer from './reducers/reducers'; 4 | 5 | 6 | const composeEnhancers = window.__REDUX_DEVTOOLS_EXTENSION_COMPOSE__ || compose; 7 | 8 | export default function configureStore() { 9 | return createStore( 10 | rootReducer, 11 | composeEnhancers(applyMiddleware(thunk)) 12 | ); 13 | } 14 | -------------------------------------------------------------------------------- /user_study/README.md: -------------------------------------------------------------------------------- 1 | # Evaluation of Ruler 2 | We evaluated Ruler alongside manual data programming using [Snorkel](https://www.snorkel.org/). 3 | 4 | Although non-programmer domain experts are a target audience for this technology, we wanted our evaluation to show that the Ruler labeling language is expressive enough to create models comparable to manual data programming. 5 | We also wanted to understand the trade-offs afforded by each method in order to help programming-proficient users decide which is best for their situation. 6 | In order to make these comparisons, we conducted a user study with 10 programming-proficient data scientists and measured their task performance accuracy in completing two labeling tasks using the two methods. 7 | In addition to task performance, we analyzed both accessibility and expressivity using the qualitative feedback elicited from participants. 8 | 9 | We asked participants to write labeling functions for two prevalent labeling tasks: spam detection and sentiment classification. Each user completed both tasks using different methods, for a within-subjects experimental design. 10 | The pairing of task/tool, as well as the order in which tasks were completed, were counterbalanced. 11 | Each session included a 15 minute tutorial on the tool, followed by 30 minutes to create as many functions as they considered necessary for the goal of the task. 12 | 13 |

14 | The data from our user study is available in this folder, along with the code to generate all of the figures included in our Findings of EMNLP paper. 15 |

16 | 17 | All participants had significant programming experience (avg=12.1 years, std=6.5). 18 | Their experience with Python ranged from 2 to 10 years with an average of 5.2 years (std=2.8). 19 | 20 | We find that Ruler and Snorkel provide comparable model performances (see figure below). 21 | The logistic regression models trained on data produced by labeling models created using Ruler have slightly higher f1 (W=35, p=0.49, r=0.24 ), precision (W=30, p=0.85, r=0.08), and recall (W=25, p=0.85, r=0.08) scores on average. 22 | Conversely, accuracy is slightly higher (W=17, p=0.32, r=0.15) for Snorkel models on average than Ruler. 23 | However these differences are not statistically significant. 24 | 25 |

26 | 27 |
Ruler and Snorkel provide comparable model performances 28 |

29 | 30 | 31 | Participants find Ruler to be significantly easier to use 32 | (W=34, p=0.03 < 0.05, r=0.72) than Snorkel. 33 | Similarly, they consider Ruler easier to learn (W=30, p=0.1, r=0.59) than Snorkel. 34 | On the other hand, as we expected, participants report Snorkel to be more expressive (W=0, p=0.05, r=0.70) than Ruler. 35 | However, our participants appear to consider accessibility (ease of use and ease of learning) to be more important criteria, rating Ruler higher (W=43, p=0.12, r=0.51) than Snorkel for overall satisfaction. 36 | 37 | 38 |

39 | 40 |
Participants' subjective ratings on ease of use, expressivity, ease of learning and overall satisfaction, on a 5-point Likert scale. 41 |

42 | 43 | Please see our [EMNLP'20 submission](https://github.com/megagonlabs/ruler/blob/main/media/Ruler_EMNLP2020.pdf) for more details. 44 | -------------------------------------------------------------------------------- /user_study/background_survey_anon.csv: -------------------------------------------------------------------------------- 1 | ,Timestamp,ruler_task,first_condition,snorkel_task,programming experience (yrs),python experience (yrs),regex experience (yrs),Have you ever used Snorkel to label data?,Have you ever used Babble Labble to label data?,How much data labeling experience do you have? ,Have you trained models that required labeled data?,Are you familiar with data programming?,"If you've tried data programming before, how and why?","If you have NOT tried data programming, why not?",participant 2 | 0,5/22/2020 18:03:23,Y,ruler,A,15,10,14,No,No,"100,000 labels of crowdsourcing, 10 years ago when I first hand labeled, many machine learning projects that I do which I did labeling",Yes,I've read papers about it,,Seems too difficult/too much time,p7 3 | 1,5/26/2020 11:27:18,Y,ruler,A,4,4,4,No,No,"Four years, and too many projects to count. I have typically done some labels myself to develop a criteria and then crowdsourced the rest, so all projects have utilized crowdsourcing.",Yes,I've read papers about it,,"Seems too difficult/too much time, My data science team had considered it for two projects but ultimately decided against learning how to use it due to development time constraints.",p3 4 | 5,5/13/2020 18:15:56,Y,snorkel,A,25,4,15,No,No,10 years,yes,I've read papers and also tried it,used in past projects,,p4 5 | 6,5/26/2020 12:51:29,Y,snorkel,A,12,6,5,No,No,1 undergrad project about gender classification of facial images. I labeled 200 images.,Yes,I've read papers about it,,Not relevant to any of my projects,p9 6 | 7,5/22/2020 19:10:21,A,ruler,Y,9,3,0,No,No,"1.5 years, 2 projects, 30k+ labels done myself, 3k+ labels crowdsourced ",Yes,I've read papers about it,Run Snorkel tutorial,Not relevant to any of my projects,p6 7 | 9,5/21/2020 15:57:40,Y,ruler,A,6,3,1,No,No,"Never manually labeled any data, but used labeled data for classification purposes in 2-3 projects",yes,"Heard of it, but I'm not sure what it is",,"I may have used it for preprocessing sometimes, but never needed more than that.",p5 8 | 10,5/12/2020 20:07:12,A,ruler,Y,7,4,6,No,No,"2 projects, 500 labels I made in total",yes,"Heard of it, but I'm not sure what it is",,Not relevant to any of my projects,p0 9 | 11,5/14/2020 15:19:59,A,snorkel,Y,10,2,5,No,No,"2 years, 3+ projects, 2k-5k labels, 10k labels from crowdsourcing platform",yes,I've read papers about it,,Seems too difficult/too much time,p1 10 | 12,5/14/2020 15:12:32,A,snorkel,Y,20,10,15,No,No,Around 10 years. 10+ projects. Several thousands labels by myself. Designed and published 20+ crowdsourcing tasks.,Yes,I've read papers about it,,"Seems too difficult/too much time, Rather directly designed features and/or selected borderline examples by active learning for additional labels. ",p2 11 | 14,5/26/2020 15:04:17,A,snorkel,Y,13,6,7,No,No,1 year. 1 project. ~500 labels by meslf.,Yes.,I've read papers about it,,Not relevant to any of my projects,p8 12 | -------------------------------------------------------------------------------- /user_study/final_survey_anon.csv: -------------------------------------------------------------------------------- 1 | ,Timestamp,"Overall, which tool did you prefer and why?","What advantages, if any, did Ruler have over Snorkel?","What advantages, if any, did Snorkel have over Ruler?",Do you have any feedback about how the study was conducted?,Any other comments?,participant 2 | 0,5/19/2020 17:36:26,"I liked them both. I think that I would probably use Ruler if I wanted to get results quickly, unless I already had a template for Snorkel like the Jupyter Notebook we used in the study. I do often like to have complete control over things, which is why I may favor the Snorkel experience in some cases, but Ruler was definitely easier to use. I think I would opt to use Ruler in most cases (particularly, simpler cases), but I would want to use Snorkel when hoping to create more complex labeling functions that utilize extraneous data like sentiment corpus, model output, etc.",The Ruler UI was definitely more intuitive and could enable a user to save time. I think that an extension that tracks the performance over time and enables the user to select a time-specific model would add another important benefit to Ruler.,"In Snorkel, you can define any function you would like. Here are the three main useful function attributes I can think of you can utilize in Snorkel but not in Ruler: 3 | 1. You can take extraneous data into account (like a token sentiment corpus, output from a sentence-level sentiment scoring model, topic labels, etc.). 4 | 2. You can also use logic at multiple levels (token, n-gram, multi-span, sentence, and document). In Ruler, you can only use logic at the token, n-gram, multi-span levels, and I am not immediately certain that you can use logic from all levels in tandem within a function. 5 | 3. You can utilize meta-analysis of the text. 6 | 7 | Here is an example of a complex function that uses extraneous data at multiple text levels: return POSITIVE if topic (extraneous data/model output, document-specific) == iPhone and mean_sentence_length_chars > 100 (meta-analysis of the sentences) and mean_vader_sentiment (extraneous data, token-specific) and review_likes > 10 (extraneous metadata, document_specific) 8 | 9 | However, I think that these could be integrated into Ruler, which is already a superior product for many use cases and for many individuals.",The study was great! I really enjoyed it. I hope we do more like it.,I feel like I will use both tools for future projects.,p3 10 | 1,5/19/2020 22:47:37,"Snorkel because I'm failiar with jupytyer. Ruler was also great, but initially I found it hard to operate.",No programming is necessary.,Powerful expression of labeling functions.,The turorials are very convincing. ,"I think I did better in Snorkel than in Ruler, but it was possibly because for me some of the Amazon reviews were hard to understand due to my poor English.... Anyway, it was great exercise also for me. Thank you!",p0 11 | 2,5/21/2020 15:01:53,Ruler,"Simple label function that rely on keywords are much easier and faster to write with Ruler. For both tasks, I did not write complex label logic, so with the same time, I can write more label functions with Ruler. ","Snorkel accept any python method, thus users can still write complex logics (e.g., logic that reply on dependency parsing or other NLP packages like NLTK or Spacy) with Snorkel.",,,p1 12 | 3,5/22/2020 1:21:39,"- I prefer Ruler if I have a limited amount of time, say ~1 hour. It's definitely easier to use. 13 | - I'd slightly prefer using Snorkel if I had more than 2-3 hours. My impression is that there could be no way left to improve the model with Ruler after 10-20 labeling functions. The same thing may be said to Snorkel. But, given the fact that Snorkel offers more flexibility, there might be a chance to make a better model with Snorkel (if I had enough time.) 14 | - In any case, I'd jump into doing by myself if I had more than a half day. In my opinion, Ruler's best use-case is when there is a limited amount of time and the user lacks skills in programming.",- 1) Much better UI - 2) Easier to create labeling functions. The user doesn't have to write python code - 3) Easier to monitor dev set performance - 1+2+3) Easier to run the label creation/modification loop ,"- 1) More flexible. A labeling function is a Python function. I could even use an external classifier to build a labeling function. 15 | - 2) Can apply any rules, independent of the example I see.","As I mentioned in the last survey, the order of the tools may affect the performance and impression. It should be randomly shuffled for a fair comparison.",,p2 16 | 4,5/22/2020 16:23:10,"It depends on what is your goal. I would prefer using Ruler in the following situations: 17 | 1) I want to quickly generate labeled examples without diving deep in very specific characteristics of the dataset; 18 | 2) I do not have much knowledge about the data, so I would benefit from creating the rules based on what I see while browsing examples; 19 | 3) I'm not familiar with the programming language used in Snorkel 20 | 21 | And I would prefer using snorkel (or a programming-language-based tool) if I need to add more complex/specific characteristics of my data/problem to the rules, or even in cases when I want to label my examples based on ""non-symbolic"" (not based on rules) models. ","Simplicity, easiness to use, speed in achieving good performance.",Allows using more complex labelers.,,,p4 22 | 5,5/26/2020 14:05:03,"Snorkel, although it needs much more time to come up with good functions, you can do anything you want and use any packages or libraries (which in my opinion, is an advantage)",It was much simpler...maybe more suitable for data that can be labeled with very specific patterns ,You could implement much more complicated functions,,,p5 23 | 6,5/27/2020 16:37:12,"Hard to say, both tools have strengths and weaknesses. ",1. coding free 2. instant statistics,1. support python,Spam task seems easier than Sentiment task. ,,p6 24 | 7,5/28/2020 0:56:26,Ruler,Creating rules by example easily ,Better control,,,p7 25 | 8,5/28/2020 16:10:18,"Ruler. Generally speaking I prefer the one with an interactive interface, which allows me to directly generate LF from the example. ",Showing the result of regex on the example; NER; no need to code for simple LFs; automatically reevaluation.,The flexibility in combining and editing LFs.,As I gave in the Ruler survey.,,p8 26 | 9,5/28/2020 18:33:22,"Ruler. I could generate reasonable labeling rules with high coverages with the help of linking, concept-building interfaces. ",Ruler saves significant time compared Snorkel because it provides functionalities to build concepts incrementally and link between tokens.,Snorkel provides a higher degree of freedom of labeling functions,,,p9 27 | --------------------------------------------------------------------------------