├── Intent_Classification_Keras_Glove.ipynb ├── README.md ├── data ├── .gitignore ├── label_encoder.pkl ├── processed_data │ ├── AddToPlaylist │ │ ├── AddToPlaylist_test.txt │ │ └── AddToPlaylist_train.txt │ ├── AddToPlaylist_test.txt │ ├── AddToPlaylist_train.txt │ ├── BookRestaurant │ │ ├── BookRestaurant_test.txt │ │ └── BookRestaurant_train.txt │ ├── BookRestaurant_test.txt │ ├── BookRestaurant_train.txt │ ├── GetWeather │ │ ├── GetWeather_test.txt │ │ └── GetWeather_train.txt │ ├── GetWeather_test.txt │ ├── GetWeather_train.txt │ ├── RateBook │ │ ├── RateBook_test.txt │ │ └── RateBook_train.txt │ ├── RateBook_test.txt │ ├── RateBook_train.txt │ ├── SearchCreativeWork │ │ ├── SearchCreativeWork_test.txt │ │ └── SearchCreativeWork_train.txt │ ├── SearchCreativeWork_test.txt │ ├── SearchCreativeWork_train.txt │ ├── SearchScreeningEvent │ │ ├── SearchScreeningEvent_test.txt │ │ └── SearchScreeningEvent_train.txt │ ├── SearchScreeningEvent_test.txt │ └── SearchScreeningEvent_train.txt ├── raw_json_data │ ├── AddToPlaylist │ │ ├── Alexa_metrics.json │ │ ├── Luis_metrics.json │ │ ├── Snips_metrics.json │ │ ├── Snips_metrics_full.json │ │ ├── Wit_metrics.json │ │ ├── api.ai_metrics.json │ │ ├── train_AddToPlaylist.json │ │ ├── train_AddToPlaylist_full.json │ │ └── validate_AddToPlaylist.json │ ├── BookRestaurant │ │ ├── Alexa_metrics.json │ │ ├── Luis_metrics.json │ │ ├── Snips_metrics.json │ │ ├── Snips_metrics_full.json │ │ ├── Wit_metrics.json │ │ ├── api.ai_metrics.json │ │ ├── train_BookRestaurant.json │ │ ├── train_BookRestaurant_full.json │ │ └── validate_BookRestaurant.json │ ├── GetWeather │ │ ├── Alexa_metrics.json │ │ ├── Luis_metrics.json │ │ ├── Snips_metrics.json │ │ ├── Snips_metrics_full.json │ │ ├── Wit_metrics.json │ │ ├── api.ai_metrics.json │ │ ├── train_GetWeather.json │ │ ├── train_GetWeather_full.json │ │ └── validate_GetWeather.json │ ├── RateBook │ │ ├── Alexa_metrics.json │ │ ├── Luis_metrics.json │ │ ├── Snips_metrics.json │ │ ├── Snips_metrics_full.json │ │ ├── Wit_metrics.json │ │ ├── api.ai_metrics.json │ │ ├── train_RateBook.json │ │ ├── train_RateBook_full.json │ │ └── validate_RateBook.json │ ├── SearchCreativeWork │ │ ├── Alexa_metrics.json │ │ ├── Luis_metrics.json │ │ ├── Snips_metrics.json │ │ ├── Snips_metrics_full.json │ │ ├── Wit_metrics.json │ │ ├── api.ai_metrics.json │ │ ├── train_SearchCreativeWork.json │ │ ├── train_SearchCreativeWork_full.json │ │ └── validate_SearchCreativeWork.json │ └── SearchScreeningEvent │ │ ├── Alexa_metrics.json │ │ ├── Luis_metrics.json │ │ ├── Snips_metrics.json │ │ ├── Snips_metrics_full.json │ │ ├── Wit_metrics.json │ │ ├── api.ai_metrics.json │ │ ├── train_SearchScreeningEvent.json │ │ ├── train_SearchScreeningEvent_full.json │ │ └── validate_SearchScreeningEvent.json ├── test_full.txt ├── test_label.npy ├── test_text.npy ├── train_full.txt ├── train_label.npy └── train_text.npy ├── glove.6B └── .gitignore └── prepare_data.py /README.md: -------------------------------------------------------------------------------- 1 | # Intent classification for a chatbot using Convolutional Neural Networks 2 | 3 | This is Keras implementation for the task of sentence classification using CNNs. 4 | 5 | Dataset for the above task was obtained from the project [Natural Language Understanding benchmark ](https://github.com/snipsco/nlu-benchmark/tree/master/2017-06-custom-intent-engines) 6 | 7 | Text used for the training falls under the six categories namely, AddToPlaylist, BookRestaurant, GetWeather , RateBook , SearchCreativeWork, SearchScreeningEvent each having nearly 2000 sentences. 8 | 9 | To prepare the dataset, from the main project's directory, open terminal and type: 10 | 11 | ```bash 12 | $ python prepare_data.py 13 | ``` 14 | 15 | Check [Intent_Classification_Keras_Glove.ipynb](https://github.com/ajinkyaT/CNN_Intent_Classification/blob/master/Intent_Classification_Keras_Glove.ipynb) for the model building and training part. Below is the model overview. 16 | 17 | ![image](https://github.com/brightmart/text_classification/raw/master/images/TextCNN.JPG "TextCNN") 18 | 19 | Although RNN's like LSTM and GRU are widely used for language modelling tasks but CNN's have also proven to be quite faster to train owing to data parallelization while training and give better results than the LSTM ones. [Here](https://github.com/brightmart/text_classification#performance) is a brief comparison between different methods to solve sentence classification, as can be seen TextCNN gives best result of all and also trains faster. I was able to achieve 99% accuracy on training and validation dataset within a minute after 3 epochs when trained on a regular i7 CPU. 20 | 21 | #### What lies ahead? 22 | 23 | Intent classification and named entity recognition are the two most important parts while making a goal oriented chatbot. 24 | 25 | There are many open source python packages for making a chatbot, Rasa is one of them. The cool thing about Rasa is that every part of the stack is fully customizable and easily interchangeable. Although Rasa has an excellent built in support for intent classification task but we can also specify our own model for the task, check [Processing Pipeline](https://nlu.rasa.com/pipeline.html) for more information on it. 26 | 27 | 28 | ## Resources 29 | 30 | [Using pre-trained word embeddings in a Keras model](https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html) 31 | 32 | [Convolutional Neural Networks for Sentence Classification 33 | ](https://arxiv.org/abs/1408.5882) 34 | 35 | [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification 36 | ](https://arxiv.org/abs/1510.03820) 37 | 38 | [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://arxiv.org/abs/1803.01271) 39 | 40 | 41 | -------------------------------------------------------------------------------- /data/.gitignore: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /data/label_encoder.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ajinkyaT/CNN_Intent_Classification/e95fe9898e0617959012cbe4488fd4e4d7ffc517/data/label_encoder.pkl -------------------------------------------------------------------------------- /data/processed_data/AddToPlaylist/AddToPlaylist_test.txt: -------------------------------------------------------------------------------- 1 | i 'd like to have this track onto my classical relaxations playlist __label__AddToPlaylist 2 | add the album to my flow espa ol playlist __label__AddToPlaylist 3 | add digging now to my young at heart playlist __label__AddToPlaylist 4 | add this song by too poetic to my piano ballads playlist __label__AddToPlaylist 5 | add this album to old school death metal __label__AddToPlaylist 6 | i need to add baro ferret to the urban hits under my name __label__AddToPlaylist 7 | add the album to the might and myth power metal playlist __label__AddToPlaylist 8 | to the travelling playlist , please add this david gahan song __label__AddToPlaylist 9 | please add some pete townshend to my playlist fiesta hits con lali __label__AddToPlaylist 10 | i 'd like for kasey chambers 's tune to be an addition to my chips and salsa playlist __label__AddToPlaylist 11 | add recalled to life to this is alejandro fern ndez __label__AddToPlaylist 12 | add nuba to my metal party playlist __label__AddToPlaylist 13 | add jo stafford music to the workout twerkout playlist __label__AddToPlaylist 14 | put jean philippe goncalves onto my running to rock 170 to 190 bpm __label__AddToPlaylist 15 | add the song virales de siempre by the cary brothers to my gym playlist __label__AddToPlaylist 16 | onto jerry 's classical moments in movies , please add the album __label__AddToPlaylist 17 | add beyond the valley of 1984 in playlist folk music at the gaslight caf __label__AddToPlaylist 18 | add jerry calliste , jr to my te quiero playlist __label__AddToPlaylist 19 | add porter wagoner to the the sleep machine waterscapes playlist __label__AddToPlaylist 20 | add the artist mike to the sexy as folk playlist __label__AddToPlaylist 21 | add brazilian flag anthem to top 100 alternative tracks on spotify __label__AddToPlaylist 22 | add andy hunter to my evening commute playlist __label__AddToPlaylist 23 | put petar georgiev kalica onto the old school hip hop playlist __label__AddToPlaylist 24 | can you add larry heard to my laundry playlist \? __label__AddToPlaylist 25 | put vandemataram srinivas 's track onto hiphop hot 50 __label__AddToPlaylist 26 | add millie corretjer to the rhythm playlist __label__AddToPlaylist 27 | add give us rest to my 70s smash hits playlist __label__AddToPlaylist 28 | add this track to my hands up playlist __label__AddToPlaylist 29 | i 'd like for you to add bobby brown to my enamor ndose playlist __label__AddToPlaylist 30 | add jonathan sprout album to my this is miranda lambert playlist __label__AddToPlaylist 31 | add ireland in the junior eurovision song contest 2015 to my jazzy dinner playlist __label__AddToPlaylist 32 | add the album to the the sweet suite playlist __label__AddToPlaylist 33 | add sarah slean to my playlist mellowed out gaming __label__AddToPlaylist 34 | add this album to the spanish beat playlist __label__AddToPlaylist 35 | add lofty fake anagram to the la mejor m sica de bso playlist __label__AddToPlaylist 36 | add the track to the work playlist __label__AddToPlaylist 37 | add a song to this is racionais mc 's __label__AddToPlaylist 38 | add track in my playlist called hands up __label__AddToPlaylist 39 | can you put this song from yutaka ozaki onto my this is miles davis playlist \? __label__AddToPlaylist 40 | add a track to playlist cena con amigos __label__AddToPlaylist 41 | add the famous flower of serving men to my evening acoustic playlist __label__AddToPlaylist 42 | add a song to indie hipster __label__AddToPlaylist 43 | add the 40 cal tune to the laundry playlist __label__AddToPlaylist 44 | add the album to my perfect concentration playlist __label__AddToPlaylist 45 | add the matt murphy tune to the flow espa ol playlist __label__AddToPlaylist 46 | add a very cellular song to masters of metal playlist __label__AddToPlaylist 47 | can i put this tune onto my sin estr s playlist \? __label__AddToPlaylist 48 | i 'd like to add jordan rudess onto the divertido para ni os playlist __label__AddToPlaylist 49 | add kent james to the disney soundtrack __label__AddToPlaylist 50 | add the artist adam deibert to my perfect concentration playlist __label__AddToPlaylist 51 | can you put the artist giovanni giacomo gastoldi onto the chill out music playlist \? __label__AddToPlaylist 52 | add the album to the hot 50 playlist __label__AddToPlaylist 53 | add the artist pete murray to my relaxing playlist __label__AddToPlaylist 54 | add the track to the drum breaks playlist __label__AddToPlaylist 55 | for my fantastic workout can you add sara bareilles \? __label__AddToPlaylist 56 | add the boy george track to the emo forever playlist __label__AddToPlaylist 57 | add ted heath to the road trip playlist __label__AddToPlaylist 58 | can you add last of the ghetto astronauts to the playlist called black sabbath the dio years \? __label__AddToPlaylist 59 | add this artist to showstopper being mary jane __label__AddToPlaylist 60 | put the artist onto top latin alternative __label__AddToPlaylist 61 | add michael wittig music to country icon playlist __label__AddToPlaylist 62 | add highway patrolman in my playlist this is al green __label__AddToPlaylist 63 | add richard mcnamara newest song to the just smile playlist __label__AddToPlaylist 64 | add annesley malewana album to playlist indietronic __label__AddToPlaylist 65 | add the artist to my dishwashing playlist __label__AddToPlaylist 66 | add this artist to fairy tales playlist __label__AddToPlaylist 67 | add muzika za decu to my crash course playlist __label__AddToPlaylist 68 | add a derek watkins tune to this is johnny cash __label__AddToPlaylist 69 | add our little corner of the world music from gilmore girls to my the funny thing about football is playlist __label__AddToPlaylist 70 | add the current track to my this is tchaikovsky playlist __label__AddToPlaylist 71 | put abe laboriel onto the escapada playlist __label__AddToPlaylist 72 | add abacab to beryl 's party on fridays playlist __label__AddToPlaylist 73 | please add this track by paul mcguigan to the deep house playlist __label__AddToPlaylist 74 | can you add the current tune to my calm before the storm playlist __label__AddToPlaylist 75 | please add the image of you to my playlist crate diggers anonymous __label__AddToPlaylist 76 | add a track to jazzy dinner __label__AddToPlaylist 77 | add the album to the hipster soul playlist __label__AddToPlaylist 78 | add this tune to my sleepify playlist __label__AddToPlaylist 79 | add jack white to my playlist this is shakira __label__AddToPlaylist 80 | add tommy johnson to the metalsucks playlist __label__AddToPlaylist 81 | add the chris clark tune to my women of the blues playlist __label__AddToPlaylist 82 | add an artist to jukebox boogie rhythm blues __label__AddToPlaylist 83 | add this artist to my electronic bliss playlist __label__AddToPlaylist 84 | i need to add to my infinite indie folk list the works of rahim shah __label__AddToPlaylist 85 | add martin barre to my punk unplugged playlist __label__AddToPlaylist 86 | add tierney sutton to my novedades viernes sudam rica playlist __label__AddToPlaylist 87 | add this tune to dorthy 's 80 's party playlist __label__AddToPlaylist 88 | a very cellular song needs to be added to my masters of metal playlist __label__AddToPlaylist 89 | add toyan to my epic gaming playlist __label__AddToPlaylist 90 | add the song to the mac 'n cheese playlist __label__AddToPlaylist 91 | add this artist to my spotlight on country 2016 playlist __label__AddToPlaylist 92 | add a song to my playlist madden nfl 16 __label__AddToPlaylist 93 | add emilie autumn to my na o reggae playlist __label__AddToPlaylist 94 | add farhad darya songs in virales de siempre __label__AddToPlaylist 95 | add a song in my all out 60s __label__AddToPlaylist 96 | add we have a theme song to my house afterwork playlist __label__AddToPlaylist 97 | add the song to my we everywhere playlist __label__AddToPlaylist 98 | add roel van velzen to my party of the century playlist __label__AddToPlaylist 99 | add the artist to the political punks playlist __label__AddToPlaylist 100 | add the album to my club hits playlist __label__AddToPlaylist 101 | -------------------------------------------------------------------------------- /data/processed_data/AddToPlaylist_test.txt: -------------------------------------------------------------------------------- 1 | i 'd like to have this track onto my classical relaxations playlist 2 | add the album to my flow espa ol playlist 3 | add digging now to my young at heart playlist 4 | add this song by too poetic to my piano ballads playlist 5 | add this album to old school death metal 6 | i need to add baro ferret to the urban hits under my name 7 | add the album to the might and myth power metal playlist 8 | to the travelling playlist , please add this david gahan song 9 | please add some pete townshend to my playlist fiesta hits con lali 10 | i 'd like for kasey chambers 's tune to be an addition to my chips and salsa playlist 11 | add recalled to life to this is alejandro fern ndez 12 | add nuba to my metal party playlist 13 | add jo stafford music to the workout twerkout playlist 14 | put jean philippe goncalves onto my running to rock 170 to 190 bpm 15 | add the song virales de siempre by the cary brothers to my gym playlist 16 | onto jerry 's classical moments in movies , please add the album 17 | add beyond the valley of 1984 in playlist folk music at the gaslight caf 18 | add jerry calliste , jr to my te quiero playlist 19 | add porter wagoner to the the sleep machine waterscapes playlist 20 | add the artist mike to the sexy as folk playlist 21 | add brazilian flag anthem to top 100 alternative tracks on spotify 22 | add andy hunter to my evening commute playlist 23 | put petar georgiev kalica onto the old school hip hop playlist 24 | can you add larry heard to my laundry playlist \? 25 | put vandemataram srinivas 's track onto hiphop hot 50 26 | add millie corretjer to the rhythm playlist 27 | add give us rest to my 70s smash hits playlist 28 | add this track to my hands up playlist 29 | i 'd like for you to add bobby brown to my enamor ndose playlist 30 | add jonathan sprout album to my this is miranda lambert playlist 31 | add ireland in the junior eurovision song contest 2015 to my jazzy dinner playlist 32 | add the album to the the sweet suite playlist 33 | add sarah slean to my playlist mellowed out gaming 34 | add this album to the spanish beat playlist 35 | add lofty fake anagram to the la mejor m sica de bso playlist 36 | add the track to the work playlist 37 | add a song to this is racionais mc 's 38 | add track in my playlist called hands up 39 | can you put this song from yutaka ozaki onto my this is miles davis playlist \? 40 | add a track to playlist cena con amigos 41 | add the famous flower of serving men to my evening acoustic playlist 42 | add a song to indie hipster 43 | add the 40 cal tune to the laundry playlist 44 | add the album to my perfect concentration playlist 45 | add the matt murphy tune to the flow espa ol playlist 46 | add a very cellular song to masters of metal playlist 47 | can i put this tune onto my sin estr s playlist \? 48 | i 'd like to add jordan rudess onto the divertido para ni os playlist 49 | add kent james to the disney soundtrack 50 | add the artist adam deibert to my perfect concentration playlist 51 | can you put the artist giovanni giacomo gastoldi onto the chill out music playlist \? 52 | add the album to the hot 50 playlist 53 | add the artist pete murray to my relaxing playlist 54 | add the track to the drum breaks playlist 55 | for my fantastic workout can you add sara bareilles \? 56 | add the boy george track to the emo forever playlist 57 | add ted heath to the road trip playlist 58 | can you add last of the ghetto astronauts to the playlist called black sabbath the dio years \? 59 | add this artist to showstopper being mary jane 60 | put the artist onto top latin alternative 61 | add michael wittig music to country icon playlist 62 | add highway patrolman in my playlist this is al green 63 | add richard mcnamara newest song to the just smile playlist 64 | add annesley malewana album to playlist indietronic 65 | add the artist to my dishwashing playlist 66 | add this artist to fairy tales playlist 67 | add muzika za decu to my crash course playlist 68 | add a derek watkins tune to this is johnny cash 69 | add our little corner of the world music from gilmore girls to my the funny thing about football is playlist 70 | add the current track to my this is tchaikovsky playlist 71 | put abe laboriel onto the escapada playlist 72 | add abacab to beryl 's party on fridays playlist 73 | please add this track by paul mcguigan to the deep house playlist 74 | can you add the current tune to my calm before the storm playlist 75 | please add the image of you to my playlist crate diggers anonymous 76 | add a track to jazzy dinner 77 | add the album to the hipster soul playlist 78 | add this tune to my sleepify playlist 79 | add jack white to my playlist this is shakira 80 | add tommy johnson to the metalsucks playlist 81 | add the chris clark tune to my women of the blues playlist 82 | add an artist to jukebox boogie rhythm blues 83 | add this artist to my electronic bliss playlist 84 | i need to add to my infinite indie folk list the works of rahim shah 85 | add martin barre to my punk unplugged playlist 86 | add tierney sutton to my novedades viernes sudam rica playlist 87 | add this tune to dorthy 's 80 's party playlist 88 | a very cellular song needs to be added to my masters of metal playlist 89 | add toyan to my epic gaming playlist 90 | add the song to the mac 'n cheese playlist 91 | add this artist to my spotlight on country 2016 playlist 92 | add a song to my playlist madden nfl 16 93 | add emilie autumn to my na o reggae playlist 94 | add farhad darya songs in virales de siempre 95 | add a song in my all out 60s 96 | add we have a theme song to my house afterwork playlist 97 | add the song to my we everywhere playlist 98 | add roel van velzen to my party of the century playlist 99 | add the artist to the political punks playlist 100 | add the album to my club hits playlist 101 | -------------------------------------------------------------------------------- /data/processed_data/BookRestaurant/BookRestaurant_test.txt: -------------------------------------------------------------------------------- 1 | book a reservation for my babies and i __label__BookRestaurant 2 | book a reservation for a restaurant not far from ma __label__BookRestaurant 3 | i would like to book a restaurant in tanzania that is within walking distance for my mom and i __label__BookRestaurant 4 | book a reservation for an oyster bar __label__BookRestaurant 5 | book a reservation for 6 people for a creole tavern in montenegro __label__BookRestaurant 6 | i need a table in sacaton at a gluten free restaurant __label__BookRestaurant 7 | book sot for me and my grandfather nearby west reading __label__BookRestaurant 8 | book me and my nieces a reservation for a seafood restaurant in cle elum , ne on ascension day __label__BookRestaurant 9 | book spot for two at city tavern __label__BookRestaurant 10 | i want to book a brasserie for 3 people in netherlands antilles __label__BookRestaurant 11 | book me a reservation for the best bistro __label__BookRestaurant 12 | book the best table in tanzania for 5 people at a diner __label__BookRestaurant 13 | i want to book a joint in a spa __label__BookRestaurant 14 | book a gastropub that serves turkish food for 4 people __label__BookRestaurant 15 | book spot for 7 at an indoor restaurant in mp now __label__BookRestaurant 16 | book a table in fiji for zero a m __label__BookRestaurant 17 | i want to book a restaurant for five people in sri lanka __label__BookRestaurant 18 | i need a table for 5 at a highly rated gastropub in concord mn __label__BookRestaurant 19 | i want to book oregon electric station in north city __label__BookRestaurant 20 | i need a table for 4 please confirm the reservation __label__BookRestaurant 21 | book a popular restaurant for 5 people __label__BookRestaurant 22 | i want to book a joint close by the naomi 's hostel for a meal for 8 people __label__BookRestaurant 23 | i want to eat a delicatessen in thirteen hours that serves eastern european food __label__BookRestaurant 24 | book a reservation for nine people at a bakery in nunez __label__BookRestaurant 25 | book a reservation at tavern for noodle __label__BookRestaurant 26 | book spot for 4 in somalia __label__BookRestaurant 27 | i want to book albany pump station in buckholts washington now for a party of 9 __label__BookRestaurant 28 | i want to book a taverna in archer city for this spring for nine people __label__BookRestaurant 29 | i want to book a top rated brasserie for 7 people __label__BookRestaurant 30 | book a reservation for 8 people in wardville , kansas __label__BookRestaurant 31 | table for breadline cafe in minnesota next friday __label__BookRestaurant 32 | i want to book a restaurant in niger for seven people __label__BookRestaurant 33 | book spot for 9 __label__BookRestaurant 34 | book me a reservation for a pub in cormorant for a party of nine __label__BookRestaurant 35 | book spot for my nieces and i at a tea house __label__BookRestaurant 36 | i want to book a jewish restaurant in gambia __label__BookRestaurant 37 | book a reservation for the dome , edinburgh close to brooklawn __label__BookRestaurant 38 | book spot for 1 at town of ramsgate in merit __label__BookRestaurant 39 | book a spot for me and kathrine at smithville __label__BookRestaurant 40 | i want to book a restaurant for my father in law and i in buckner a year from now __label__BookRestaurant 41 | book a restaurant reservation in 6 weeks __label__BookRestaurant 42 | book a reservation for a bar with a spa nearby id __label__BookRestaurant 43 | book spot for four at cliff house , san francisco in martinique __label__BookRestaurant 44 | i need a table for 4 in saint helena at settha palace hotel __label__BookRestaurant 45 | i want to book a restaurant in frenier 12 years from now for 4 people __label__BookRestaurant 46 | book seven in neighboring moorpark __label__BookRestaurant 47 | i want to eat by five pm in ne for a six people __label__BookRestaurant 48 | i want to book tupelo honey cafe in new jersey for five people __label__BookRestaurant 49 | book a reservation for two at mickies dairy bar in weedsport __label__BookRestaurant 50 | book a table at a fried chicken restaurant __label__BookRestaurant 51 | book spot for mavis , sheila and i in syria at elevenses __label__BookRestaurant 52 | can you book me a table at windows on the world in cokeville , mi \? __label__BookRestaurant 53 | book me a table for 5 this year at cherwell boathouse __label__BookRestaurant 54 | book spot for six at 8 pm at a coffeehouse in ne that serves hog fry __label__BookRestaurant 55 | i want to book a restaurant close by in inman for five people __label__BookRestaurant 56 | i need a table at eddie 's attic in nevada for one __label__BookRestaurant 57 | book a reservation for an osteria restaurant for 4 people on november 4 __label__BookRestaurant 58 | i want to book a top rated restaurant close by in la for me , rebecca and loraine on 2 6 2020 __label__BookRestaurant 59 | book a reservation for 1 at a diner in wi __label__BookRestaurant 60 | book a reservation for 5 people at the top rated brasserie restaurant __label__BookRestaurant 61 | book a table on 1 20 2023 for 5 people in mh __label__BookRestaurant 62 | book a table near pat 's college __label__BookRestaurant 63 | i want to book a steakhouse in vimy ridge __label__BookRestaurant 64 | i want a table at james d conrey house in urbank california __label__BookRestaurant 65 | like to book a seat in monaco for the yankee doodle coffee shop __label__BookRestaurant 66 | i want to book a table in a restaurant in bouvet island __label__BookRestaurant 67 | i would like to book a restaurant for souvlaki cuisine in the state of ne __label__BookRestaurant 68 | book a reservation for 10 people at an oyster bar with a pool within the same area of cowansburg for 10 pm __label__BookRestaurant 69 | book a reservation for velma , ana and rebecca for an american pizzeria at 5 am in ma __label__BookRestaurant 70 | book a spot for 4 in oklahoma at south street diner __label__BookRestaurant 71 | book a reservation for my mommy and i at a restaurant in central african republic __label__BookRestaurant 72 | book a reservation for five people for a tatar taverna in sargents __label__BookRestaurant 73 | phyllis ward and veronica need a table at a restaurant in 152 days __label__BookRestaurant 74 | book a reservation for ten at a restaurant in ohio __label__BookRestaurant 75 | i want to book a tea house that serves salade far from here at midnight in panama for two people __label__BookRestaurant 76 | i want to book a food truck for seven people in the republic of the congo __label__BookRestaurant 77 | i want to book a restaurant for ten people __label__BookRestaurant 78 | lets eat near oakfield 17 seconds from now at ted peters famous smoked fish __label__BookRestaurant 79 | book sot for 7 at a restaurant that serves european in stringtown on feb the 28th , 2034 __label__BookRestaurant 80 | book a restaurant for six at an outdoor cafe in land __label__BookRestaurant 81 | book a table for 12 am at our step mother 's secondary residence within walking distance for one __label__BookRestaurant 82 | please book me a table at a pizzeria with a parking facility in ghana __label__BookRestaurant 83 | book spot for four at a indoor pub within the same area of louisiana in one minute __label__BookRestaurant 84 | please book me a restaurant __label__BookRestaurant 85 | book a reservation for me and my step brother at amt coffee in lakemoor __label__BookRestaurant 86 | i want to book a churrascaria in romeoville at ten a m for four people __label__BookRestaurant 87 | table for 5 a m at baker 's keyboard lounge __label__BookRestaurant 88 | please book me a table at a bistro which serves lorna doone __label__BookRestaurant 89 | i want to book a restaurant for six people in wagstaff ak __label__BookRestaurant 90 | i would like to book a highly rated restaurant for a party of ten __label__BookRestaurant 91 | i want to book a sundanese gastropub nearby in texas for 3 people on 5 20 2025 __label__BookRestaurant 92 | book a party of five at seagoville for 06 42 __label__BookRestaurant 93 | book spot for 9 at thurmont __label__BookRestaurant 94 | i want to book a restaurant in sixteen seconds for 5 people in gold point montana __label__BookRestaurant 95 | i want to eat in ramona __label__BookRestaurant 96 | book a party at their campus within the same area for churrascaria __label__BookRestaurant 97 | book me a reservation for a party of 3 at a pub in northern mariana islands __label__BookRestaurant 98 | i want to book a bougatsa restaurant in next year nearby penn for three people __label__BookRestaurant 99 | book a reservation for nine people at the best pub nearby tangier in six months __label__BookRestaurant 100 | need a table somewhere in quarryville 14 hours from now __label__BookRestaurant 101 | -------------------------------------------------------------------------------- /data/processed_data/BookRestaurant_test.txt: -------------------------------------------------------------------------------- 1 | book a reservation for my babies and i 2 | book a reservation for a restaurant not far from ma 3 | i would like to book a restaurant in tanzania that is within walking distance for my mom and i 4 | book a reservation for an oyster bar 5 | book a reservation for 6 people for a creole tavern in montenegro 6 | i need a table in sacaton at a gluten free restaurant 7 | book sot for me and my grandfather nearby west reading 8 | book me and my nieces a reservation for a seafood restaurant in cle elum , ne on ascension day 9 | book spot for two at city tavern 10 | i want to book a brasserie for 3 people in netherlands antilles 11 | book me a reservation for the best bistro 12 | book the best table in tanzania for 5 people at a diner 13 | i want to book a joint in a spa 14 | book a gastropub that serves turkish food for 4 people 15 | book spot for 7 at an indoor restaurant in mp now 16 | book a table in fiji for zero a m 17 | i want to book a restaurant for five people in sri lanka 18 | i need a table for 5 at a highly rated gastropub in concord mn 19 | i want to book oregon electric station in north city 20 | i need a table for 4 please confirm the reservation 21 | book a popular restaurant for 5 people 22 | i want to book a joint close by the naomi 's hostel for a meal for 8 people 23 | i want to eat a delicatessen in thirteen hours that serves eastern european food 24 | book a reservation for nine people at a bakery in nunez 25 | book a reservation at tavern for noodle 26 | book spot for 4 in somalia 27 | i want to book albany pump station in buckholts washington now for a party of 9 28 | i want to book a taverna in archer city for this spring for nine people 29 | i want to book a top rated brasserie for 7 people 30 | book a reservation for 8 people in wardville , kansas 31 | table for breadline cafe in minnesota next friday 32 | i want to book a restaurant in niger for seven people 33 | book spot for 9 34 | book me a reservation for a pub in cormorant for a party of nine 35 | book spot for my nieces and i at a tea house 36 | i want to book a jewish restaurant in gambia 37 | book a reservation for the dome , edinburgh close to brooklawn 38 | book spot for 1 at town of ramsgate in merit 39 | book a spot for me and kathrine at smithville 40 | i want to book a restaurant for my father in law and i in buckner a year from now 41 | book a restaurant reservation in 6 weeks 42 | book a reservation for a bar with a spa nearby id 43 | book spot for four at cliff house , san francisco in martinique 44 | i need a table for 4 in saint helena at settha palace hotel 45 | i want to book a restaurant in frenier 12 years from now for 4 people 46 | book seven in neighboring moorpark 47 | i want to eat by five pm in ne for a six people 48 | i want to book tupelo honey cafe in new jersey for five people 49 | book a reservation for two at mickies dairy bar in weedsport 50 | book a table at a fried chicken restaurant 51 | book spot for mavis , sheila and i in syria at elevenses 52 | can you book me a table at windows on the world in cokeville , mi \? 53 | book me a table for 5 this year at cherwell boathouse 54 | book spot for six at 8 pm at a coffeehouse in ne that serves hog fry 55 | i want to book a restaurant close by in inman for five people 56 | i need a table at eddie 's attic in nevada for one 57 | book a reservation for an osteria restaurant for 4 people on november 4 58 | i want to book a top rated restaurant close by in la for me , rebecca and loraine on 2 6 2020 59 | book a reservation for 1 at a diner in wi 60 | book a reservation for 5 people at the top rated brasserie restaurant 61 | book a table on 1 20 2023 for 5 people in mh 62 | book a table near pat 's college 63 | i want to book a steakhouse in vimy ridge 64 | i want a table at james d conrey house in urbank california 65 | like to book a seat in monaco for the yankee doodle coffee shop 66 | i want to book a table in a restaurant in bouvet island 67 | i would like to book a restaurant for souvlaki cuisine in the state of ne 68 | book a reservation for 10 people at an oyster bar with a pool within the same area of cowansburg for 10 pm 69 | book a reservation for velma , ana and rebecca for an american pizzeria at 5 am in ma 70 | book a spot for 4 in oklahoma at south street diner 71 | book a reservation for my mommy and i at a restaurant in central african republic 72 | book a reservation for five people for a tatar taverna in sargents 73 | phyllis ward and veronica need a table at a restaurant in 152 days 74 | book a reservation for ten at a restaurant in ohio 75 | i want to book a tea house that serves salade far from here at midnight in panama for two people 76 | i want to book a food truck for seven people in the republic of the congo 77 | i want to book a restaurant for ten people 78 | lets eat near oakfield 17 seconds from now at ted peters famous smoked fish 79 | book sot for 7 at a restaurant that serves european in stringtown on feb the 28th , 2034 80 | book a restaurant for six at an outdoor cafe in land 81 | book a table for 12 am at our step mother 's secondary residence within walking distance for one 82 | please book me a table at a pizzeria with a parking facility in ghana 83 | book spot for four at a indoor pub within the same area of louisiana in one minute 84 | please book me a restaurant 85 | book a reservation for me and my step brother at amt coffee in lakemoor 86 | i want to book a churrascaria in romeoville at ten a m for four people 87 | table for 5 a m at baker 's keyboard lounge 88 | please book me a table at a bistro which serves lorna doone 89 | i want to book a restaurant for six people in wagstaff ak 90 | i would like to book a highly rated restaurant for a party of ten 91 | i want to book a sundanese gastropub nearby in texas for 3 people on 5 20 2025 92 | book a party of five at seagoville for 06 42 93 | book spot for 9 at thurmont 94 | i want to book a restaurant in sixteen seconds for 5 people in gold point montana 95 | i want to eat in ramona 96 | book a party at their campus within the same area for churrascaria 97 | book me a reservation for a party of 3 at a pub in northern mariana islands 98 | i want to book a bougatsa restaurant in next year nearby penn for three people 99 | book a reservation for nine people at the best pub nearby tangier in six months 100 | need a table somewhere in quarryville 14 hours from now 101 | -------------------------------------------------------------------------------- /data/processed_data/GetWeather/GetWeather_test.txt: -------------------------------------------------------------------------------- 1 | what will the weather be faraway from here \? __label__GetWeather 2 | will there be fog in tahquamenon falls state park \? __label__GetWeather 3 | tell me the weather forecast for gibsland __label__GetWeather 4 | is there a storm now in nc \? __label__GetWeather 5 | what will the weather be in monument of lihula on december the 5th \? __label__GetWeather 6 | weather next year in dominica __label__GetWeather 7 | when will it be hot here __label__GetWeather 8 | what will the weather be in 1 day in kuwait \? __label__GetWeather 9 | what kind of weather will be in ukraine one minute from now \? __label__GetWeather 10 | humidity in olvey new hampshire __label__GetWeather 11 | what 's the weather going to be in ut \? __label__GetWeather 12 | humidity not far from colorado city on november the 7th , 2024 __label__GetWeather 13 | what is the forecast for wyoming at stanardsville during the storm __label__GetWeather 14 | what will the weather be in north carolina \? __label__GetWeather 15 | what is the forecast starting 11 weeks from now nearby the state of wisconsin __label__GetWeather 16 | will it be rainy at sunrise in ramey saudi arabia \? __label__GetWeather 17 | check the forecast for nebraska __label__GetWeather 18 | will it be warmer in north korea at nineteen o'clock __label__GetWeather 19 | let me know the weather forecast around ten pm faraway from here in park narodowy brimstone hill fortress __label__GetWeather 20 | will it be stormy in the ouachita national forest \? __label__GetWeather 21 | tell me if it will be snowy 8 hours from now in mount airy , vi __label__GetWeather 22 | what will the weather be nineteen hours from now neighboring saint kitts and nevis \? __label__GetWeather 23 | will there be hail on 11 12 2036 in singapore __label__GetWeather 24 | will it be colder here in 48 and a half weeks __label__GetWeather 25 | what 's the weather going to be in knobel \? __label__GetWeather 26 | what will the weather be in dane on sep the fifth , 2030 \? __label__GetWeather 27 | what will the weather be in ohio \? __label__GetWeather 28 | i need to know the weather for jan the 3rd in mexico when i go to port vue __label__GetWeather 29 | what is the forecast for tone prefectural natural park in 1 hour and within the same area __label__GetWeather 30 | what kind of weather is forecast around one pmnear vatican \? __label__GetWeather 31 | will it be chilly in weldona \? __label__GetWeather 32 | will it be colder in virgin islands national park \? __label__GetWeather 33 | will it be hot at 13 19 in de funiak springs serbia and montenegro \? __label__GetWeather 34 | what is the weather going to be like in virginia on st patrick 's day \? __label__GetWeather 35 | weather in kaneville maryland __label__GetWeather 36 | when is sunrise for ar __label__GetWeather 37 | what will the weather be not far from here on october the nineteenth , 2026 \? __label__GetWeather 38 | what is the forecast for waurika in samoa __label__GetWeather 39 | tell me the weather forecast here __label__GetWeather 40 | what is the weather forecast nearby nicodemus __label__GetWeather 41 | what will the weather be in nov in brookneal \? __label__GetWeather 42 | will it be colder four months from now in suwanee ak __label__GetWeather 43 | what is the weather forecast for burundi __label__GetWeather 44 | what 's the weather in benton city \? __label__GetWeather 45 | what will the weather be in ky on oct 16 , 2036 \? __label__GetWeather 46 | will the sun be out in 1 minute in searcy , uganda __label__GetWeather 47 | what is the weather here __label__GetWeather 48 | what will the weather be one second from now in chad \? __label__GetWeather 49 | what kind of weather is forecast in ms now \? __label__GetWeather 50 | what is the forecast for la for freezing __label__GetWeather 51 | how cold will it be here in 1 second __label__GetWeather 52 | what is the forecast for hotter weather at southford falls state park __label__GetWeather 53 | what is the overcast forecast for the current position starting on jul 19 , 2030 __label__GetWeather 54 | what is the forecast for morocco at lake ozark on december seventeenth , 2022 __label__GetWeather 55 | what will the humidity be in the current spot at 15 19 29 \? __label__GetWeather 56 | what is the forecast in nicodemus and nearby __label__GetWeather 57 | what is the weather going to be like in benton colorado in 2 and a half months __label__GetWeather 58 | what 's the weather forecast for bothe napa valley state park close by february 20 \? __label__GetWeather 59 | what is the forecast for beginning on nov 17 for franklinville __label__GetWeather 60 | what 's the forecast for sep 26 in emerado saint pierre and miquelon __label__GetWeather 61 | will there be a blizzard next winter in visalia , idaho __label__GetWeather 62 | will it be warmer in the district of columbia on may 25 , 2033 \? __label__GetWeather 63 | what will the weather be here on dec 7th \? __label__GetWeather 64 | what is the forecast for colder temps beginning on law day here __label__GetWeather 65 | what 's the weather like in tyonek , new jersey __label__GetWeather 66 | what is the forecast for here for blizzard conditions at five pm __label__GetWeather 67 | will there be a storm in gibsonia at 8 p m \? __label__GetWeather 68 | what is the cold condition of our current position for tomorrow __label__GetWeather 69 | what will the weather be in hialeah gardens on october the 24th \? __label__GetWeather 70 | will it be freezing today in delaware and lehigh national heritage corridor \? __label__GetWeather 71 | what is the forecast in admire in tx starting at seventeen __label__GetWeather 72 | what is the forecast in north carolina for edgemoor __label__GetWeather 73 | what is the forecast for costa rica __label__GetWeather 74 | need weather for parc national tolhuaca to see if it will be fog today __label__GetWeather 75 | weather in walden russia on 12 26 2018 __label__GetWeather 76 | what 's the humidity here right now \? __label__GetWeather 77 | how 's the weather at petit manan national wildlife refuge and nearby right now __label__GetWeather 78 | what is the forecast for lansford for temperate weather __label__GetWeather 79 | overcast on state holiday in pawling nature reserve and neighboring places __label__GetWeather 80 | i need the weather in wakarusa __label__GetWeather 81 | tell me the forecast for 6 am in tatra nationalpark __label__GetWeather 82 | tell me the weather forecast for ut on thursday __label__GetWeather 83 | what is the forecast for turtle islands national park __label__GetWeather 84 | will it be hotter in pr at 23 o'clock \? __label__GetWeather 85 | weather in two hours in uzbekistan __label__GetWeather 86 | what is the forecast for this afternoon for blizzard conditions in dieterich chad __label__GetWeather 87 | how 's the weather here at two am __label__GetWeather 88 | will custer national forest be chillier at seven pm __label__GetWeather 89 | what is the forecast for starting at three a m in two buttes for warm weather __label__GetWeather 90 | what 's the weather in fox chapel \? __label__GetWeather 91 | what is the rain forecast for one hour from now in south korea __label__GetWeather 92 | tell me the weather forecast here __label__GetWeather 93 | will there be a cloud in vi in 14 minutes \? __label__GetWeather 94 | how much colder will it be not far from utah around 3 am \? __label__GetWeather 95 | will it be chilly midday in cresbard afghanistan \? __label__GetWeather 96 | what will the weather be in sarygamy sanctuary on august 21 , 2035 \? __label__GetWeather 97 | will it be rainy in tenino \? __label__GetWeather 98 | will it be hot in the netherlands on february 16th \? __label__GetWeather 99 | where is belgium located __label__GetWeather 100 | what will the weather be in milleville beach \? __label__GetWeather 101 | -------------------------------------------------------------------------------- /data/processed_data/GetWeather_test.txt: -------------------------------------------------------------------------------- 1 | what will the weather be faraway from here \? 2 | will there be fog in tahquamenon falls state park \? 3 | tell me the weather forecast for gibsland 4 | is there a storm now in nc \? 5 | what will the weather be in monument of lihula on december the 5th \? 6 | weather next year in dominica 7 | when will it be hot here 8 | what will the weather be in 1 day in kuwait \? 9 | what kind of weather will be in ukraine one minute from now \? 10 | humidity in olvey new hampshire 11 | what 's the weather going to be in ut \? 12 | humidity not far from colorado city on november the 7th , 2024 13 | what is the forecast for wyoming at stanardsville during the storm 14 | what will the weather be in north carolina \? 15 | what is the forecast starting 11 weeks from now nearby the state of wisconsin 16 | will it be rainy at sunrise in ramey saudi arabia \? 17 | check the forecast for nebraska 18 | will it be warmer in north korea at nineteen o'clock 19 | let me know the weather forecast around ten pm faraway from here in park narodowy brimstone hill fortress 20 | will it be stormy in the ouachita national forest \? 21 | tell me if it will be snowy 8 hours from now in mount airy , vi 22 | what will the weather be nineteen hours from now neighboring saint kitts and nevis \? 23 | will there be hail on 11 12 2036 in singapore 24 | will it be colder here in 48 and a half weeks 25 | what 's the weather going to be in knobel \? 26 | what will the weather be in dane on sep the fifth , 2030 \? 27 | what will the weather be in ohio \? 28 | i need to know the weather for jan the 3rd in mexico when i go to port vue 29 | what is the forecast for tone prefectural natural park in 1 hour and within the same area 30 | what kind of weather is forecast around one pmnear vatican \? 31 | will it be chilly in weldona \? 32 | will it be colder in virgin islands national park \? 33 | will it be hot at 13 19 in de funiak springs serbia and montenegro \? 34 | what is the weather going to be like in virginia on st patrick 's day \? 35 | weather in kaneville maryland 36 | when is sunrise for ar 37 | what will the weather be not far from here on october the nineteenth , 2026 \? 38 | what is the forecast for waurika in samoa 39 | tell me the weather forecast here 40 | what is the weather forecast nearby nicodemus 41 | what will the weather be in nov in brookneal \? 42 | will it be colder four months from now in suwanee ak 43 | what is the weather forecast for burundi 44 | what 's the weather in benton city \? 45 | what will the weather be in ky on oct 16 , 2036 \? 46 | will the sun be out in 1 minute in searcy , uganda 47 | what is the weather here 48 | what will the weather be one second from now in chad \? 49 | what kind of weather is forecast in ms now \? 50 | what is the forecast for la for freezing 51 | how cold will it be here in 1 second 52 | what is the forecast for hotter weather at southford falls state park 53 | what is the overcast forecast for the current position starting on jul 19 , 2030 54 | what is the forecast for morocco at lake ozark on december seventeenth , 2022 55 | what will the humidity be in the current spot at 15 19 29 \? 56 | what is the forecast in nicodemus and nearby 57 | what is the weather going to be like in benton colorado in 2 and a half months 58 | what 's the weather forecast for bothe napa valley state park close by february 20 \? 59 | what is the forecast for beginning on nov 17 for franklinville 60 | what 's the forecast for sep 26 in emerado saint pierre and miquelon 61 | will there be a blizzard next winter in visalia , idaho 62 | will it be warmer in the district of columbia on may 25 , 2033 \? 63 | what will the weather be here on dec 7th \? 64 | what is the forecast for colder temps beginning on law day here 65 | what 's the weather like in tyonek , new jersey 66 | what is the forecast for here for blizzard conditions at five pm 67 | will there be a storm in gibsonia at 8 p m \? 68 | what is the cold condition of our current position for tomorrow 69 | what will the weather be in hialeah gardens on october the 24th \? 70 | will it be freezing today in delaware and lehigh national heritage corridor \? 71 | what is the forecast in admire in tx starting at seventeen 72 | what is the forecast in north carolina for edgemoor 73 | what is the forecast for costa rica 74 | need weather for parc national tolhuaca to see if it will be fog today 75 | weather in walden russia on 12 26 2018 76 | what 's the humidity here right now \? 77 | how 's the weather at petit manan national wildlife refuge and nearby right now 78 | what is the forecast for lansford for temperate weather 79 | overcast on state holiday in pawling nature reserve and neighboring places 80 | i need the weather in wakarusa 81 | tell me the forecast for 6 am in tatra nationalpark 82 | tell me the weather forecast for ut on thursday 83 | what is the forecast for turtle islands national park 84 | will it be hotter in pr at 23 o'clock \? 85 | weather in two hours in uzbekistan 86 | what is the forecast for this afternoon for blizzard conditions in dieterich chad 87 | how 's the weather here at two am 88 | will custer national forest be chillier at seven pm 89 | what is the forecast for starting at three a m in two buttes for warm weather 90 | what 's the weather in fox chapel \? 91 | what is the rain forecast for one hour from now in south korea 92 | tell me the weather forecast here 93 | will there be a cloud in vi in 14 minutes \? 94 | how much colder will it be not far from utah around 3 am \? 95 | will it be chilly midday in cresbard afghanistan \? 96 | what will the weather be in sarygamy sanctuary on august 21 , 2035 \? 97 | will it be rainy in tenino \? 98 | will it be hot in the netherlands on february 16th \? 99 | where is belgium located 100 | what will the weather be in milleville beach \? 101 | -------------------------------------------------------------------------------- /data/processed_data/RateBook/RateBook_test.txt: -------------------------------------------------------------------------------- 1 | rate this album four out of 6 stars __label__RateBook 2 | give this textbook four stars __label__RateBook 3 | rate a twist in the tale zero out of 6 points __label__RateBook 4 | rate the children of niobe 1 out of 6 points __label__RateBook 5 | give zero stars to halo ghosts of onyx __label__RateBook 6 | give this novel a score of 5 __label__RateBook 7 | give the current series four of 6 points __label__RateBook 8 | give 4 out of 6 points to the spirit ring chronicle __label__RateBook 9 | give two stars out of 6 to 36 children __label__RateBook 10 | rate the sneetches and other stories a three __label__RateBook 11 | rate the current series four stars __label__RateBook 12 | rate this book a 4 out of 6 __label__RateBook 13 | rate the current novel 5 of 6 stars __label__RateBook 14 | rate this book a 1 __label__RateBook 15 | give zero out of 6 to the current album __label__RateBook 16 | give this album 5 points __label__RateBook 17 | rate the mystery of the tolling bell series 4 stars __label__RateBook 18 | give the current novel two stars __label__RateBook 19 | give the current book 4 stars __label__RateBook 20 | give joe magarac and his usa citizen papers 5 points __label__RateBook 21 | rate the guilty 0 of 6 points __label__RateBook 22 | rate this textbook four out of 6 __label__RateBook 23 | give the catedral series four stars __label__RateBook 24 | reminiscences of the anti japanese guerillas chronicle deserves zero points out of 6 for a rating __label__RateBook 25 | give small screen , big picture a 0 out of 6 rating __label__RateBook 26 | gods and pawns should get a three __label__RateBook 27 | give zero stars to this textbook __label__RateBook 28 | rate the current novel a 4 out of 6 stars __label__RateBook 29 | rate the book the atmospheric railway 5 out of 6 __label__RateBook 30 | rate black boy 4 out of 6 __label__RateBook 31 | rate the chronicle current 1 star __label__RateBook 32 | mark this album a score of 5 __label__RateBook 33 | rate the current novel zero out of 6 __label__RateBook 34 | rate the current novel a 2 __label__RateBook 35 | give the giant devil dingo 4 points __label__RateBook 36 | rate this current novel two out of 6 __label__RateBook 37 | give monthly index of medical specialities a two out of 6 rating __label__RateBook 38 | rate this novel 2 out of 6 points __label__RateBook 39 | rate the current novel 3 stars __label__RateBook 40 | rate the current essay zero out of 6 stars __label__RateBook 41 | rate this current album 0 stars __label__RateBook 42 | give a brief stop on the road from auschwitz 1 out of 6 stars __label__RateBook 43 | rate this album 4 out of 6 stars __label__RateBook 44 | rate hate that cat 1 out of 6 stars __label__RateBook 45 | give my current book one of 6 stars __label__RateBook 46 | rate current novel one stars __label__RateBook 47 | give five out of 6 points to this album __label__RateBook 48 | give a rating of 2 to juneteenth __label__RateBook 49 | rate ruth five out of 6 points __label__RateBook 50 | rate the sea of trolls 1 stars out of 6 __label__RateBook 51 | give the zenith angle one out of 6 points __label__RateBook 52 | give zero stars to rhialto the marvellous __label__RateBook 53 | give the current book a zero of 6 __label__RateBook 54 | rate personal demons 0 out of 6 points __label__RateBook 55 | rate the current series a 4 __label__RateBook 56 | give one of 6 points to who will cry when you die __label__RateBook 57 | give zero out of 6 stars to this album __label__RateBook 58 | give this novel 2 stars __label__RateBook 59 | rate the 8 week cholesterol cure three out of 6 __label__RateBook 60 | rate this novel 3 out of 6 points __label__RateBook 61 | rate the lives of john lennon five points __label__RateBook 62 | give the american scene 2 of 6 stars __label__RateBook 63 | rate this textbook a one __label__RateBook 64 | give summer of the swans 1 points __label__RateBook 65 | give the current textbook a rating of five __label__RateBook 66 | give 4 points to the person and the common good __label__RateBook 67 | give a four rating to a world apart __label__RateBook 68 | rate this chronicle 0 points __label__RateBook 69 | give wilco learning how to die a rating of four points __label__RateBook 70 | rate this saga two out of 6 __label__RateBook 71 | rate the gift imagination and the erotic life of property five stars __label__RateBook 72 | rate neverwhere four out of 6 __label__RateBook 73 | rate in the company of cheerful ladies a zero out of 6 __label__RateBook 74 | give one start to the current book __label__RateBook 75 | give this chronicle a 2 rating __label__RateBook 76 | rate this essay a 1 __label__RateBook 77 | out of 6 , give rivers of babylon a 1 __label__RateBook 78 | give 5 of 6 stars to expressive processing __label__RateBook 79 | rate the ghost house series a one __label__RateBook 80 | rate know ye not agincourt \? 2 out of 6 stars __label__RateBook 81 | i would rate theft a love story four out of 6 stars __label__RateBook 82 | rate the further adventures of the joker four stars __label__RateBook 83 | give 0 rating to in the heart of the country __label__RateBook 84 | give 1 out of 6 rating to the current textbook __label__RateBook 85 | give the current chronicle five of 6 points __label__RateBook 86 | rate cotton comes to harlem a 2 __label__RateBook 87 | give this album one stars __label__RateBook 88 | rate the adventures of augie march one points __label__RateBook 89 | rate soul music a 0 __label__RateBook 90 | give hindu temples what happened to them a 5 out of 6 stars __label__RateBook 91 | give this novel a 1 __label__RateBook 92 | rate the current textbook 1 out of 6 __label__RateBook 93 | give this textbook 0 out of 6 stars __label__RateBook 94 | give the crystal snare 5 stars __label__RateBook 95 | rate this saga two out of 6 __label__RateBook 96 | give wilco learning how to die a rating of four points __label__RateBook 97 | rate this book 3 stars out of 6 __label__RateBook 98 | rate the three junes one out of 6 __label__RateBook 99 | give four stars to the broken window __label__RateBook 100 | rate the current series 4 points __label__RateBook 101 | -------------------------------------------------------------------------------- /data/processed_data/RateBook_test.txt: -------------------------------------------------------------------------------- 1 | rate this album four out of 6 stars 2 | give this textbook four stars 3 | rate a twist in the tale zero out of 6 points 4 | rate the children of niobe 1 out of 6 points 5 | give zero stars to halo ghosts of onyx 6 | give this novel a score of 5 7 | give the current series four of 6 points 8 | give 4 out of 6 points to the spirit ring chronicle 9 | give two stars out of 6 to 36 children 10 | rate the sneetches and other stories a three 11 | rate the current series four stars 12 | rate this book a 4 out of 6 13 | rate the current novel 5 of 6 stars 14 | rate this book a 1 15 | give zero out of 6 to the current album 16 | give this album 5 points 17 | rate the mystery of the tolling bell series 4 stars 18 | give the current novel two stars 19 | give the current book 4 stars 20 | give joe magarac and his usa citizen papers 5 points 21 | rate the guilty 0 of 6 points 22 | rate this textbook four out of 6 23 | give the catedral series four stars 24 | reminiscences of the anti japanese guerillas chronicle deserves zero points out of 6 for a rating 25 | give small screen , big picture a 0 out of 6 rating 26 | gods and pawns should get a three 27 | give zero stars to this textbook 28 | rate the current novel a 4 out of 6 stars 29 | rate the book the atmospheric railway 5 out of 6 30 | rate black boy 4 out of 6 31 | rate the chronicle current 1 star 32 | mark this album a score of 5 33 | rate the current novel zero out of 6 34 | rate the current novel a 2 35 | give the giant devil dingo 4 points 36 | rate this current novel two out of 6 37 | give monthly index of medical specialities a two out of 6 rating 38 | rate this novel 2 out of 6 points 39 | rate the current novel 3 stars 40 | rate the current essay zero out of 6 stars 41 | rate this current album 0 stars 42 | give a brief stop on the road from auschwitz 1 out of 6 stars 43 | rate this album 4 out of 6 stars 44 | rate hate that cat 1 out of 6 stars 45 | give my current book one of 6 stars 46 | rate current novel one stars 47 | give five out of 6 points to this album 48 | give a rating of 2 to juneteenth 49 | rate ruth five out of 6 points 50 | rate the sea of trolls 1 stars out of 6 51 | give the zenith angle one out of 6 points 52 | give zero stars to rhialto the marvellous 53 | give the current book a zero of 6 54 | rate personal demons 0 out of 6 points 55 | rate the current series a 4 56 | give one of 6 points to who will cry when you die 57 | give zero out of 6 stars to this album 58 | give this novel 2 stars 59 | rate the 8 week cholesterol cure three out of 6 60 | rate this novel 3 out of 6 points 61 | rate the lives of john lennon five points 62 | give the american scene 2 of 6 stars 63 | rate this textbook a one 64 | give summer of the swans 1 points 65 | give the current textbook a rating of five 66 | give 4 points to the person and the common good 67 | give a four rating to a world apart 68 | rate this chronicle 0 points 69 | give wilco learning how to die a rating of four points 70 | rate this saga two out of 6 71 | rate the gift imagination and the erotic life of property five stars 72 | rate neverwhere four out of 6 73 | rate in the company of cheerful ladies a zero out of 6 74 | give one start to the current book 75 | give this chronicle a 2 rating 76 | rate this essay a 1 77 | out of 6 , give rivers of babylon a 1 78 | give 5 of 6 stars to expressive processing 79 | rate the ghost house series a one 80 | rate know ye not agincourt \? 2 out of 6 stars 81 | i would rate theft a love story four out of 6 stars 82 | rate the further adventures of the joker four stars 83 | give 0 rating to in the heart of the country 84 | give 1 out of 6 rating to the current textbook 85 | give the current chronicle five of 6 points 86 | rate cotton comes to harlem a 2 87 | give this album one stars 88 | rate the adventures of augie march one points 89 | rate soul music a 0 90 | give hindu temples what happened to them a 5 out of 6 stars 91 | give this novel a 1 92 | rate the current textbook 1 out of 6 93 | give this textbook 0 out of 6 stars 94 | give the crystal snare 5 stars 95 | rate this saga two out of 6 96 | give wilco learning how to die a rating of four points 97 | rate this book 3 stars out of 6 98 | rate the three junes one out of 6 99 | give four stars to the broken window 100 | rate the current series 4 points 101 | -------------------------------------------------------------------------------- /data/processed_data/SearchCreativeWork/SearchCreativeWork_test.txt: -------------------------------------------------------------------------------- 1 | wish to find the movie the heart beat __label__SearchCreativeWork 2 | please look up the tv show , vanity __label__SearchCreativeWork 3 | get me the elvis' christmas album tv show __label__SearchCreativeWork 4 | please find me the saga , the deep six __label__SearchCreativeWork 5 | wish to see the photograph with the name live right here __label__SearchCreativeWork 6 | looking for a novel called death march __label__SearchCreativeWork 7 | can you find me the work , the curse of oak island \? __label__SearchCreativeWork 8 | please get me the sacred and profane love machine game __label__SearchCreativeWork 9 | need a creative work called hit by love __label__SearchCreativeWork 10 | search for the trailer for the office __label__SearchCreativeWork 11 | looking for a creative work called plant ecology __label__SearchCreativeWork 12 | find the television show to me __label__SearchCreativeWork 13 | can you please find me the saga chump change \? __label__SearchCreativeWork 14 | can you find me the ridiculous 6 book \? __label__SearchCreativeWork 15 | please fine me the tv series , now we are married __label__SearchCreativeWork 16 | please look up the work , bachelor pad __label__SearchCreativeWork 17 | please help me find the late night heartbroken blues television show __label__SearchCreativeWork 18 | please help me find , bend it like beckham the musical __label__SearchCreativeWork 19 | please look up the tv series parables for wooden ears __label__SearchCreativeWork 20 | can you find me , hey man \? __label__SearchCreativeWork 21 | please search for switched __label__SearchCreativeWork 22 | can you get me the controlled conversations tv series \? __label__SearchCreativeWork 23 | please look up the song the mad magician __label__SearchCreativeWork 24 | please search for the tv show , the best of white lion __label__SearchCreativeWork 25 | please find me phineas redux __label__SearchCreativeWork 26 | get me the procession of ants tv show __label__SearchCreativeWork 27 | looking for a game called phinally phamous __label__SearchCreativeWork 28 | can you search the daring youth saga \? __label__SearchCreativeWork 29 | look for the book the girl who was plugged in __label__SearchCreativeWork 30 | find me a tv show called baby blue __label__SearchCreativeWork 31 | search for appalachian journey __label__SearchCreativeWork 32 | look for the television show meet the prince __label__SearchCreativeWork 33 | can you find me cracks the safe \? __label__SearchCreativeWork 34 | please help me search the hell money saga __label__SearchCreativeWork 35 | get me the secret south song __label__SearchCreativeWork 36 | can you find me the work titled music for millions \? __label__SearchCreativeWork 37 | please search for the painting titled this is the night __label__SearchCreativeWork 38 | could you locate the epic conditions picture \? __label__SearchCreativeWork 39 | get me the trailer of good morning sunshine __label__SearchCreativeWork 40 | please search the an introduction to karl marx painting __label__SearchCreativeWork 41 | can you find me the blue spring trailer \? __label__SearchCreativeWork 42 | could you find the tv series the approach __label__SearchCreativeWork 43 | search for the tv show , a lawless street __label__SearchCreativeWork 44 | please look up three essays on the theory of sexuality show __label__SearchCreativeWork 45 | please get me the compulsive disclosure song __label__SearchCreativeWork 46 | can you look up the molecular oncology saga \? __label__SearchCreativeWork 47 | search for the sound of one hand clapping __label__SearchCreativeWork 48 | find the creative work deadly weapons __label__SearchCreativeWork 49 | need the creative work called the logic of scientific discovery __label__SearchCreativeWork 50 | can you find me the national anthem of the ancient britons television show \? __label__SearchCreativeWork 51 | can you please find me the harry hood saga \? __label__SearchCreativeWork 52 | can you find me the work , bible translations into hawaii pidgin \? __label__SearchCreativeWork 53 | please look up and find me monty python live at the hollywood bowl __label__SearchCreativeWork 54 | please search for mary __label__SearchCreativeWork 55 | please search the game atla all this life allows __label__SearchCreativeWork 56 | find me the novel with the name to lose my life __label__SearchCreativeWork 57 | looking for a song with the title of live at the kings center __label__SearchCreativeWork 58 | can you find the american bison photograph \? __label__SearchCreativeWork 59 | can you find me the free for all show \? __label__SearchCreativeWork 60 | please find me the olympia 74 soundtrack __label__SearchCreativeWork 61 | look for the album slave to the grind __label__SearchCreativeWork 62 | please find me the projekt the new face of goth __label__SearchCreativeWork 63 | can you get me the message from god saga \? __label__SearchCreativeWork 64 | find me the soundtrack a honeymoon adventure __label__SearchCreativeWork 65 | please get me the henderson kids saga __label__SearchCreativeWork 66 | find the movie splendor in the grass __label__SearchCreativeWork 67 | am looking for a book with the title free to play __label__SearchCreativeWork 68 | look for the tv series jersey boys __label__SearchCreativeWork 69 | can you search the book , paris when it sizzles \? __label__SearchCreativeWork 70 | looking for a painting with the title with you __label__SearchCreativeWork 71 | please find me the classified book __label__SearchCreativeWork 72 | look for the show v the new mythology suite __label__SearchCreativeWork 73 | find the creative work face down __label__SearchCreativeWork 74 | find four songs __label__SearchCreativeWork 75 | find me the soundtrack live at the greek theatre __label__SearchCreativeWork 76 | please search for the television show , episodi di the blacklist __label__SearchCreativeWork 77 | find a creative work called fire in the hole __label__SearchCreativeWork 78 | looking for the picture with the name of who made stevie crye \? __label__SearchCreativeWork 79 | look for the album wolves within __label__SearchCreativeWork 80 | find the album orphan girl at the cemetery __label__SearchCreativeWork 81 | please find me the journal of the british astronomical association movie __label__SearchCreativeWork 82 | find the tv show the daydreamer __label__SearchCreativeWork 83 | can you please get me the book dracula 5 the blood legacy \? __label__SearchCreativeWork 84 | please look up the novel , live to dance __label__SearchCreativeWork 85 | please find me the video game titled 20 hours in america __label__SearchCreativeWork 86 | find the creative work the devil in stitches __label__SearchCreativeWork 87 | please look up the work , prophets __label__SearchCreativeWork 88 | i'm looking for welcome to the canteen __label__SearchCreativeWork 89 | please search for the journal of official statistics show __label__SearchCreativeWork 90 | please look up show biz blues photograph __label__SearchCreativeWork 91 | please search the woodsmen of the west __label__SearchCreativeWork 92 | can you find the creative works associated with caryl marilyn real friends __label__SearchCreativeWork 93 | please get me the dead soul saga __label__SearchCreativeWork 94 | please search the live from leeds album __label__SearchCreativeWork 95 | please look up the johnny english la rinascita painting __label__SearchCreativeWork 96 | can you find me the sword with no name trailer \? __label__SearchCreativeWork 97 | i wish to watch the fold trailer , please search __label__SearchCreativeWork 98 | can you find me the almost human painting \? __label__SearchCreativeWork 99 | please find me the work , serious awesomeness ! __label__SearchCreativeWork 100 | search for the game difficult loves __label__SearchCreativeWork 101 | -------------------------------------------------------------------------------- /data/processed_data/SearchCreativeWork_test.txt: -------------------------------------------------------------------------------- 1 | wish to find the movie the heart beat 2 | please look up the tv show , vanity 3 | get me the elvis' christmas album tv show 4 | please find me the saga , the deep six 5 | wish to see the photograph with the name live right here 6 | looking for a novel called death march 7 | can you find me the work , the curse of oak island \? 8 | please get me the sacred and profane love machine game 9 | need a creative work called hit by love 10 | search for the trailer for the office 11 | looking for a creative work called plant ecology 12 | find the television show to me 13 | can you please find me the saga chump change \? 14 | can you find me the ridiculous 6 book \? 15 | please fine me the tv series , now we are married 16 | please look up the work , bachelor pad 17 | please help me find the late night heartbroken blues television show 18 | please help me find , bend it like beckham the musical 19 | please look up the tv series parables for wooden ears 20 | can you find me , hey man \? 21 | please search for switched 22 | can you get me the controlled conversations tv series \? 23 | please look up the song the mad magician 24 | please search for the tv show , the best of white lion 25 | please find me phineas redux 26 | get me the procession of ants tv show 27 | looking for a game called phinally phamous 28 | can you search the daring youth saga \? 29 | look for the book the girl who was plugged in 30 | find me a tv show called baby blue 31 | search for appalachian journey 32 | look for the television show meet the prince 33 | can you find me cracks the safe \? 34 | please help me search the hell money saga 35 | get me the secret south song 36 | can you find me the work titled music for millions \? 37 | please search for the painting titled this is the night 38 | could you locate the epic conditions picture \? 39 | get me the trailer of good morning sunshine 40 | please search the an introduction to karl marx painting 41 | can you find me the blue spring trailer \? 42 | could you find the tv series the approach 43 | search for the tv show , a lawless street 44 | please look up three essays on the theory of sexuality show 45 | please get me the compulsive disclosure song 46 | can you look up the molecular oncology saga \? 47 | search for the sound of one hand clapping 48 | find the creative work deadly weapons 49 | need the creative work called the logic of scientific discovery 50 | can you find me the national anthem of the ancient britons television show \? 51 | can you please find me the harry hood saga \? 52 | can you find me the work , bible translations into hawaii pidgin \? 53 | please look up and find me monty python live at the hollywood bowl 54 | please search for mary 55 | please search the game atla all this life allows 56 | find me the novel with the name to lose my life 57 | looking for a song with the title of live at the kings center 58 | can you find the american bison photograph \? 59 | can you find me the free for all show \? 60 | please find me the olympia 74 soundtrack 61 | look for the album slave to the grind 62 | please find me the projekt the new face of goth 63 | can you get me the message from god saga \? 64 | find me the soundtrack a honeymoon adventure 65 | please get me the henderson kids saga 66 | find the movie splendor in the grass 67 | am looking for a book with the title free to play 68 | look for the tv series jersey boys 69 | can you search the book , paris when it sizzles \? 70 | looking for a painting with the title with you 71 | please find me the classified book 72 | look for the show v the new mythology suite 73 | find the creative work face down 74 | find four songs 75 | find me the soundtrack live at the greek theatre 76 | please search for the television show , episodi di the blacklist 77 | find a creative work called fire in the hole 78 | looking for the picture with the name of who made stevie crye \? 79 | look for the album wolves within 80 | find the album orphan girl at the cemetery 81 | please find me the journal of the british astronomical association movie 82 | find the tv show the daydreamer 83 | can you please get me the book dracula 5 the blood legacy \? 84 | please look up the novel , live to dance 85 | please find me the video game titled 20 hours in america 86 | find the creative work the devil in stitches 87 | please look up the work , prophets 88 | i'm looking for welcome to the canteen 89 | please search for the journal of official statistics show 90 | please look up show biz blues photograph 91 | please search the woodsmen of the west 92 | can you find the creative works associated with caryl marilyn real friends 93 | please get me the dead soul saga 94 | please search the live from leeds album 95 | please look up the johnny english la rinascita painting 96 | can you find me the sword with no name trailer \? 97 | i wish to watch the fold trailer , please search 98 | can you find me the almost human painting \? 99 | please find me the work , serious awesomeness ! 100 | search for the game difficult loves 101 | -------------------------------------------------------------------------------- /data/processed_data/SearchScreeningEvent/SearchScreeningEvent_test.txt: -------------------------------------------------------------------------------- 1 | is babar king of the elephants playing __label__SearchScreeningEvent 2 | is the ghost playing __label__SearchScreeningEvent 3 | is bartok the magnificent playing at seven am \? __label__SearchScreeningEvent 4 | what 's the movie schedule __label__SearchScreeningEvent 5 | i want to see jla adventures trapped in time __label__SearchScreeningEvent 6 | when is the fox and the child playing in this cinema __label__SearchScreeningEvent 7 | show me the schedule for rat rod rockers ! __label__SearchScreeningEvent 8 | is any which way you can playing in 15 seconds __label__SearchScreeningEvent 9 | i want to see the portrait of a lady at the nearest cinema __label__SearchScreeningEvent 10 | where can i see the prime ministers the pioneers __label__SearchScreeningEvent 11 | i need to find the movie theatre showing the crooked web closest to me __label__SearchScreeningEvent 12 | i want to see while the sun shines at the closest movie house __label__SearchScreeningEvent 13 | i want to see those kids from town , when will it be showing \? __label__SearchScreeningEvent 14 | find the schedule for the comedian at santikos theatres __label__SearchScreeningEvent 15 | what are the movie schedules for my favorite theaters __label__SearchScreeningEvent 16 | what are the movies showing in the neighbourhood __label__SearchScreeningEvent 17 | is without witness playing twenty two hours from now __label__SearchScreeningEvent 18 | i need animated movies in the area for dinner time __label__SearchScreeningEvent 19 | i want to see i dream of jeanie in a movie theatre __label__SearchScreeningEvent 20 | can i see ellis island revisited in 1 minute __label__SearchScreeningEvent 21 | i want animated movies at mjr theatres __label__SearchScreeningEvent 22 | show me the schedule for the oblong box __label__SearchScreeningEvent 23 | i want to know if there are any movies playing in the area __label__SearchScreeningEvent 24 | is what a wonderful place showing at cinemark theatres __label__SearchScreeningEvent 25 | show the closest movie theatre that shows boycott __label__SearchScreeningEvent 26 | i want to see doa dead or alive at loews cineplex entertainment __label__SearchScreeningEvent 27 | is the nightmare showing six hours from now at the nearest cinema __label__SearchScreeningEvent 28 | what is the nearest movie house with window connection playing at lunch __label__SearchScreeningEvent 29 | is patrick still lives showing at amc theaters __label__SearchScreeningEvent 30 | fine the movie schedules for the wanda group __label__SearchScreeningEvent 31 | give me the movie schedule nearby __label__SearchScreeningEvent 32 | find the schedule at the douglas theatre company __label__SearchScreeningEvent 33 | show me the movies at harkins theatres __label__SearchScreeningEvent 34 | what movies at star theatres __label__SearchScreeningEvent 35 | i want a movie schedule __label__SearchScreeningEvent 36 | can i get the movie times __label__SearchScreeningEvent 37 | i want to see medal for the general __label__SearchScreeningEvent 38 | can i get the times for movies in the neighbourhood __label__SearchScreeningEvent 39 | may i have the movie schedules for speakeasy theaters __label__SearchScreeningEvent 40 | find animated movies close by __label__SearchScreeningEvent 41 | is american primitive showing in santikos theatres __label__SearchScreeningEvent 42 | what are the movie schedules in the neighborhood __label__SearchScreeningEvent 43 | check the schedule for bow tie cinemas __label__SearchScreeningEvent 44 | check the timings for snowbound at the closest movie theatre __label__SearchScreeningEvent 45 | what are the movie times at caribbean cinemas __label__SearchScreeningEvent 46 | i need films in the neighborhood __label__SearchScreeningEvent 47 | show the movie schedules in the neighborhood __label__SearchScreeningEvent 48 | where 's the nearest movie house showing foreign films __label__SearchScreeningEvent 49 | what movies are showing now at the closest cinema \? __label__SearchScreeningEvent 50 | is rumor has it playing __label__SearchScreeningEvent 51 | i need a list of speakeasy theaters movie times __label__SearchScreeningEvent 52 | when is the outer space connection playing at the nearest cinema __label__SearchScreeningEvent 53 | find the movie times at harkins theatres __label__SearchScreeningEvent 54 | find the films at century theatres __label__SearchScreeningEvent 55 | show the animated movies playing in the neighbourhood __label__SearchScreeningEvent 56 | i want to see fear chamber __label__SearchScreeningEvent 57 | show me southern theatres movie times __label__SearchScreeningEvent 58 | is the unnaturals showing at 13 __label__SearchScreeningEvent 59 | is no time to be young showing at amc theaters __label__SearchScreeningEvent 60 | find the movie schedules for regal entertainment group __label__SearchScreeningEvent 61 | i want to see shattered image __label__SearchScreeningEvent 62 | find the schedule at star theatres __label__SearchScreeningEvent 63 | will i think i do be playing at 7 pm \? __label__SearchScreeningEvent 64 | show me the schedule for arclight hollywood for only animated movies __label__SearchScreeningEvent 65 | find the schedule for great mail robbery __label__SearchScreeningEvent 66 | give me the movies in the neighborhood __label__SearchScreeningEvent 67 | what movies are playing close by __label__SearchScreeningEvent 68 | is the two gladiators playing __label__SearchScreeningEvent 69 | what 's the movie schedule for great escape theatres __label__SearchScreeningEvent 70 | find the movie schedule close by __label__SearchScreeningEvent 71 | i want to see outcast __label__SearchScreeningEvent 72 | show me the schedule of movie the great gildersleeve near movie house __label__SearchScreeningEvent 73 | i need times for a yiddish world remembered at dipson theatres __label__SearchScreeningEvent 74 | find the movie schedules at goodrich quality theaters __label__SearchScreeningEvent 75 | show me the movie schedule in the neighbourhood __label__SearchScreeningEvent 76 | show me the movie times for films nearby __label__SearchScreeningEvent 77 | show the movie times for animated movies in the neighbourhood __label__SearchScreeningEvent 78 | is the eye infinity playing at general cinema corporation __label__SearchScreeningEvent 79 | can you check the timings for super sweet 16 the movie \? __label__SearchScreeningEvent 80 | is we are northern lights playing in any movie theatre __label__SearchScreeningEvent 81 | what times will the young swordsman be showing at my cinema \? __label__SearchScreeningEvent 82 | show the sexy dance 2times at the closest movie house __label__SearchScreeningEvent 83 | what are some close by animated movies showing __label__SearchScreeningEvent 84 | movie schedules close by for animated movies __label__SearchScreeningEvent 85 | what films are playing close by __label__SearchScreeningEvent 86 | find the movie schedule in the area __label__SearchScreeningEvent 87 | is cowboy canteen playing __label__SearchScreeningEvent 88 | is rare birds showing at the nearest movie theatre at noon __label__SearchScreeningEvent 89 | what are the movie times __label__SearchScreeningEvent 90 | where can i find the movie schedules __label__SearchScreeningEvent 91 | find the movie schedule for north american cinemas in eleven seconds __label__SearchScreeningEvent 92 | find the nearest cinema with movies playing __label__SearchScreeningEvent 93 | what are the movie times __label__SearchScreeningEvent 94 | what are the times for the gingerbread man __label__SearchScreeningEvent 95 | what films are playing close by __label__SearchScreeningEvent 96 | is any cinema playing the spirit of youth __label__SearchScreeningEvent 97 | what are the movie times for animated movies in the neighbourhood __label__SearchScreeningEvent 98 | what 's the movie schedule at great escape theatres __label__SearchScreeningEvent 99 | show the times for cheers for miss bishop at dipson theatres __label__SearchScreeningEvent 100 | i want to see married to the enemy 2 at a cinema __label__SearchScreeningEvent 101 | -------------------------------------------------------------------------------- /data/processed_data/SearchScreeningEvent_test.txt: -------------------------------------------------------------------------------- 1 | is babar king of the elephants playing 2 | is the ghost playing 3 | is bartok the magnificent playing at seven am \? 4 | what 's the movie schedule 5 | i want to see jla adventures trapped in time 6 | when is the fox and the child playing in this cinema 7 | show me the schedule for rat rod rockers ! 8 | is any which way you can playing in 15 seconds 9 | i want to see the portrait of a lady at the nearest cinema 10 | where can i see the prime ministers the pioneers 11 | i need to find the movie theatre showing the crooked web closest to me 12 | i want to see while the sun shines at the closest movie house 13 | i want to see those kids from town , when will it be showing \? 14 | find the schedule for the comedian at santikos theatres 15 | what are the movie schedules for my favorite theaters 16 | what are the movies showing in the neighbourhood 17 | is without witness playing twenty two hours from now 18 | i need animated movies in the area for dinner time 19 | i want to see i dream of jeanie in a movie theatre 20 | can i see ellis island revisited in 1 minute 21 | i want animated movies at mjr theatres 22 | show me the schedule for the oblong box 23 | i want to know if there are any movies playing in the area 24 | is what a wonderful place showing at cinemark theatres 25 | show the closest movie theatre that shows boycott 26 | i want to see doa dead or alive at loews cineplex entertainment 27 | is the nightmare showing six hours from now at the nearest cinema 28 | what is the nearest movie house with window connection playing at lunch 29 | is patrick still lives showing at amc theaters 30 | fine the movie schedules for the wanda group 31 | give me the movie schedule nearby 32 | find the schedule at the douglas theatre company 33 | show me the movies at harkins theatres 34 | what movies at star theatres 35 | i want a movie schedule 36 | can i get the movie times 37 | i want to see medal for the general 38 | can i get the times for movies in the neighbourhood 39 | may i have the movie schedules for speakeasy theaters 40 | find animated movies close by 41 | is american primitive showing in santikos theatres 42 | what are the movie schedules in the neighborhood 43 | check the schedule for bow tie cinemas 44 | check the timings for snowbound at the closest movie theatre 45 | what are the movie times at caribbean cinemas 46 | i need films in the neighborhood 47 | show the movie schedules in the neighborhood 48 | where 's the nearest movie house showing foreign films 49 | what movies are showing now at the closest cinema \? 50 | is rumor has it playing 51 | i need a list of speakeasy theaters movie times 52 | when is the outer space connection playing at the nearest cinema 53 | find the movie times at harkins theatres 54 | find the films at century theatres 55 | show the animated movies playing in the neighbourhood 56 | i want to see fear chamber 57 | show me southern theatres movie times 58 | is the unnaturals showing at 13 59 | is no time to be young showing at amc theaters 60 | find the movie schedules for regal entertainment group 61 | i want to see shattered image 62 | find the schedule at star theatres 63 | will i think i do be playing at 7 pm \? 64 | show me the schedule for arclight hollywood for only animated movies 65 | find the schedule for great mail robbery 66 | give me the movies in the neighborhood 67 | what movies are playing close by 68 | is the two gladiators playing 69 | what 's the movie schedule for great escape theatres 70 | find the movie schedule close by 71 | i want to see outcast 72 | show me the schedule of movie the great gildersleeve near movie house 73 | i need times for a yiddish world remembered at dipson theatres 74 | find the movie schedules at goodrich quality theaters 75 | show me the movie schedule in the neighbourhood 76 | show me the movie times for films nearby 77 | show the movie times for animated movies in the neighbourhood 78 | is the eye infinity playing at general cinema corporation 79 | can you check the timings for super sweet 16 the movie \? 80 | is we are northern lights playing in any movie theatre 81 | what times will the young swordsman be showing at my cinema \? 82 | show the sexy dance 2times at the closest movie house 83 | what are some close by animated movies showing 84 | movie schedules close by for animated movies 85 | what films are playing close by 86 | find the movie schedule in the area 87 | is cowboy canteen playing 88 | is rare birds showing at the nearest movie theatre at noon 89 | what are the movie times 90 | where can i find the movie schedules 91 | find the movie schedule for north american cinemas in eleven seconds 92 | find the nearest cinema with movies playing 93 | what are the movie times 94 | what are the times for the gingerbread man 95 | what films are playing close by 96 | is any cinema playing the spirit of youth 97 | what are the movie times for animated movies in the neighbourhood 98 | what 's the movie schedule at great escape theatres 99 | show the times for cheers for miss bishop at dipson theatres 100 | i want to see married to the enemy 2 at a cinema 101 | -------------------------------------------------------------------------------- /data/raw_json_data/AddToPlaylist/Alexa_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "AddToPlaylist": { 4 | "recall": { 5 | "70": [ 6 | 0.99, 7 | 0.99, 8 | 1.0 9 | ] 10 | }, 11 | "slots": { 12 | "playlist": { 13 | "recall": { 14 | "70": [ 15 | 0.86, 16 | 0.85, 17 | 0.8 18 | ] 19 | }, 20 | "rate": 300, 21 | "precision": { 22 | "70": [ 23 | 0.8775510204081632, 24 | 0.8673469387755102, 25 | 0.8 26 | ] 27 | } 28 | }, 29 | "music_item": { 30 | "recall": { 31 | "70": [ 32 | 0.5272727272727272, 33 | 0.4727272727272727, 34 | 0.6 35 | ] 36 | }, 37 | "rate": 139, 38 | "precision": { 39 | "70": [ 40 | 0.9666666666666667, 41 | 1.0, 42 | 0.9705882352941176 43 | ] 44 | } 45 | }, 46 | "playlist_owner": { 47 | "recall": { 48 | "70": [ 49 | 0.9074074074074074, 50 | 0.9074074074074074, 51 | 0.9444444444444444 52 | ] 53 | }, 54 | "rate": 194, 55 | "precision": { 56 | "70": [ 57 | 1.0, 58 | 1.0, 59 | 1.0 60 | ] 61 | } 62 | }, 63 | "entity_name": { 64 | "recall": { 65 | "70": [ 66 | 0.6111111111111112, 67 | 0.5555555555555556, 68 | 0.7222222222222222 69 | ] 70 | }, 71 | "rate": 89, 72 | "precision": { 73 | "70": [ 74 | 0.28205128205128205, 75 | 0.35714285714285715, 76 | 0.38235294117647056 77 | ] 78 | } 79 | }, 80 | "artist": { 81 | "recall": { 82 | "70": [ 83 | 0.32608695652173914, 84 | 0.45652173913043476, 85 | 0.41304347826086957 86 | ] 87 | }, 88 | "rate": 113, 89 | "precision": { 90 | "70": [ 91 | 0.36585365853658536, 92 | 0.38181818181818183, 93 | 0.5135135135135135 94 | ] 95 | } 96 | } 97 | }, 98 | "precision": { 99 | "70": [ 100 | 1.0, 101 | 1.0, 102 | 1.0 103 | ] 104 | } 105 | } 106 | }, 107 | "execution_time": { 108 | "pred": { 109 | "70": [ 110 | 0.030013699531555176, 111 | 0.030045461654663087, 112 | 0.030027780532836914 113 | ] 114 | }, 115 | "fit": { 116 | "70": [ 117 | 3.000154972076416, 118 | 3.0000460147857666, 119 | 3.0026140213012695 120 | ] 121 | } 122 | }, 123 | "validation_path": "../alexa-bench/alexa_bench/resources/predictions_relabeled/", 124 | "training_path": "../alexa-bench/alexa_bench/resources/", 125 | "training_name": "AddToPlaylist", 126 | "model_name": "Alexa", 127 | "training_sizes": [ 128 | 70 129 | ] 130 | } -------------------------------------------------------------------------------- /data/raw_json_data/AddToPlaylist/Luis_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "AddToPlaylist": { 4 | "recall": { 5 | "10": [ 6 | 1.0, 7 | 1.0, 8 | 1.0 9 | ], 10 | "20": [ 11 | 1.0, 12 | 1.0, 13 | 1.0 14 | ], 15 | "50": [ 16 | 1.0, 17 | 1.0, 18 | 1.0 19 | ], 20 | "70": [ 21 | 1.0, 22 | 1.0, 23 | 1.0 24 | ] 25 | }, 26 | "slots": { 27 | "playlist": { 28 | "recall": { 29 | "10": [ 30 | 0.11, 31 | 0.61, 32 | 0.67 33 | ], 34 | "20": [ 35 | 0.64, 36 | 0.64, 37 | 0.73 38 | ], 39 | "50": [ 40 | 0.77, 41 | 0.77, 42 | 0.81 43 | ], 44 | "70": [ 45 | 0.85, 46 | 0.85, 47 | 0.8 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | 1.0, 54 | 1.0, 55 | 1.0 56 | ], 57 | "20": [ 58 | 1.0, 59 | 1.0, 60 | 1.0 61 | ], 62 | "50": [ 63 | 1.0, 64 | 1.0, 65 | 1.0 66 | ], 67 | "70": [ 68 | 1.0, 69 | 1.0, 70 | 1.0 71 | ] 72 | } 73 | }, 74 | "music_item": { 75 | "recall": { 76 | "10": [ 77 | 0.36363636363636365, 78 | 0.43636363636363634, 79 | 0.2 80 | ], 81 | "20": [ 82 | 0.5272727272727272, 83 | 0.5272727272727272, 84 | 0.5636363636363636 85 | ], 86 | "50": [ 87 | 0.9090909090909091, 88 | 0.9090909090909091, 89 | 0.7636363636363637 90 | ], 91 | "70": [ 92 | 0.9454545454545454, 93 | 0.9454545454545454, 94 | 0.9636363636363636 95 | ] 96 | }, 97 | "rate": 46, 98 | "precision": { 99 | "10": [ 100 | 1.0, 101 | 1.0, 102 | 1.0 103 | ], 104 | "20": [ 105 | 1.0, 106 | 1.0, 107 | 1.0 108 | ], 109 | "50": [ 110 | 0.9803921568627451, 111 | 0.9803921568627451, 112 | 1.0 113 | ], 114 | "70": [ 115 | 0.9629629629629629, 116 | 0.9629629629629629, 117 | 0.9636363636363636 118 | ] 119 | } 120 | }, 121 | "playlist_owner": { 122 | "recall": { 123 | "10": [ 124 | 0.8888888888888888, 125 | 0.7777777777777778, 126 | 0.7777777777777778 127 | ], 128 | "20": [ 129 | 0.9074074074074074, 130 | 0.9074074074074074, 131 | 0.8148148148148148 132 | ], 133 | "50": [ 134 | 0.9444444444444444, 135 | 0.9444444444444444, 136 | 0.9444444444444444 137 | ], 138 | "70": [ 139 | 0.9444444444444444, 140 | 0.9444444444444444, 141 | 0.9444444444444444 142 | ] 143 | }, 144 | "rate": 64, 145 | "precision": { 146 | "10": [ 147 | 1.0, 148 | 1.0, 149 | 1.0 150 | ], 151 | "20": [ 152 | 1.0, 153 | 1.0, 154 | 1.0 155 | ], 156 | "50": [ 157 | 1.0, 158 | 1.0, 159 | 1.0 160 | ], 161 | "70": [ 162 | 1.0, 163 | 1.0, 164 | 1.0 165 | ] 166 | } 167 | }, 168 | "entity_name": { 169 | "recall": { 170 | "10": [ 171 | 0.0, 172 | 0.0, 173 | 0.1111111111111111 174 | ], 175 | "20": [ 176 | 0.05555555555555555, 177 | 0.05555555555555555, 178 | 0.1111111111111111 179 | ], 180 | "50": [ 181 | 0.05555555555555555, 182 | 0.05555555555555555, 183 | 0.1111111111111111 184 | ], 185 | "70": [ 186 | 0.0, 187 | 0.0, 188 | 0.0 189 | ] 190 | }, 191 | "rate": 29, 192 | "precision": { 193 | "10": [ 194 | NaN, 195 | NaN, 196 | 0.6666666666666666 197 | ], 198 | "20": [ 199 | 0.5, 200 | 0.5, 201 | 0.6666666666666666 202 | ], 203 | "50": [ 204 | 0.3333333333333333, 205 | 0.3333333333333333, 206 | 0.5 207 | ], 208 | "70": [ 209 | 0.0, 210 | 0.0, 211 | 0.0 212 | ] 213 | } 214 | }, 215 | "artist": { 216 | "recall": { 217 | "10": [ 218 | 0.0, 219 | 0.0, 220 | 0.0 221 | ], 222 | "20": [ 223 | 0.0, 224 | 0.0, 225 | 0.08695652173913043 226 | ], 227 | "50": [ 228 | 0.10869565217391304, 229 | 0.10869565217391304, 230 | 0.10869565217391304 231 | ], 232 | "70": [ 233 | 0.15217391304347827, 234 | 0.15217391304347827, 235 | 0.13043478260869565 236 | ] 237 | }, 238 | "rate": 38, 239 | "precision": { 240 | "10": [ 241 | NaN, 242 | NaN, 243 | NaN 244 | ], 245 | "20": [ 246 | NaN, 247 | NaN, 248 | 1.0 249 | ], 250 | "50": [ 251 | 1.0, 252 | 1.0, 253 | 1.0 254 | ], 255 | "70": [ 256 | 0.875, 257 | 0.875, 258 | 0.75 259 | ] 260 | } 261 | } 262 | }, 263 | "precision": { 264 | "10": [ 265 | 1.0, 266 | 1.0, 267 | 1.0 268 | ], 269 | "20": [ 270 | 1.0, 271 | 1.0, 272 | 1.0 273 | ], 274 | "50": [ 275 | 1.0, 276 | 1.0, 277 | 1.0 278 | ], 279 | "70": [ 280 | 1.0, 281 | 1.0, 282 | 1.0 283 | ] 284 | } 285 | } 286 | }, 287 | "execution_time": { 288 | "pred": { 289 | "10": [ 290 | 0.40758161067962645, 291 | 0.4702987289428711, 292 | 0.3975417113304138 293 | ], 294 | "20": [ 295 | 0.3986947512626648, 296 | 0.4452948999404907, 297 | 0.403567841053009 298 | ], 299 | "50": [ 300 | 0.4285732102394104, 301 | 0.42478991985321046, 302 | 0.4033720183372498 303 | ], 304 | "70": [ 305 | 0.39888944149017336, 306 | 0.41414758920669553, 307 | 0.40927855014801023 308 | ] 309 | }, 310 | "fit": { 311 | "10": [ 312 | 255.05641293525696, 313 | 193.69374012947083, 314 | 203.8070662021637 315 | ], 316 | "20": [ 317 | 207.07637310028076, 318 | 216.4292550086975, 319 | 215.54409193992615 320 | ], 321 | "50": [ 322 | 219.1778860092163, 323 | 243.33319091796875, 324 | 207.10618996620178 325 | ], 326 | "70": [ 327 | 260.4378180503845, 328 | 268.9427089691162, 329 | 242.18272399902344 330 | ] 331 | } 332 | }, 333 | "validation_path": "data/en/amt/bench/bench_may/validate_AddToPlaylist.json", 334 | "training_path": "data/en/amt/bench/bench_may/train_AddToPlaylist.json", 335 | "training_name": "AddToPlaylist", 336 | "model_name": "Luis", 337 | "training_sizes": [ 338 | 10, 339 | 20, 340 | 50, 341 | 70 342 | ] 343 | } -------------------------------------------------------------------------------- /data/raw_json_data/AddToPlaylist/Snips_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "AddToPlaylist": { 4 | "recall": { 5 | "10": [ 6 | 1.0, 7 | 1.0, 8 | 1.0 9 | ], 10 | "20": [ 11 | 1.0, 12 | 1.0, 13 | 1.0 14 | ], 15 | "50": [ 16 | 1.0, 17 | 1.0, 18 | 1.0 19 | ], 20 | "70": [ 21 | 1.0, 22 | 1.0, 23 | 0.99 24 | ] 25 | }, 26 | "slots": { 27 | "playlist": { 28 | "recall": { 29 | "10": [ 30 | 0.74, 31 | 0.83, 32 | 0.76 33 | ], 34 | "20": [ 35 | 0.77, 36 | 0.87, 37 | 0.89 38 | ], 39 | "50": [ 40 | 0.85, 41 | 0.89, 42 | 0.86 43 | ], 44 | "70": [ 45 | 0.87, 46 | 0.9, 47 | 0.91 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | 0.6379310344827587, 54 | 0.7410714285714286, 55 | 0.6785714285714286 56 | ], 57 | "20": [ 58 | 0.6260162601626016, 59 | 0.7565217391304347, 60 | 0.7739130434782608 61 | ], 62 | "50": [ 63 | 0.7798165137614679, 64 | 0.7807017543859649, 65 | 0.7818181818181819 66 | ], 67 | "70": [ 68 | 0.7699115044247787, 69 | 0.8411214953271028, 70 | 0.8666666666666667 71 | ] 72 | } 73 | }, 74 | "music_item": { 75 | "recall": { 76 | "10": [ 77 | 0.6909090909090909, 78 | 0.8545454545454545, 79 | 0.8363636363636363 80 | ], 81 | "20": [ 82 | 0.8545454545454545, 83 | 0.8727272727272727, 84 | 0.8363636363636363 85 | ], 86 | "50": [ 87 | 0.9090909090909091, 88 | 0.9818181818181818, 89 | 0.8727272727272727 90 | ], 91 | "70": [ 92 | 0.9636363636363636, 93 | 1.0, 94 | 0.9090909090909091 95 | ] 96 | }, 97 | "rate": 46, 98 | "precision": { 99 | "10": [ 100 | 0.7037037037037037, 101 | 0.6811594202898551, 102 | 0.7301587301587301 103 | ], 104 | "20": [ 105 | 0.8245614035087719, 106 | 0.7164179104477612, 107 | 0.8679245283018868 108 | ], 109 | "50": [ 110 | 0.8771929824561403, 111 | 0.8181818181818182, 112 | 0.8421052631578947 113 | ], 114 | "70": [ 115 | 0.9464285714285714, 116 | 0.9166666666666666, 117 | 0.8771929824561403 118 | ] 119 | } 120 | }, 121 | "playlist_owner": { 122 | "recall": { 123 | "10": [ 124 | 0.5, 125 | 0.8333333333333334, 126 | 0.8333333333333334 127 | ], 128 | "20": [ 129 | 0.9259259259259259, 130 | 0.9259259259259259, 131 | 0.9074074074074074 132 | ], 133 | "50": [ 134 | 0.9814814814814815, 135 | 0.8888888888888888, 136 | 1.0 137 | ], 138 | "70": [ 139 | 1.0, 140 | 0.9444444444444444, 141 | 0.9814814814814815 142 | ] 143 | }, 144 | "rate": 64, 145 | "precision": { 146 | "10": [ 147 | 0.8181818181818182, 148 | 0.6716417910447762, 149 | 0.9183673469387755 150 | ], 151 | "20": [ 152 | 0.9090909090909091, 153 | 0.9090909090909091, 154 | 0.98 155 | ], 156 | "50": [ 157 | 0.9636363636363636, 158 | 0.9411764705882353, 159 | 0.9473684210526315 160 | ], 161 | "70": [ 162 | 0.9818181818181818, 163 | 1.0, 164 | 0.9636363636363636 165 | ] 166 | } 167 | }, 168 | "entity_name": { 169 | "recall": { 170 | "10": [ 171 | 0.2222222222222222, 172 | 0.1111111111111111, 173 | 0.2222222222222222 174 | ], 175 | "20": [ 176 | 0.2777777777777778, 177 | 0.2222222222222222, 178 | 0.3333333333333333 179 | ], 180 | "50": [ 181 | 0.3888888888888889, 182 | 0.2777777777777778, 183 | 0.6666666666666666 184 | ], 185 | "70": [ 186 | 0.5555555555555556, 187 | 0.3333333333333333, 188 | 0.3888888888888889 189 | ] 190 | }, 191 | "rate": 29, 192 | "precision": { 193 | "10": [ 194 | 0.10810810810810811, 195 | 0.08, 196 | 0.4444444444444444 197 | ], 198 | "20": [ 199 | 0.15625, 200 | 0.12121212121212122, 201 | 0.46153846153846156 202 | ], 203 | "50": [ 204 | 0.28, 205 | 0.20833333333333334, 206 | 0.41379310344827586 207 | ], 208 | "70": [ 209 | 0.47619047619047616, 210 | 0.25, 211 | 0.3888888888888889 212 | ] 213 | } 214 | }, 215 | "artist": { 216 | "recall": { 217 | "10": [ 218 | 0.10869565217391304, 219 | 0.32608695652173914, 220 | 0.6956521739130435 221 | ], 222 | "20": [ 223 | 0.3695652173913043, 224 | 0.2608695652173913, 225 | 0.7391304347826086 226 | ], 227 | "50": [ 228 | 0.6521739130434783, 229 | 0.5434782608695652, 230 | 0.782608695652174 231 | ], 232 | "70": [ 233 | 0.7608695652173914, 234 | 0.6304347826086957, 235 | 0.6739130434782609 236 | ] 237 | }, 238 | "rate": 38, 239 | "precision": { 240 | "10": [ 241 | 0.8333333333333334, 242 | 0.6818181818181818, 243 | 0.7619047619047619 244 | ], 245 | "20": [ 246 | 1.0, 247 | 0.5217391304347826, 248 | 0.6181818181818182 249 | ], 250 | "50": [ 251 | 1.0, 252 | 0.78125, 253 | 0.8571428571428571 254 | ], 255 | "70": [ 256 | 0.875, 257 | 0.8285714285714286, 258 | 0.8157894736842105 259 | ] 260 | } 261 | } 262 | }, 263 | "precision": { 264 | "10": [ 265 | 1.0, 266 | 1.0, 267 | 1.0 268 | ], 269 | "20": [ 270 | 1.0, 271 | 1.0, 272 | 1.0 273 | ], 274 | "50": [ 275 | 1.0, 276 | 1.0, 277 | 1.0 278 | ], 279 | "70": [ 280 | 1.0, 281 | 1.0, 282 | 1.0 283 | ] 284 | } 285 | } 286 | }, 287 | "execution_time": { 288 | "pred": { 289 | "10": [ 290 | 0.01969225883483887, 291 | 0.015240488052368164, 292 | 0.01576357126235962 293 | ], 294 | "20": [ 295 | 0.015320358276367187, 296 | 0.015827860832214356, 297 | 0.015606288909912109 298 | ], 299 | "50": [ 300 | 0.016241941452026367, 301 | 0.020297949314117433, 302 | 0.016072509288787843 303 | ], 304 | "70": [ 305 | 0.016018178462982178, 306 | 0.017858569622039796, 307 | 0.016982600688934327 308 | ] 309 | }, 310 | "fit": { 311 | "10": [ 312 | 4.33983302116394, 313 | 3.5559380054473877, 314 | 3.0274100303649902 315 | ], 316 | "20": [ 317 | 3.346565008163452, 318 | 3.8841171264648438, 319 | 4.1025779247283936 320 | ], 321 | "50": [ 322 | 4.232771158218384, 323 | 4.529258966445923, 324 | 3.820490837097168 325 | ], 326 | "70": [ 327 | 4.40479302406311, 328 | 4.67580509185791, 329 | 3.979642868041992 330 | ] 331 | } 332 | }, 333 | "validation_path": "data/en/amt/bench/bench_may/snips/validate_AddToPlaylist.json", 334 | "training_path": "data/en/amt/bench/bench_may/snips/train_AddToPlaylist.json", 335 | "training_name": "AddToPlaylist", 336 | "model_name": "Snips", 337 | "training_sizes": [ 338 | 10, 339 | 20, 340 | 50, 341 | 70 342 | ] 343 | } -------------------------------------------------------------------------------- /data/raw_json_data/AddToPlaylist/Snips_metrics_full.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "AddToPlaylist": { 4 | "recall": { 5 | "1999": [ 6 | 1.0 7 | ] 8 | }, 9 | "slots": { 10 | "playlist": { 11 | "recall": { 12 | "1999": [ 13 | 0.97 14 | ] 15 | }, 16 | "rate": 100, 17 | "precision": { 18 | "1999": [ 19 | 0.9603960396039604 20 | ] 21 | } 22 | }, 23 | "music_item": { 24 | "recall": { 25 | "1999": [ 26 | 1.0 27 | ] 28 | }, 29 | "rate": 46, 30 | "precision": { 31 | "1999": [ 32 | 0.9821428571428571 33 | ] 34 | } 35 | }, 36 | "playlist_owner": { 37 | "recall": { 38 | "1999": [ 39 | 0.9814814814814815 40 | ] 41 | }, 42 | "rate": 57, 43 | "precision": { 44 | "1999": [ 45 | 0.9814814814814815 46 | ] 47 | } 48 | }, 49 | "entity_name": { 50 | "recall": { 51 | "1999": [ 52 | 0.6111111111111112 53 | ] 54 | }, 55 | "rate": 30, 56 | "precision": { 57 | "1999": [ 58 | 0.7333333333333333 59 | ] 60 | } 61 | }, 62 | "artist": { 63 | "recall": { 64 | "1999": [ 65 | 0.9565217391304348 66 | ] 67 | }, 68 | "rate": 38, 69 | "precision": { 70 | "1999": [ 71 | 0.88 72 | ] 73 | } 74 | } 75 | }, 76 | "precision": { 77 | "1999": [ 78 | 1.0 79 | ] 80 | } 81 | } 82 | }, 83 | "execution_time": { 84 | "pred": { 85 | "1999": [ 86 | 0.025904839038848878 87 | ] 88 | }, 89 | "fit": { 90 | "1999": [ 91 | 53.05514192581177 92 | ] 93 | } 94 | }, 95 | "validation_path": "data/en/amt/bench/heavy/2000/snips/validate_AddToPlaylist.json", 96 | "training_path": "data/en/amt/bench/heavy/2000/snips/train_AddToPlaylist.json", 97 | "training_name": "AddToPlaylist", 98 | "model_name": "Snips", 99 | "training_sizes": [ 100 | 1999 101 | ] 102 | } -------------------------------------------------------------------------------- /data/raw_json_data/AddToPlaylist/Wit_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "AddToPlaylist": { 4 | "recall": { 5 | "10": [ 6 | 0.0, 7 | 0.0, 8 | 0.0 9 | ], 10 | "20": [ 11 | 0.0, 12 | 0.0, 13 | 0.0 14 | ], 15 | "50": [ 16 | 0.0, 17 | 0.0, 18 | 0.0 19 | ], 20 | "70": [ 21 | 0.0, 22 | 0.0, 23 | 0.0 24 | ] 25 | }, 26 | "slots": { 27 | "playlist": { 28 | "recall": { 29 | "10": [ 30 | 0.02, 31 | 0.01, 32 | 0.01 33 | ], 34 | "20": [ 35 | 0.02, 36 | 0.01, 37 | 0.01 38 | ], 39 | "50": [ 40 | 0.55, 41 | 0.03, 42 | 0.07 43 | ], 44 | "70": [ 45 | 0.64, 46 | 0.03, 47 | 0.55 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | 0.5, 54 | 1.0, 55 | 1.0 56 | ], 57 | "20": [ 58 | 0.2857142857142857, 59 | 0.3333333333333333, 60 | 0.3333333333333333 61 | ], 62 | "50": [ 63 | 0.9322033898305084, 64 | 0.3, 65 | 0.5833333333333334 66 | ], 67 | "70": [ 68 | 0.9696969696969697, 69 | 0.3, 70 | 0.859375 71 | ] 72 | } 73 | }, 74 | "music_item": { 75 | "recall": { 76 | "10": [ 77 | 0.7818181818181819, 78 | 0.7818181818181819, 79 | 0.7818181818181819 80 | ], 81 | "20": [ 82 | 0.7818181818181819, 83 | 1.0, 84 | 0.7818181818181819 85 | ], 86 | "50": [ 87 | 0.8727272727272727, 88 | 1.0, 89 | 1.0 90 | ], 91 | "70": [ 92 | 0.8727272727272727, 93 | 1.0, 94 | 0.9090909090909091 95 | ] 96 | }, 97 | "rate": 46, 98 | "precision": { 99 | "10": [ 100 | 0.8775510204081632, 101 | 0.8775510204081632, 102 | 0.8775510204081632 103 | ], 104 | "20": [ 105 | 0.8775510204081632, 106 | 0.9016393442622951, 107 | 0.8958333333333334 108 | ], 109 | "50": [ 110 | 0.8888888888888888, 111 | 0.9016393442622951, 112 | 0.9166666666666666 113 | ], 114 | "70": [ 115 | 0.8888888888888888, 116 | 0.9016393442622951, 117 | 0.8928571428571429 118 | ] 119 | } 120 | }, 121 | "playlist_owner": { 122 | "recall": { 123 | "10": [ 124 | 0.9444444444444444, 125 | 0.9444444444444444, 126 | 0.9444444444444444 127 | ], 128 | "20": [ 129 | 0.9444444444444444, 130 | 0.9444444444444444, 131 | 0.9444444444444444 132 | ], 133 | "50": [ 134 | 0.8148148148148148, 135 | 0.9444444444444444, 136 | 0.9444444444444444 137 | ], 138 | "70": [ 139 | 0.8518518518518519, 140 | 0.9444444444444444, 141 | 0.8518518518518519 142 | ] 143 | }, 144 | "rate": 64, 145 | "precision": { 146 | "10": [ 147 | 1.0, 148 | 1.0, 149 | 1.0 150 | ], 151 | "20": [ 152 | 1.0, 153 | 1.0, 154 | 1.0 155 | ], 156 | "50": [ 157 | 1.0, 158 | 1.0, 159 | 1.0 160 | ], 161 | "70": [ 162 | 1.0, 163 | 1.0, 164 | 1.0 165 | ] 166 | } 167 | }, 168 | "entity_name": { 169 | "recall": { 170 | "10": [ 171 | 0.0, 172 | 0.0, 173 | 0.0 174 | ], 175 | "20": [ 176 | 0.0, 177 | 0.0, 178 | 0.05555555555555555 179 | ], 180 | "50": [ 181 | 0.1111111111111111, 182 | 0.0, 183 | 0.05555555555555555 184 | ], 185 | "70": [ 186 | 0.05555555555555555, 187 | 0.0, 188 | 0.05555555555555555 189 | ] 190 | }, 191 | "rate": 29, 192 | "precision": { 193 | "10": [ 194 | NaN, 195 | NaN, 196 | NaN 197 | ], 198 | "20": [ 199 | NaN, 200 | NaN, 201 | 1.0 202 | ], 203 | "50": [ 204 | 0.16666666666666666, 205 | NaN, 206 | 1.0 207 | ], 208 | "70": [ 209 | 0.2, 210 | NaN, 211 | 0.125 212 | ] 213 | } 214 | }, 215 | "artist": { 216 | "recall": { 217 | "10": [ 218 | 0.0, 219 | 0.0, 220 | 0.0 221 | ], 222 | "20": [ 223 | 0.0, 224 | 0.0, 225 | 0.0 226 | ], 227 | "50": [ 228 | 0.08695652173913043, 229 | 0.0, 230 | 0.0 231 | ], 232 | "70": [ 233 | 0.2608695652173913, 234 | 0.0, 235 | 0.15217391304347827 236 | ] 237 | }, 238 | "rate": 38, 239 | "precision": { 240 | "10": [ 241 | NaN, 242 | NaN, 243 | NaN 244 | ], 245 | "20": [ 246 | NaN, 247 | NaN, 248 | NaN 249 | ], 250 | "50": [ 251 | 0.6666666666666666, 252 | 0.0, 253 | NaN 254 | ], 255 | "70": [ 256 | 0.7058823529411765, 257 | 0.0, 258 | 0.7 259 | ] 260 | } 261 | } 262 | }, 263 | "precision": { 264 | "10": [ 265 | NaN, 266 | NaN, 267 | NaN 268 | ], 269 | "20": [ 270 | NaN, 271 | NaN, 272 | NaN 273 | ], 274 | "50": [ 275 | NaN, 276 | NaN, 277 | NaN 278 | ], 279 | "70": [ 280 | NaN, 281 | NaN, 282 | NaN 283 | ] 284 | } 285 | } 286 | }, 287 | "execution_time": { 288 | "pred": { 289 | "10": [ 290 | 1.3793263411521912, 291 | 1.5975637006759644, 292 | 1.566757049560547 293 | ], 294 | "20": [ 295 | 1.477235929965973, 296 | 1.3136553192138671, 297 | 1.437917068004608 298 | ], 299 | "50": [ 300 | 1.3548951292037963, 301 | 1.3771403193473817, 302 | 1.3484582209587097 303 | ], 304 | "70": [ 305 | 1.566943428516388, 306 | 1.4838984298706055, 307 | 1.3785893988609315 308 | ] 309 | }, 310 | "fit": { 311 | "10": [ 312 | 81.51960706710815, 313 | 80.57115507125854, 314 | 79.74800491333008 315 | ], 316 | "20": [ 317 | 84.08215713500977, 318 | 78.63052701950073, 319 | 86.6970739364624 320 | ], 321 | "50": [ 322 | 50.260215044021606, 323 | 83.12329602241516, 324 | 81.91675400733948 325 | ], 326 | "70": [ 327 | 116.18417501449585, 328 | 81.76909494400024, 329 | 76.26733088493347 330 | ] 331 | } 332 | }, 333 | "validation_path": "data/en/amt/bench/bench_may/validate_AddToPlaylist.json", 334 | "training_path": "data/en/amt/bench/bench_may/train_AddToPlaylist.json", 335 | "training_name": "AddToPlaylist", 336 | "model_name": "Wit", 337 | "training_sizes": [ 338 | 10, 339 | 20, 340 | 50, 341 | 70 342 | ] 343 | } -------------------------------------------------------------------------------- /data/raw_json_data/AddToPlaylist/api.ai_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "AddToPlaylist": { 4 | "recall": { 5 | "10": [ 6 | 1.0, 7 | 1.0, 8 | 0.98 9 | ], 10 | "20": [ 11 | 1.0, 12 | 1.0, 13 | 1.0 14 | ], 15 | "50": [ 16 | 1.0, 17 | 1.0, 18 | 1.0 19 | ], 20 | "70": [ 21 | 1.0, 22 | 1.0, 23 | 1.0 24 | ] 25 | }, 26 | "slots": { 27 | "playlist": { 28 | "recall": { 29 | "10": [ 30 | 0.85, 31 | 0.6, 32 | 0.8 33 | ], 34 | "20": [ 35 | 0.66, 36 | 0.9, 37 | 0.87 38 | ], 39 | "50": [ 40 | 0.74, 41 | 0.95, 42 | 0.83 43 | ], 44 | "70": [ 45 | 0.76, 46 | 0.94, 47 | 0.79 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | 1.0, 54 | 1.0, 55 | 1.0 56 | ], 57 | "20": [ 58 | 1.0, 59 | 1.0, 60 | 1.0 61 | ], 62 | "50": [ 63 | 1.0, 64 | 1.0, 65 | 1.0 66 | ], 67 | "70": [ 68 | 1.0, 69 | 1.0, 70 | 1.0 71 | ] 72 | } 73 | }, 74 | "music_item": { 75 | "recall": { 76 | "10": [ 77 | 0.45454545454545453, 78 | 0.8545454545454545, 79 | 0.4727272727272727 80 | ], 81 | "20": [ 82 | 0.7818181818181819, 83 | 0.8181818181818182, 84 | 0.8545454545454545 85 | ], 86 | "50": [ 87 | 0.8, 88 | 0.8727272727272727, 89 | 0.8181818181818182 90 | ], 91 | "70": [ 92 | 0.7090909090909091, 93 | 0.7090909090909091, 94 | 0.8909090909090909 95 | ] 96 | }, 97 | "rate": 46, 98 | "precision": { 99 | "10": [ 100 | 0.8333333333333334, 101 | 0.8703703703703703, 102 | 0.8387096774193549 103 | ], 104 | "20": [ 105 | 0.8958333333333334, 106 | 0.8653846153846154, 107 | 0.8867924528301887 108 | ], 109 | "50": [ 110 | 0.9565217391304348, 111 | 0.8888888888888888, 112 | 0.9 113 | ], 114 | "70": [ 115 | 0.9069767441860465, 116 | 0.8666666666666667, 117 | 0.875 118 | ] 119 | } 120 | }, 121 | "playlist_owner": { 122 | "recall": { 123 | "10": [ 124 | 0.42592592592592593, 125 | 0.7592592592592593, 126 | 0.5925925925925926 127 | ], 128 | "20": [ 129 | 0.7037037037037037, 130 | 0.9259259259259259, 131 | 0.7222222222222222 132 | ], 133 | "50": [ 134 | 0.8333333333333334, 135 | 0.9074074074074074, 136 | 0.8148148148148148 137 | ], 138 | "70": [ 139 | 0.8333333333333334, 140 | 0.9629629629629629, 141 | 0.7777777777777778 142 | ] 143 | }, 144 | "rate": 64, 145 | "precision": { 146 | "10": [ 147 | 0.7666666666666667, 148 | 0.6507936507936508, 149 | 0.8648648648648649 150 | ], 151 | "20": [ 152 | 0.7037037037037037, 153 | 0.8771929824561403, 154 | 0.8478260869565217 155 | ], 156 | "50": [ 157 | 0.7894736842105263, 158 | 1.0, 159 | 0.9166666666666666 160 | ], 161 | "70": [ 162 | 0.7894736842105263, 163 | 1.0, 164 | 0.8571428571428571 165 | ] 166 | } 167 | }, 168 | "entity_name": { 169 | "recall": { 170 | "10": [ 171 | 0.8333333333333334, 172 | 0.6111111111111112, 173 | 0.8333333333333334 174 | ], 175 | "20": [ 176 | 0.6666666666666666, 177 | 0.4444444444444444, 178 | 0.5 179 | ], 180 | "50": [ 181 | 0.6111111111111112, 182 | 0.4444444444444444, 183 | 0.5555555555555556 184 | ], 185 | "70": [ 186 | 0.2777777777777778, 187 | 0.3888888888888889, 188 | 0.4444444444444444 189 | ] 190 | }, 191 | "rate": 29, 192 | "precision": { 193 | "10": [ 194 | 0.4411764705882353, 195 | 0.4230769230769231, 196 | 0.45454545454545453 197 | ], 198 | "20": [ 199 | 0.46153846153846156, 200 | 0.4, 201 | 0.75 202 | ], 203 | "50": [ 204 | 0.6111111111111112, 205 | 0.8, 206 | 0.47619047619047616 207 | ], 208 | "70": [ 209 | 0.3125, 210 | 0.3684210526315789, 211 | 0.6666666666666666 212 | ] 213 | } 214 | }, 215 | "artist": { 216 | "recall": { 217 | "10": [ 218 | 0.5217391304347826, 219 | 0.5652173913043478, 220 | 0.6956521739130435 221 | ], 222 | "20": [ 223 | 0.717391304347826, 224 | 0.717391304347826, 225 | 0.8478260869565217 226 | ], 227 | "50": [ 228 | 0.8478260869565217, 229 | 0.9347826086956522, 230 | 0.8260869565217391 231 | ], 232 | "70": [ 233 | 0.8913043478260869, 234 | 0.8913043478260869, 235 | 0.8260869565217391 236 | ] 237 | }, 238 | "rate": 38, 239 | "precision": { 240 | "10": [ 241 | 0.6153846153846154, 242 | 0.8125, 243 | 0.6666666666666666 244 | ], 245 | "20": [ 246 | 0.8048780487804879, 247 | 0.8048780487804879, 248 | 0.8666666666666667 249 | ], 250 | "50": [ 251 | 0.8125, 252 | 0.7962962962962963, 253 | 0.8444444444444444 254 | ], 255 | "70": [ 256 | 0.803921568627451, 257 | 0.7884615384615384, 258 | 0.8085106382978723 259 | ] 260 | } 261 | } 262 | }, 263 | "precision": { 264 | "10": [ 265 | 1.0, 266 | 1.0, 267 | 1.0 268 | ], 269 | "20": [ 270 | 1.0, 271 | 1.0, 272 | 1.0 273 | ], 274 | "50": [ 275 | 1.0, 276 | 1.0, 277 | 1.0 278 | ], 279 | "70": [ 280 | 1.0, 281 | 1.0, 282 | 1.0 283 | ] 284 | } 285 | } 286 | }, 287 | "execution_time": { 288 | "pred": { 289 | "10": [ 290 | 0.24699048995971679, 291 | 0.2243523406982422, 292 | 0.2275744390487671 293 | ], 294 | "20": [ 295 | 0.23958879947662354, 296 | 0.2225857400894165, 297 | 0.21620534896850585 298 | ], 299 | "50": [ 300 | 0.22865473985671997, 301 | 0.22550529956817628, 302 | 0.23173770904541016 303 | ], 304 | "70": [ 305 | 0.22947643995285033, 306 | 0.22451690912246705, 307 | 0.23010250091552734 308 | ] 309 | }, 310 | "fit": { 311 | "10": [ 312 | 92.10977697372437, 313 | 88.70381212234497, 314 | 96.56319212913513 315 | ], 316 | "20": [ 317 | 167.51917386054993, 318 | 81.12561202049255, 319 | 91.1316430568695 320 | ], 321 | "50": [ 322 | 102.45939803123474, 323 | 82.88040804862976, 324 | 168.1596508026123 325 | ], 326 | "70": [ 327 | 91.65270113945007, 328 | 86.24956202507019, 329 | 86.16888689994812 330 | ] 331 | } 332 | }, 333 | "validation_path": "data/en/amt/bench/bench_may/validate_AddToPlaylist.json", 334 | "training_path": "data/en/amt/bench/bench_may/train_AddToPlaylist.json", 335 | "training_name": "AddToPlaylist", 336 | "model_name": "api.ai", 337 | "training_sizes": [ 338 | 10, 339 | 20, 340 | 50, 341 | 70 342 | ] 343 | } -------------------------------------------------------------------------------- /data/raw_json_data/BookRestaurant/Alexa_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "BookRestaurant": { 4 | "recall": { 5 | "70": [ 6 | 0.77, 7 | 0.83, 8 | 0.7 9 | ] 10 | }, 11 | "slots": { 12 | "sort": { 13 | "recall": { 14 | "70": [ 15 | 0.8888888888888888, 16 | 0.7777777777777778, 17 | 0.7777777777777778 18 | ] 19 | }, 20 | "rate": 35, 21 | "precision": { 22 | "70": [ 23 | 1.0, 24 | 1.0, 25 | 1.0 26 | ] 27 | } 28 | }, 29 | "city": { 30 | "recall": { 31 | "70": [ 32 | 0.30303030303030304, 33 | 0.09090909090909091, 34 | 0.30303030303030304 35 | ] 36 | }, 37 | "rate": 74, 38 | "precision": { 39 | "70": [ 40 | 0.5263157894736842, 41 | 0.25, 42 | 0.5 43 | ] 44 | } 45 | }, 46 | "spatial_relation": { 47 | "recall": { 48 | "70": [ 49 | 0.21052631578947367, 50 | 0.21052631578947367, 51 | 0.21052631578947367 52 | ] 53 | }, 54 | "rate": 44, 55 | "precision": { 56 | "70": [ 57 | 0.6666666666666666, 58 | 0.6666666666666666, 59 | 0.5714285714285714 60 | ] 61 | } 62 | }, 63 | "facility": { 64 | "recall": { 65 | "70": [ 66 | 0.2857142857142857, 67 | 0.5714285714285714, 68 | 0.2857142857142857 69 | ] 70 | }, 71 | "rate": 15, 72 | "precision": { 73 | "70": [ 74 | 1.0, 75 | 1.0, 76 | 1.0 77 | ] 78 | } 79 | }, 80 | "country": { 81 | "recall": { 82 | "70": [ 83 | 0.45, 84 | 0.5, 85 | 0.3 86 | ] 87 | }, 88 | "rate": 55, 89 | "precision": { 90 | "70": [ 91 | 0.391304347826087, 92 | 0.38461538461538464, 93 | 0.42857142857142855 94 | ] 95 | } 96 | }, 97 | "party_size_description": { 98 | "recall": { 99 | "70": [ 100 | 0.38461538461538464, 101 | 0.38461538461538464, 102 | 0.46153846153846156 103 | ] 104 | }, 105 | "rate": 56, 106 | "precision": { 107 | "70": [ 108 | 0.35714285714285715, 109 | 0.5, 110 | 0.5 111 | ] 112 | } 113 | }, 114 | "restaurant_type": { 115 | "recall": { 116 | "70": [ 117 | 0.532258064516129, 118 | 0.6612903225806451, 119 | 0.5645161290322581 120 | ] 121 | }, 122 | "rate": 183, 123 | "precision": { 124 | "70": [ 125 | 0.7674418604651163, 126 | 0.82, 127 | 0.813953488372093 128 | ] 129 | } 130 | }, 131 | "cuisine": { 132 | "recall": { 133 | "70": [ 134 | 0.0, 135 | 0.0, 136 | 0.0 137 | ] 138 | }, 139 | "rate": 26, 140 | "precision": { 141 | "70": [ 142 | NaN, 143 | 0.0, 144 | NaN 145 | ] 146 | } 147 | }, 148 | "state": { 149 | "recall": { 150 | "70": [ 151 | 0.68, 152 | 0.48, 153 | 0.56 154 | ] 155 | }, 156 | "rate": 84, 157 | "precision": { 158 | "70": [ 159 | 0.6538461538461539, 160 | 0.5, 161 | 0.56 162 | ] 163 | } 164 | }, 165 | "restaurant_name": { 166 | "recall": { 167 | "70": [ 168 | 0.4, 169 | 0.5, 170 | 0.4 171 | ] 172 | }, 173 | "rate": 69, 174 | "precision": { 175 | "70": [ 176 | 0.6666666666666666, 177 | 0.7142857142857143, 178 | 0.7272727272727273 179 | ] 180 | } 181 | }, 182 | "poi": { 183 | "recall": { 184 | "70": [ 185 | 0.0, 186 | 0.0, 187 | 0.0 188 | ] 189 | }, 190 | "rate": 25, 191 | "precision": { 192 | "70": [ 193 | NaN, 194 | NaN, 195 | NaN 196 | ] 197 | } 198 | }, 199 | "party_size_number": { 200 | "recall": { 201 | "70": [ 202 | 0.5263157894736842, 203 | 0.5789473684210527, 204 | 0.5087719298245614 205 | ] 206 | }, 207 | "rate": 135, 208 | "precision": { 209 | "70": [ 210 | 0.8571428571428571, 211 | 0.825, 212 | 0.8529411764705882 213 | ] 214 | } 215 | }, 216 | "served_dish": { 217 | "recall": { 218 | "70": [ 219 | 0.2, 220 | 0.2, 221 | 0.0 222 | ] 223 | }, 224 | "rate": 22, 225 | "precision": { 226 | "70": [ 227 | 0.3333333333333333, 228 | 0.25, 229 | 0.0 230 | ] 231 | } 232 | }, 233 | "snips/datetime": { 234 | "recall": { 235 | "70": [ 236 | 0.35294117647058826, 237 | 0.47058823529411764, 238 | 0.29411764705882354 239 | ] 240 | }, 241 | "rate": 100, 242 | "precision": { 243 | "70": [ 244 | 0.631578947368421, 245 | 0.7619047619047619, 246 | 0.8333333333333334 247 | ] 248 | } 249 | } 250 | }, 251 | "precision": { 252 | "70": [ 253 | 1.0, 254 | 1.0, 255 | 1.0 256 | ] 257 | } 258 | } 259 | }, 260 | "execution_time": { 261 | "pred": { 262 | "70": [ 263 | 0.030037548542022705, 264 | 0.030049262046813966, 265 | 0.030028879642486572 266 | ] 267 | }, 268 | "fit": { 269 | "70": [ 270 | 3.002169132232666, 271 | 3.0046050548553467, 272 | 3.0011110305786133 273 | ] 274 | } 275 | }, 276 | "validation_path": "../alexa-bench/alexa_bench/resources/predictions_relabeled/", 277 | "training_path": "../alexa-bench/alexa_bench/resources/", 278 | "training_name": "BookRestaurant", 279 | "model_name": "Alexa", 280 | "training_sizes": [ 281 | 70 282 | ] 283 | } -------------------------------------------------------------------------------- /data/raw_json_data/BookRestaurant/Snips_metrics_full.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "BookRestaurant": { 4 | "recall": { 5 | "1973": [ 6 | 1.0 7 | ] 8 | }, 9 | "slots": { 10 | "sort": { 11 | "recall": { 12 | "1973": [ 13 | 0.8888888888888888 14 | ] 15 | }, 16 | "rate": 10, 17 | "precision": { 18 | "1973": [ 19 | 0.8888888888888888 20 | ] 21 | } 22 | }, 23 | "city": { 24 | "recall": { 25 | "1973": [ 26 | 0.9393939393939394 27 | ] 28 | }, 29 | "rate": 26, 30 | "precision": { 31 | "1973": [ 32 | 0.8857142857142857 33 | ] 34 | } 35 | }, 36 | "spatial_relation": { 37 | "recall": { 38 | "1973": [ 39 | 0.9473684210526315 40 | ] 41 | }, 42 | "rate": 16, 43 | "precision": { 44 | "1973": [ 45 | 1.0 46 | ] 47 | } 48 | }, 49 | "facility": { 50 | "recall": { 51 | "1973": [ 52 | 1.0 53 | ] 54 | }, 55 | "rate": 8, 56 | "precision": { 57 | "1973": [ 58 | 1.0 59 | ] 60 | } 61 | }, 62 | "country": { 63 | "recall": { 64 | "1973": [ 65 | 0.85 66 | ] 67 | }, 68 | "rate": 18, 69 | "precision": { 70 | "1973": [ 71 | 1.0 72 | ] 73 | } 74 | }, 75 | "party_size_description": { 76 | "recall": { 77 | "1973": [ 78 | 1.0 79 | ] 80 | }, 81 | "rate": 16, 82 | "precision": { 83 | "1973": [ 84 | 1.0 85 | ] 86 | } 87 | }, 88 | "restaurant_type": { 89 | "recall": { 90 | "1973": [ 91 | 0.9516129032258065 92 | ] 93 | }, 94 | "rate": 68, 95 | "precision": { 96 | "1973": [ 97 | 0.9365079365079365 98 | ] 99 | } 100 | }, 101 | "cuisine": { 102 | "recall": { 103 | "1973": [ 104 | 0.8181818181818182 105 | ] 106 | }, 107 | "rate": 10, 108 | "precision": { 109 | "1973": [ 110 | 0.8181818181818182 111 | ] 112 | } 113 | }, 114 | "state": { 115 | "recall": { 116 | "1973": [ 117 | 1.0 118 | ] 119 | }, 120 | "rate": 26, 121 | "precision": { 122 | "1973": [ 123 | 0.9615384615384616 124 | ] 125 | } 126 | }, 127 | "restaurant_name": { 128 | "recall": { 129 | "1973": [ 130 | 0.95 131 | ] 132 | }, 133 | "rate": 17, 134 | "precision": { 135 | "1973": [ 136 | 0.95 137 | ] 138 | } 139 | }, 140 | "poi": { 141 | "recall": { 142 | "1973": [ 143 | 0.6666666666666666 144 | ] 145 | }, 146 | "rate": 7, 147 | "precision": { 148 | "1973": [ 149 | 0.8 150 | ] 151 | } 152 | }, 153 | "party_size_number": { 154 | "recall": { 155 | "1973": [ 156 | 1.0 157 | ] 158 | }, 159 | "rate": 52, 160 | "precision": { 161 | "1973": [ 162 | 1.0 163 | ] 164 | } 165 | }, 166 | "served_dish": { 167 | "recall": { 168 | "1973": [ 169 | 0.6 170 | ] 171 | }, 172 | "rate": 13, 173 | "precision": { 174 | "1973": [ 175 | 0.42857142857142855 176 | ] 177 | } 178 | }, 179 | "snips/datetime": { 180 | "recall": { 181 | "1973": [ 182 | 1.0 183 | ] 184 | }, 185 | "rate": 34, 186 | "precision": { 187 | "1973": [ 188 | 1.0 189 | ] 190 | } 191 | } 192 | }, 193 | "precision": { 194 | "1973": [ 195 | 1.0 196 | ] 197 | } 198 | } 199 | }, 200 | "execution_time": { 201 | "pred": { 202 | "1973": [ 203 | 0.07689863920211792 204 | ] 205 | }, 206 | "fit": { 207 | "1973": [ 208 | 146.56977605819702 209 | ] 210 | } 211 | }, 212 | "validation_path": "data/en/amt/bench/heavy/2000/snips/validate_BookRestaurant.json", 213 | "training_path": "data/en/amt/bench/heavy/2000/snips/train_BookRestaurant.json", 214 | "training_name": "BookRestaurant", 215 | "model_name": "Snips", 216 | "training_sizes": [ 217 | 1973 218 | ] 219 | } -------------------------------------------------------------------------------- /data/raw_json_data/GetWeather/Alexa_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "GetWeather": { 4 | "recall": { 5 | "70": [ 6 | 0.77, 7 | 0.77, 8 | 0.84 9 | ] 10 | }, 11 | "slots": { 12 | "city": { 13 | "recall": { 14 | "70": [ 15 | 0.4473684210526316, 16 | 0.47368421052631576, 17 | 0.5 18 | ] 19 | }, 20 | "rate": 129, 21 | "precision": { 22 | "70": [ 23 | 0.6296296296296297, 24 | 0.5454545454545454, 25 | 0.5588235294117647 26 | ] 27 | } 28 | }, 29 | "spatial_relation": { 30 | "recall": { 31 | "70": [ 32 | 0.0, 33 | 0.0, 34 | 0.14285714285714285 35 | ] 36 | }, 37 | "rate": 30, 38 | "precision": { 39 | "70": [ 40 | NaN, 41 | NaN, 42 | 0.5 43 | ] 44 | } 45 | }, 46 | "condition_description": { 47 | "recall": { 48 | "70": [ 49 | 0.2727272727272727, 50 | 0.45454545454545453, 51 | 0.5454545454545454 52 | ] 53 | }, 54 | "rate": 63, 55 | "precision": { 56 | "70": [ 57 | 1.0, 58 | 1.0, 59 | 0.8571428571428571 60 | ] 61 | } 62 | }, 63 | "condition_temperature": { 64 | "recall": { 65 | "70": [ 66 | 0.8095238095238095, 67 | 0.6666666666666666, 68 | 0.6190476190476191 69 | ] 70 | }, 71 | "rate": 86, 72 | "precision": { 73 | "70": [ 74 | 0.9444444444444444, 75 | 0.875, 76 | 1.0 77 | ] 78 | } 79 | }, 80 | "state": { 81 | "recall": { 82 | "70": [ 83 | 0.6153846153846154, 84 | 0.6538461538461539, 85 | 0.6538461538461539 86 | ] 87 | }, 88 | "rate": 80, 89 | "precision": { 90 | "70": [ 91 | 0.6956521739130435, 92 | 0.68, 93 | 0.5666666666666667 94 | ] 95 | } 96 | }, 97 | "current_location": { 98 | "recall": { 99 | "70": [ 100 | 0.7647058823529411, 101 | 0.8235294117647058, 102 | 0.7058823529411765 103 | ] 104 | }, 105 | "rate": 38, 106 | "precision": { 107 | "70": [ 108 | 1.0, 109 | 1.0, 110 | 0.9230769230769231 111 | ] 112 | } 113 | }, 114 | "country": { 115 | "recall": { 116 | "70": [ 117 | 0.25, 118 | 0.25, 119 | 0.25 120 | ] 121 | }, 122 | "rate": 73, 123 | "precision": { 124 | "70": [ 125 | 0.42857142857142855, 126 | 0.5, 127 | 0.4 128 | ] 129 | } 130 | }, 131 | "geographic_poi": { 132 | "recall": { 133 | "70": [ 134 | 0.3125, 135 | 0.25, 136 | 0.5 137 | ] 138 | }, 139 | "rate": 38, 140 | "precision": { 141 | "70": [ 142 | 0.7142857142857143, 143 | 1.0, 144 | 0.8888888888888888 145 | ] 146 | } 147 | }, 148 | "snips/datetime": { 149 | "recall": { 150 | "70": [ 151 | 0.515625, 152 | 0.53125, 153 | 0.6875 154 | ] 155 | }, 156 | "rate": 144, 157 | "precision": { 158 | "70": [ 159 | 0.9428571428571428, 160 | 0.9714285714285714, 161 | 1.0 162 | ] 163 | } 164 | } 165 | }, 166 | "precision": { 167 | "70": [ 168 | 1.0, 169 | 1.0, 170 | 1.0 171 | ] 172 | } 173 | } 174 | }, 175 | "execution_time": { 176 | "pred": { 177 | "70": [ 178 | 0.030041329860687256, 179 | 0.030047550201416015, 180 | 0.030009379386901857 181 | ] 182 | }, 183 | "fit": { 184 | "70": [ 185 | 3.0021090507507324, 186 | 3.0019030570983887, 187 | 3.0013070106506348 188 | ] 189 | } 190 | }, 191 | "validation_path": "../alexa-bench/alexa_bench/resources/predictions_relabeled/", 192 | "training_path": "../alexa-bench/alexa_bench/resources/", 193 | "training_name": "GetWeather", 194 | "model_name": "Alexa", 195 | "training_sizes": [ 196 | 70 197 | ] 198 | } -------------------------------------------------------------------------------- /data/raw_json_data/GetWeather/Snips_metrics_full.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "GetWeather": { 4 | "recall": { 5 | "2000": [ 6 | 0.98 7 | ] 8 | }, 9 | "slots": { 10 | "city": { 11 | "recall": { 12 | "2000": [ 13 | 0.9736842105263158 14 | ] 15 | }, 16 | "rate": 42, 17 | "precision": { 18 | "2000": [ 19 | 0.9487179487179487 20 | ] 21 | } 22 | }, 23 | "spatial_relation": { 24 | "recall": { 25 | "2000": [ 26 | 0.8571428571428571 27 | ] 28 | }, 29 | "rate": 10, 30 | "precision": { 31 | "2000": [ 32 | 1.0 33 | ] 34 | } 35 | }, 36 | "condition_description": { 37 | "recall": { 38 | "2000": [ 39 | 0.9545454545454546 40 | ] 41 | }, 42 | "rate": 22, 43 | "precision": { 44 | "2000": [ 45 | 1.0 46 | ] 47 | } 48 | }, 49 | "condition_temperature": { 50 | "recall": { 51 | "2000": [ 52 | 1.0 53 | ] 54 | }, 55 | "rate": 23, 56 | "precision": { 57 | "2000": [ 58 | 1.0 59 | ] 60 | } 61 | }, 62 | "state": { 63 | "recall": { 64 | "2000": [ 65 | 0.9615384615384616 66 | ] 67 | }, 68 | "rate": 24, 69 | "precision": { 70 | "2000": [ 71 | 0.9615384615384616 72 | ] 73 | } 74 | }, 75 | "current_location": { 76 | "recall": { 77 | "2000": [ 78 | 1.0 79 | ] 80 | }, 81 | "rate": 13, 82 | "precision": { 83 | "2000": [ 84 | 1.0 85 | ] 86 | } 87 | }, 88 | "country": { 89 | "recall": { 90 | "2000": [ 91 | 0.9583333333333334 92 | ] 93 | }, 94 | "rate": 25, 95 | "precision": { 96 | "2000": [ 97 | 0.92 98 | ] 99 | } 100 | }, 101 | "geographic_poi": { 102 | "recall": { 103 | "2000": [ 104 | 0.875 105 | ] 106 | }, 107 | "rate": 14, 108 | "precision": { 109 | "2000": [ 110 | 0.875 111 | ] 112 | } 113 | }, 114 | "snips/datetime": { 115 | "recall": { 116 | "2000": [ 117 | 0.90625 118 | ] 119 | }, 120 | "rate": 52, 121 | "precision": { 122 | "2000": [ 123 | 0.90625 124 | ] 125 | } 126 | } 127 | }, 128 | "precision": { 129 | "2000": [ 130 | 1.0 131 | ] 132 | } 133 | } 134 | }, 135 | "execution_time": { 136 | "pred": { 137 | "2000": [ 138 | 0.04777778148651123 139 | ] 140 | }, 141 | "fit": { 142 | "2000": [ 143 | 65.39982104301453 144 | ] 145 | } 146 | }, 147 | "validation_path": "data/en/amt/bench/heavy/2000/snips/validate_GetWeather.json", 148 | "training_path": "data/en/amt/bench/heavy/2000/snips/train_GetWeather.json", 149 | "training_name": "GetWeather", 150 | "model_name": "Snips", 151 | "training_sizes": [ 152 | 2000 153 | ] 154 | } -------------------------------------------------------------------------------- /data/raw_json_data/RateBook/Alexa_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "RateBook": { 4 | "recall": { 5 | "70": [ 6 | 1.0, 7 | 0.99, 8 | 1.0 9 | ] 10 | }, 11 | "slots": { 12 | "object_part_of_series_type": { 13 | "recall": { 14 | "70": [ 15 | 0.9333333333333333, 16 | 0.8666666666666667, 17 | 1.0 18 | ] 19 | }, 20 | "rate": 42, 21 | "precision": { 22 | "70": [ 23 | 1.0, 24 | 1.0, 25 | 1.0 26 | ] 27 | } 28 | }, 29 | "rating_value": { 30 | "recall": { 31 | "70": [ 32 | 0.52, 33 | 0.53, 34 | 0.56 35 | ] 36 | }, 37 | "rate": 300, 38 | "precision": { 39 | "70": [ 40 | 0.8, 41 | 0.803030303030303, 42 | 0.8115942028985508 43 | ] 44 | } 45 | }, 46 | "object_type": { 47 | "recall": { 48 | "70": [ 49 | 0.95, 50 | 0.975, 51 | 0.9 52 | ] 53 | }, 54 | "rate": 133, 55 | "precision": { 56 | "70": [ 57 | 0.9743589743589743, 58 | 1.0, 59 | 1.0 60 | ] 61 | } 62 | }, 63 | "object_select": { 64 | "recall": { 65 | "70": [ 66 | 0.9795918367346939, 67 | 0.9795918367346939, 68 | 0.9183673469387755 69 | ] 70 | }, 71 | "rate": 158, 72 | "precision": { 73 | "70": [ 74 | 1.0, 75 | 1.0, 76 | 1.0 77 | ] 78 | } 79 | }, 80 | "object_name": { 81 | "recall": { 82 | "70": [ 83 | 0.35294117647058826, 84 | 0.39215686274509803, 85 | 0.37254901960784315 86 | ] 87 | }, 88 | "rate": 142, 89 | "precision": { 90 | "70": [ 91 | 0.33962264150943394, 92 | 0.38461538461538464, 93 | 0.3392857142857143 94 | ] 95 | } 96 | }, 97 | "rating_unit": { 98 | "recall": { 99 | "70": [ 100 | 0.7704918032786885, 101 | 0.7704918032786885, 102 | 0.8360655737704918 103 | ] 104 | }, 105 | "rate": 198, 106 | "precision": { 107 | "70": [ 108 | 1.0, 109 | 1.0, 110 | 1.0 111 | ] 112 | } 113 | }, 114 | "best_rating": { 115 | "recall": { 116 | "70": [ 117 | 0.6078431372549019, 118 | 0.6862745098039216, 119 | 0.7058823529411765 120 | ] 121 | }, 122 | "rate": 173, 123 | "precision": { 124 | "70": [ 125 | 0.9393939393939394, 126 | 0.9459459459459459, 127 | 1.0 128 | ] 129 | } 130 | } 131 | }, 132 | "precision": { 133 | "70": [ 134 | 1.0, 135 | 1.0, 136 | 1.0 137 | ] 138 | } 139 | } 140 | }, 141 | "execution_time": { 142 | "pred": { 143 | "70": [ 144 | 0.030000669956207274, 145 | 0.03003091096878052, 146 | 0.030024120807647704 147 | ] 148 | }, 149 | "fit": { 150 | "70": [ 151 | 3.000063896179199, 152 | 3.0017800331115723, 153 | 3.00360107421875 154 | ] 155 | } 156 | }, 157 | "validation_path": "../alexa-bench/alexa_bench/resources/predictions_relabeled/", 158 | "training_path": "../alexa-bench/alexa_bench/resources/", 159 | "training_name": "RateBook", 160 | "model_name": "Alexa", 161 | "training_sizes": [ 162 | 70 163 | ] 164 | } -------------------------------------------------------------------------------- /data/raw_json_data/RateBook/Snips_metrics_full.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "RateBook": { 4 | "recall": { 5 | "1956": [ 6 | 1.0 7 | ] 8 | }, 9 | "slots": { 10 | "object_part_of_series_type": { 11 | "recall": { 12 | "1956": [ 13 | 0.8666666666666667 14 | ] 15 | }, 16 | "rate": 15, 17 | "precision": { 18 | "1956": [ 19 | 1.0 20 | ] 21 | } 22 | }, 23 | "rating_value": { 24 | "recall": { 25 | "1956": [ 26 | 1.0 27 | ] 28 | }, 29 | "rate": 100, 30 | "precision": { 31 | "1956": [ 32 | 0.9900990099009901 33 | ] 34 | } 35 | }, 36 | "object_type": { 37 | "recall": { 38 | "1956": [ 39 | 1.0 40 | ] 41 | }, 42 | "rate": 48, 43 | "precision": { 44 | "1956": [ 45 | 1.0 46 | ] 47 | } 48 | }, 49 | "object_select": { 50 | "recall": { 51 | "1956": [ 52 | 0.9183673469387755 53 | ] 54 | }, 55 | "rate": 50, 56 | "precision": { 57 | "1956": [ 58 | 0.9782608695652174 59 | ] 60 | } 61 | }, 62 | "object_name": { 63 | "recall": { 64 | "1956": [ 65 | 0.9411764705882353 66 | ] 67 | }, 68 | "rate": 50, 69 | "precision": { 70 | "1956": [ 71 | 0.96 72 | ] 73 | } 74 | }, 75 | "rating_unit": { 76 | "recall": { 77 | "1956": [ 78 | 1.0 79 | ] 80 | }, 81 | "rate": 57, 82 | "precision": { 83 | "1956": [ 84 | 0.9838709677419355 85 | ] 86 | } 87 | }, 88 | "best_rating": { 89 | "recall": { 90 | "1956": [ 91 | 1.0 92 | ] 93 | }, 94 | "rate": 53, 95 | "precision": { 96 | "1956": [ 97 | 1.0 98 | ] 99 | } 100 | } 101 | }, 102 | "precision": { 103 | "1956": [ 104 | 1.0 105 | ] 106 | } 107 | } 108 | }, 109 | "execution_time": { 110 | "pred": { 111 | "1956": [ 112 | 0.021336560249328614 113 | ] 114 | }, 115 | "fit": { 116 | "1956": [ 117 | 54.609976053237915 118 | ] 119 | } 120 | }, 121 | "validation_path": "data/en/amt/bench/heavy/2000/snips/validate_RateBook.json", 122 | "training_path": "data/en/amt/bench/heavy/2000/snips/train_RateBook.json", 123 | "training_name": "RateBook", 124 | "model_name": "Snips", 125 | "training_sizes": [ 126 | 1956 127 | ] 128 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchCreativeWork/Alexa_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchCreativeWork": { 4 | "recall": { 5 | "70": [ 6 | 1.0, 7 | 0.98, 8 | 1.0 9 | ] 10 | }, 11 | "slots": { 12 | "object_name": { 13 | "recall": { 14 | "70": [ 15 | 0.35, 16 | 0.44, 17 | 0.34 18 | ] 19 | }, 20 | "rate": 300, 21 | "precision": { 22 | "70": [ 23 | 0.4375, 24 | 0.4835164835164835, 25 | 0.4358974358974359 26 | ] 27 | } 28 | }, 29 | "object_type": { 30 | "recall": { 31 | "70": [ 32 | 0.2876712328767123, 33 | 0.2465753424657534, 34 | 0.273972602739726 35 | ] 36 | }, 37 | "rate": 233, 38 | "precision": { 39 | "70": [ 40 | 0.4883720930232558, 41 | 0.6206896551724138, 42 | 0.4878048780487805 43 | ] 44 | } 45 | } 46 | }, 47 | "precision": { 48 | "70": [ 49 | 1.0, 50 | 1.0, 51 | 1.0 52 | ] 53 | } 54 | } 55 | }, 56 | "execution_time": { 57 | "pred": { 58 | "70": [ 59 | 0.030044379234313964, 60 | 0.03003464937210083, 61 | 0.03000720977783203 62 | ] 63 | }, 64 | "fit": { 65 | "70": [ 66 | 3.004272937774658, 67 | 3.005048990249634, 68 | 3.004242181777954 69 | ] 70 | } 71 | }, 72 | "validation_path": "../alexa-bench/alexa_bench/resources/predictions_relabeled/", 73 | "training_path": "../alexa-bench/alexa_bench/resources/", 74 | "training_name": "SearchCreativeWork", 75 | "model_name": "Alexa", 76 | "training_sizes": [ 77 | 70 78 | ] 79 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchCreativeWork/Luis_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchCreativeWork": { 4 | "recall": { 5 | "10": [ 6 | 1.0, 7 | 1.0, 8 | 1.0 9 | ], 10 | "20": [ 11 | 1.0, 12 | 1.0, 13 | 1.0 14 | ], 15 | "50": [ 16 | 1.0, 17 | 1.0, 18 | 1.0 19 | ], 20 | "70": [ 21 | 1.0, 22 | 1.0, 23 | 1.0 24 | ] 25 | }, 26 | "slots": { 27 | "object_name": { 28 | "recall": { 29 | "10": [ 30 | 0.19, 31 | 0.18, 32 | 0.21 33 | ], 34 | "20": [ 35 | 0.42, 36 | 0.48, 37 | 0.42 38 | ], 39 | "50": [ 40 | 0.65, 41 | 0.7, 42 | 0.66 43 | ], 44 | "70": [ 45 | 0.73, 46 | 0.76, 47 | 0.78 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | 1.0, 54 | 1.0, 55 | 1.0 56 | ], 57 | "20": [ 58 | 1.0, 59 | 1.0, 60 | 1.0 61 | ], 62 | "50": [ 63 | 1.0, 64 | 1.0, 65 | 1.0 66 | ], 67 | "70": [ 68 | 1.0, 69 | 1.0, 70 | 1.0 71 | ] 72 | } 73 | }, 74 | "object_type": { 75 | "recall": { 76 | "10": [ 77 | 0.0273972602739726, 78 | 0.2054794520547945, 79 | 0.136986301369863 80 | ], 81 | "20": [ 82 | 0.1917808219178082, 83 | 0.2465753424657534, 84 | 0.410958904109589 85 | ], 86 | "50": [ 87 | 0.6712328767123288, 88 | 0.5616438356164384, 89 | 0.7945205479452054 90 | ], 91 | "70": [ 92 | 0.6712328767123288, 93 | 0.6164383561643836, 94 | 0.9178082191780822 95 | ] 96 | }, 97 | "rate": 77, 98 | "precision": { 99 | "10": [ 100 | 1.0, 101 | 1.0, 102 | 1.0 103 | ], 104 | "20": [ 105 | 1.0, 106 | 1.0, 107 | 1.0 108 | ], 109 | "50": [ 110 | 0.98, 111 | 0.9534883720930233, 112 | 0.9508196721311475 113 | ], 114 | "70": [ 115 | 0.9607843137254902, 116 | 1.0, 117 | 1.0 118 | ] 119 | } 120 | } 121 | }, 122 | "precision": { 123 | "10": [ 124 | 1.0, 125 | 1.0, 126 | 1.0 127 | ], 128 | "20": [ 129 | 1.0, 130 | 1.0, 131 | 1.0 132 | ], 133 | "50": [ 134 | 1.0, 135 | 1.0, 136 | 1.0 137 | ], 138 | "70": [ 139 | 1.0, 140 | 1.0, 141 | 1.0 142 | ] 143 | } 144 | } 145 | }, 146 | "execution_time": { 147 | "pred": { 148 | "10": [ 149 | 0.4012924695014954, 150 | 0.4152055811882019, 151 | 0.4040277600288391 152 | ], 153 | "20": [ 154 | 0.4006899309158325, 155 | 0.4064324188232422, 156 | 0.40599990844726563 157 | ], 158 | "50": [ 159 | 0.4074488115310669, 160 | 0.4034702301025391, 161 | 0.40434554100036624 162 | ], 163 | "70": [ 164 | 0.4016836905479431, 165 | 0.40735544919967653, 166 | 0.4000175786018372 167 | ] 168 | }, 169 | "fit": { 170 | "10": [ 171 | 309.37376499176025, 172 | 141.23220705986023, 173 | 139.81973600387573 174 | ], 175 | "20": [ 176 | 141.27941513061523, 177 | 151.9643108844757, 178 | 151.69257998466492 179 | ], 180 | "50": [ 181 | 150.66960501670837, 182 | 177.38863897323608, 183 | 184.09078788757324 184 | ], 185 | "70": [ 186 | 177.55351209640503, 187 | 200.4946038722992, 188 | 197.9243140220642 189 | ] 190 | } 191 | }, 192 | "validation_path": "data/en/amt/bench/bench_may/validate_SearchCreativeWork.json", 193 | "training_path": "data/en/amt/bench/bench_may/train_SearchCreativeWork.json", 194 | "training_name": "SearchCreativeWork", 195 | "model_name": "Luis", 196 | "training_sizes": [ 197 | 10, 198 | 20, 199 | 50, 200 | 70 201 | ] 202 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchCreativeWork/Snips_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchCreativeWork": { 4 | "recall": { 5 | "10": [ 6 | 0.9, 7 | 1.0, 8 | 0.94 9 | ], 10 | "20": [ 11 | 0.97, 12 | 0.99, 13 | 0.92 14 | ], 15 | "50": [ 16 | 0.98, 17 | 1.0, 18 | 0.98 19 | ], 20 | "70": [ 21 | 1.0, 22 | 1.0, 23 | 0.99 24 | ] 25 | }, 26 | "slots": { 27 | "object_name": { 28 | "recall": { 29 | "10": [ 30 | 0.61, 31 | 0.6, 32 | 0.38 33 | ], 34 | "20": [ 35 | 0.69, 36 | 0.65, 37 | 0.56 38 | ], 39 | "50": [ 40 | 0.7, 41 | 0.86, 42 | 0.75 43 | ], 44 | "70": [ 45 | 0.83, 46 | 0.91, 47 | 0.83 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | 0.5922330097087378, 54 | 0.625, 55 | 0.3958333333333333 56 | ], 57 | "20": [ 58 | 0.7040816326530612, 59 | 0.6632653061224489, 60 | 0.5833333333333334 61 | ], 62 | "50": [ 63 | 0.6862745098039216, 64 | 0.8686868686868687, 65 | 0.7075471698113207 66 | ], 67 | "70": [ 68 | 0.83, 69 | 0.91, 70 | 0.8469387755102041 71 | ] 72 | } 73 | }, 74 | "object_type": { 75 | "recall": { 76 | "10": [ 77 | 0.5205479452054794, 78 | 0.7397260273972602, 79 | 0.589041095890411 80 | ], 81 | "20": [ 82 | 0.8493150684931506, 83 | 0.7397260273972602, 84 | 0.821917808219178 85 | ], 86 | "50": [ 87 | 0.9315068493150684, 88 | 0.958904109589041, 89 | 0.9863013698630136 90 | ], 91 | "70": [ 92 | 0.9863013698630136, 93 | 0.9452054794520548, 94 | 0.9452054794520548 95 | ] 96 | }, 97 | "rate": 77, 98 | "precision": { 99 | "10": [ 100 | 0.8636363636363636, 101 | 0.6506024096385542, 102 | 0.5584415584415584 103 | ], 104 | "20": [ 105 | 0.7848101265822784, 106 | 0.675, 107 | 0.631578947368421 108 | ], 109 | "50": [ 110 | 0.8, 111 | 0.8433734939759037, 112 | 0.8470588235294118 113 | ], 114 | "70": [ 115 | 0.8888888888888888, 116 | 0.9078947368421053, 117 | 0.8023255813953488 118 | ] 119 | } 120 | } 121 | }, 122 | "precision": { 123 | "10": [ 124 | 1.0, 125 | 1.0, 126 | 1.0 127 | ], 128 | "20": [ 129 | 1.0, 130 | 1.0, 131 | 1.0 132 | ], 133 | "50": [ 134 | 1.0, 135 | 1.0, 136 | 1.0 137 | ], 138 | "70": [ 139 | 1.0, 140 | 1.0, 141 | 1.0 142 | ] 143 | } 144 | } 145 | }, 146 | "execution_time": { 147 | "pred": { 148 | "10": [ 149 | 0.01473242998123169, 150 | 0.01032721996307373, 151 | 0.008402550220489502 152 | ], 153 | "20": [ 154 | 0.009213619232177735, 155 | 0.009868199825286866, 156 | 0.008027780055999755 157 | ], 158 | "50": [ 159 | 0.00843899965286255, 160 | 0.008612279891967773, 161 | 0.01267204999923706 162 | ], 163 | "70": [ 164 | 0.008748631477355957, 165 | 0.008724081516265868, 166 | 0.009338111877441406 167 | ] 168 | }, 169 | "fit": { 170 | "10": [ 171 | 2.145020008087158, 172 | 3.2602479457855225, 173 | 2.6426949501037598 174 | ], 175 | "20": [ 176 | 2.421066999435425, 177 | 2.4188501834869385, 178 | 3.267810106277466 179 | ], 180 | "50": [ 181 | 2.2642509937286377, 182 | 2.640615940093994, 183 | 2.358196973800659 184 | ], 185 | "70": [ 186 | 2.4430739879608154, 187 | 2.9639840126037598, 188 | 3.018695831298828 189 | ] 190 | } 191 | }, 192 | "validation_path": "data/en/amt/bench/bench_may/snips/validate_SearchCreativeWork.json", 193 | "training_path": "data/en/amt/bench/bench_may/snips/train_SearchCreativeWork.json", 194 | "training_name": "SearchCreativeWork", 195 | "model_name": "Snips", 196 | "training_sizes": [ 197 | 10, 198 | 20, 199 | 50, 200 | 70 201 | ] 202 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchCreativeWork/Snips_metrics_full.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchCreativeWork": { 4 | "recall": { 5 | "1954": [ 6 | 1.0 7 | ] 8 | }, 9 | "slots": { 10 | "object_name": { 11 | "recall": { 12 | "1954": [ 13 | 0.92 14 | ] 15 | }, 16 | "rate": 100, 17 | "precision": { 18 | "1954": [ 19 | 0.9292929292929293 20 | ] 21 | } 22 | }, 23 | "object_type": { 24 | "recall": { 25 | "1954": [ 26 | 0.9452054794520548 27 | ] 28 | }, 29 | "rate": 74, 30 | "precision": { 31 | "1954": [ 32 | 0.9857142857142858 33 | ] 34 | } 35 | } 36 | }, 37 | "precision": { 38 | "1954": [ 39 | 1.0 40 | ] 41 | } 42 | } 43 | }, 44 | "execution_time": { 45 | "pred": { 46 | "1954": [ 47 | 0.01446274995803833 48 | ] 49 | }, 50 | "fit": { 51 | "1954": [ 52 | 26.101171016693115 53 | ] 54 | } 55 | }, 56 | "validation_path": "data/en/amt/bench/heavy/2000/snips/validate_SearchCreativeWork.json", 57 | "training_path": "data/en/amt/bench/heavy/2000/snips/train_SearchCreativeWork.json", 58 | "training_name": "SearchCreativeWork", 59 | "model_name": "Snips", 60 | "training_sizes": [ 61 | 1954 62 | ] 63 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchCreativeWork/Wit_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchCreativeWork": { 4 | "recall": { 5 | "10": [ 6 | 0.0, 7 | 0.0, 8 | 0.0 9 | ], 10 | "20": [ 11 | 0.0, 12 | 0.0, 13 | 0.0 14 | ], 15 | "50": [ 16 | 0.0, 17 | 0.0, 18 | 0.0 19 | ], 20 | "70": [ 21 | 0.0, 22 | 0.0, 23 | 0.0 24 | ] 25 | }, 26 | "slots": { 27 | "object_name": { 28 | "recall": { 29 | "10": [ 30 | 0.0, 31 | 0.0, 32 | 0.0 33 | ], 34 | "20": [ 35 | 0.14, 36 | 0.0, 37 | 0.0 38 | ], 39 | "50": [ 40 | 0.0, 41 | 0.0, 42 | 0.45 43 | ], 44 | "70": [ 45 | 0.0, 46 | 0.59, 47 | 0.0 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | NaN, 54 | NaN, 55 | NaN 56 | ], 57 | "20": [ 58 | 0.6086956521739131, 59 | NaN, 60 | NaN 61 | ], 62 | "50": [ 63 | NaN, 64 | NaN, 65 | 0.8823529411764706 66 | ], 67 | "70": [ 68 | NaN, 69 | 0.8805970149253731, 70 | NaN 71 | ] 72 | } 73 | }, 74 | "object_type": { 75 | "recall": { 76 | "10": [ 77 | 0.3013698630136986, 78 | 0.3972602739726027, 79 | 0.410958904109589 80 | ], 81 | "20": [ 82 | 0.589041095890411, 83 | 0.4520547945205479, 84 | 0.7808219178082192 85 | ], 86 | "50": [ 87 | 0.9041095890410958, 88 | 0.958904109589041, 89 | 0.9315068493150684 90 | ], 91 | "70": [ 92 | 0.9041095890410958, 93 | 0.9178082191780822, 94 | 1.0 95 | ] 96 | }, 97 | "rate": 77, 98 | "precision": { 99 | "10": [ 100 | 0.7333333333333333, 101 | 0.7631578947368421, 102 | 0.75 103 | ], 104 | "20": [ 105 | 0.7962962962962963, 106 | 0.7857142857142857, 107 | 0.8382352941176471 108 | ], 109 | "50": [ 110 | 0.9565217391304348, 111 | 0.958904109589041, 112 | 0.918918918918919 113 | ], 114 | "70": [ 115 | 0.9565217391304348, 116 | 0.9436619718309859, 117 | 0.9605263157894737 118 | ] 119 | } 120 | } 121 | }, 122 | "precision": { 123 | "10": [ 124 | NaN, 125 | NaN, 126 | NaN 127 | ], 128 | "20": [ 129 | NaN, 130 | NaN, 131 | NaN 132 | ], 133 | "50": [ 134 | NaN, 135 | NaN, 136 | NaN 137 | ], 138 | "70": [ 139 | NaN, 140 | NaN, 141 | NaN 142 | ] 143 | } 144 | } 145 | }, 146 | "execution_time": { 147 | "pred": { 148 | "10": [ 149 | 1.2374331402778624, 150 | 1.260684108734131, 151 | 1.2524772310256957 152 | ], 153 | "20": [ 154 | 1.1966094017028808, 155 | 1.4005684399604796, 156 | 2.047036101818085 157 | ], 158 | "50": [ 159 | 1.837898361682892, 160 | 1.2359887290000915, 161 | 1.6489326190948486 162 | ], 163 | "70": [ 164 | 1.2401251602172851, 165 | 1.1942154002189636, 166 | 1.2698214507102967 167 | ] 168 | }, 169 | "fit": { 170 | "10": [ 171 | 131.85978817939758, 172 | 133.05734205245972, 173 | 128.8786211013794 174 | ], 175 | "20": [ 176 | 128.83473706245422, 177 | 131.91290998458862, 178 | 131.56610989570618 179 | ], 180 | "50": [ 181 | 132.88493990898132, 182 | 131.90618681907654, 183 | 127.38174986839294 184 | ], 185 | "70": [ 186 | 133.51782083511353, 187 | 127.47976398468018, 188 | 127.64315986633301 189 | ] 190 | } 191 | }, 192 | "validation_path": "data/en/amt/bench/bench_may/validate_SearchCreativeWork.json", 193 | "training_path": "data/en/amt/bench/bench_may/train_SearchCreativeWork.json", 194 | "training_name": "SearchCreativeWork", 195 | "model_name": "Wit", 196 | "training_sizes": [ 197 | 10, 198 | 20, 199 | 50, 200 | 70 201 | ] 202 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchCreativeWork/api.ai_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchCreativeWork": { 4 | "recall": { 5 | "10": [ 6 | 0.96, 7 | 0.98, 8 | 0.97 9 | ], 10 | "20": [ 11 | 0.97, 12 | 0.99, 13 | 0.98 14 | ], 15 | "50": [ 16 | 0.6, 17 | 0.95, 18 | 0.95 19 | ], 20 | "70": [ 21 | 0.99, 22 | 0.98, 23 | 0.97 24 | ] 25 | }, 26 | "slots": { 27 | "object_name": { 28 | "recall": { 29 | "10": [ 30 | 0.4, 31 | 0.64, 32 | 0.49 33 | ], 34 | "20": [ 35 | 0.63, 36 | 0.73, 37 | 0.67 38 | ], 39 | "50": [ 40 | 0.46, 41 | 0.62, 42 | 0.65 43 | ], 44 | "70": [ 45 | 0.82, 46 | 0.75, 47 | 0.68 48 | ] 49 | }, 50 | "rate": 100, 51 | "precision": { 52 | "10": [ 53 | 1.0, 54 | 1.0, 55 | 1.0 56 | ], 57 | "20": [ 58 | 1.0, 59 | 1.0, 60 | 1.0 61 | ], 62 | "50": [ 63 | 1.0, 64 | 1.0, 65 | 1.0 66 | ], 67 | "70": [ 68 | 1.0, 69 | 1.0, 70 | 1.0 71 | ] 72 | } 73 | }, 74 | "object_type": { 75 | "recall": { 76 | "10": [ 77 | 0.7397260273972602, 78 | 0.547945205479452, 79 | 0.684931506849315 80 | ], 81 | "20": [ 82 | 0.6712328767123288, 83 | 0.6575342465753424, 84 | 0.6438356164383562 85 | ], 86 | "50": [ 87 | 0.4794520547945205, 88 | 0.7671232876712328, 89 | 0.7808219178082192 90 | ], 91 | "70": [ 92 | 0.5616438356164384, 93 | 0.6301369863013698, 94 | 0.821917808219178 95 | ] 96 | }, 97 | "rate": 77, 98 | "precision": { 99 | "10": [ 100 | 0.7714285714285715, 101 | 0.8163265306122449, 102 | 0.78125 103 | ], 104 | "20": [ 105 | 0.7538461538461538, 106 | 0.8727272727272727, 107 | 0.7230769230769231 108 | ], 109 | "50": [ 110 | 0.8536585365853658, 111 | 0.875, 112 | 0.7916666666666666 113 | ], 114 | "70": [ 115 | 0.8541666666666666, 116 | 0.7931034482758621, 117 | 0.8450704225352113 118 | ] 119 | } 120 | } 121 | }, 122 | "precision": { 123 | "10": [ 124 | 1.0, 125 | 1.0, 126 | 1.0 127 | ], 128 | "20": [ 129 | 1.0, 130 | 1.0, 131 | 1.0 132 | ], 133 | "50": [ 134 | 1.0, 135 | 1.0, 136 | 1.0 137 | ], 138 | "70": [ 139 | 1.0, 140 | 1.0, 141 | 1.0 142 | ] 143 | } 144 | } 145 | }, 146 | "execution_time": { 147 | "pred": { 148 | "10": [ 149 | 0.40560964107513425, 150 | 0.3001484799385071, 151 | 0.26229347944259646 152 | ], 153 | "20": [ 154 | 0.23784173965454103, 155 | 0.23868534803390504, 156 | 0.24279438018798827 157 | ], 158 | "50": [ 159 | 0.3018018388748169, 160 | 0.2525062203407288, 161 | 0.2525571298599243 162 | ], 163 | "70": [ 164 | 0.2439579677581787, 165 | 0.2451993417739868, 166 | 0.2865855503082275 167 | ] 168 | }, 169 | "fit": { 170 | "10": [ 171 | 93.13007998466492, 172 | 92.57890295982361, 173 | 92.22457504272461 174 | ], 175 | "20": [ 176 | 85.13665699958801, 177 | 79.98982286453247, 178 | 85.93652105331421 179 | ], 180 | "50": [ 181 | 162.52885818481445, 182 | 163.41672706604004, 183 | 86.14306402206421 184 | ], 185 | "70": [ 186 | 167.57556295394897, 187 | 63.703505992889404, 188 | 86.7500810623169 189 | ] 190 | } 191 | }, 192 | "validation_path": "data/en/amt/bench/bench_may/validate_SearchCreativeWork.json", 193 | "training_path": "data/en/amt/bench/bench_may/train_SearchCreativeWork.json", 194 | "training_name": "SearchCreativeWork", 195 | "model_name": "api.ai", 196 | "training_sizes": [ 197 | 10, 198 | 20, 199 | 50, 200 | 70 201 | ] 202 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchScreeningEvent/Alexa_metrics.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchScreeningEvent": { 4 | "recall": { 5 | "70": [ 6 | 0.95, 7 | 0.97, 8 | 0.96 9 | ] 10 | }, 11 | "slots": { 12 | "spatial_relation": { 13 | "recall": { 14 | "70": [ 15 | 0.6571428571428571, 16 | 0.7428571428571429, 17 | 0.6857142857142857 18 | ] 19 | }, 20 | "rate": 103, 21 | "precision": { 22 | "70": [ 23 | 0.8214285714285714, 24 | 0.896551724137931, 25 | 0.75 26 | ] 27 | } 28 | }, 29 | "movie_name": { 30 | "recall": { 31 | "70": [ 32 | 0.6122448979591837, 33 | 0.6122448979591837, 34 | 0.6326530612244898 35 | ] 36 | }, 37 | "rate": 149, 38 | "precision": { 39 | "70": [ 40 | 0.5555555555555556, 41 | 0.5882352941176471, 42 | 0.6078431372549019 43 | ] 44 | } 45 | }, 46 | "object_type": { 47 | "recall": { 48 | "70": [ 49 | 0.18604651162790697, 50 | 0.27906976744186046, 51 | 0.23255813953488372 52 | ] 53 | }, 54 | "rate": 118, 55 | "precision": { 56 | "70": [ 57 | 1.0, 58 | 0.8571428571428571, 59 | 0.6666666666666666 60 | ] 61 | } 62 | }, 63 | "movie_type": { 64 | "recall": { 65 | "70": [ 66 | 1.0, 67 | 0.9583333333333334, 68 | 1.0 69 | ] 70 | }, 71 | "rate": 85, 72 | "precision": { 73 | "70": [ 74 | 1.0, 75 | 1.0, 76 | 1.0 77 | ] 78 | } 79 | }, 80 | "location_name": { 81 | "recall": { 82 | "70": [ 83 | 0.2413793103448276, 84 | 0.27586206896551724, 85 | 0.20689655172413793 86 | ] 87 | }, 88 | "rate": 92, 89 | "precision": { 90 | "70": [ 91 | 0.23333333333333334, 92 | 0.24242424242424243, 93 | 0.1935483870967742 94 | ] 95 | } 96 | }, 97 | "object_location_type": { 98 | "recall": { 99 | "70": [ 100 | 0.55, 101 | 0.5, 102 | 0.6 103 | ] 104 | }, 105 | "rate": 80, 106 | "precision": { 107 | "70": [ 108 | 0.4230769230769231, 109 | 0.47619047619047616, 110 | 0.8571428571428571 111 | ] 112 | } 113 | }, 114 | "snips/datetime": { 115 | "recall": { 116 | "70": [ 117 | 0.25, 118 | 0.25, 119 | 0.3333333333333333 120 | ] 121 | }, 122 | "rate": 39, 123 | "precision": { 124 | "70": [ 125 | 1.0, 126 | 0.75, 127 | 1.0 128 | ] 129 | } 130 | } 131 | }, 132 | "precision": { 133 | "70": [ 134 | 1.0, 135 | 1.0, 136 | 1.0 137 | ] 138 | } 139 | } 140 | }, 141 | "execution_time": { 142 | "pred": { 143 | "70": [ 144 | 0.030026400089263917, 145 | 0.030002751350402833, 146 | 0.030009000301361083 147 | ] 148 | }, 149 | "fit": { 150 | "70": [ 151 | 3.001260995864868, 152 | 3.0033509731292725, 153 | 3.001224994659424 154 | ] 155 | } 156 | }, 157 | "validation_path": "../alexa-bench/alexa_bench/resources/predictions_relabeled/", 158 | "training_path": "../alexa-bench/alexa_bench/resources/", 159 | "training_name": "SearchScreeningEvent", 160 | "model_name": "Alexa", 161 | "training_sizes": [ 162 | 70 163 | ] 164 | } -------------------------------------------------------------------------------- /data/raw_json_data/SearchScreeningEvent/Snips_metrics_full.json: -------------------------------------------------------------------------------- 1 | { 2 | "intents": { 3 | "SearchScreeningEvent": { 4 | "recall": { 5 | "1959": [ 6 | 0.96 7 | ] 8 | }, 9 | "slots": { 10 | "spatial_relation": { 11 | "recall": { 12 | "1959": [ 13 | 1.0 14 | ] 15 | }, 16 | "rate": 35, 17 | "precision": { 18 | "1959": [ 19 | 1.0 20 | ] 21 | } 22 | }, 23 | "movie_name": { 24 | "recall": { 25 | "1959": [ 26 | 0.8979591836734694 27 | ] 28 | }, 29 | "rate": 41, 30 | "precision": { 31 | "1959": [ 32 | 0.9565217391304348 33 | ] 34 | } 35 | }, 36 | "object_type": { 37 | "recall": { 38 | "1959": [ 39 | 0.9302325581395349 40 | ] 41 | }, 42 | "rate": 39, 43 | "precision": { 44 | "1959": [ 45 | 0.9523809523809523 46 | ] 47 | } 48 | }, 49 | "movie_type": { 50 | "recall": { 51 | "1959": [ 52 | 1.0 53 | ] 54 | }, 55 | "rate": 35, 56 | "precision": { 57 | "1959": [ 58 | 0.9230769230769231 59 | ] 60 | } 61 | }, 62 | "location_name": { 63 | "recall": { 64 | "1959": [ 65 | 0.9655172413793104 66 | ] 67 | }, 68 | "rate": 29, 69 | "precision": { 70 | "1959": [ 71 | 0.9655172413793104 72 | ] 73 | } 74 | }, 75 | "object_location_type": { 76 | "recall": { 77 | "1959": [ 78 | 1.0 79 | ] 80 | }, 81 | "rate": 24, 82 | "precision": { 83 | "1959": [ 84 | 1.0 85 | ] 86 | } 87 | }, 88 | "snips/datetime": { 89 | "recall": { 90 | "1959": [ 91 | 1.0 92 | ] 93 | }, 94 | "rate": 13, 95 | "precision": { 96 | "1959": [ 97 | 0.9230769230769231 98 | ] 99 | } 100 | } 101 | }, 102 | "precision": { 103 | "1959": [ 104 | 1.0 105 | ] 106 | } 107 | } 108 | }, 109 | "execution_time": { 110 | "pred": { 111 | "1959": [ 112 | 0.021230380535125732 113 | ] 114 | }, 115 | "fit": { 116 | "1959": [ 117 | 53.977948904037476 118 | ] 119 | } 120 | }, 121 | "validation_path": "data/en/amt/bench/heavy/2000/snips/validate_SearchScreeningEvent.json", 122 | "training_path": "data/en/amt/bench/heavy/2000/snips/train_SearchScreeningEvent.json", 123 | "training_name": "SearchScreeningEvent", 124 | "model_name": "Snips", 125 | "training_sizes": [ 126 | 1959 127 | ] 128 | } -------------------------------------------------------------------------------- /data/test_label.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ajinkyaT/CNN_Intent_Classification/e95fe9898e0617959012cbe4488fd4e4d7ffc517/data/test_label.npy -------------------------------------------------------------------------------- /data/test_text.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ajinkyaT/CNN_Intent_Classification/e95fe9898e0617959012cbe4488fd4e4d7ffc517/data/test_text.npy -------------------------------------------------------------------------------- /data/train_label.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ajinkyaT/CNN_Intent_Classification/e95fe9898e0617959012cbe4488fd4e4d7ffc517/data/train_label.npy -------------------------------------------------------------------------------- /data/train_text.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ajinkyaT/CNN_Intent_Classification/e95fe9898e0617959012cbe4488fd4e4d7ffc517/data/train_text.npy -------------------------------------------------------------------------------- /glove.6B/.gitignore: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /prepare_data.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Tue May 15 23:27:42 2018 4 | 5 | @author: batman 6 | """ 7 | 8 | 9 | import json 10 | import re 11 | import numpy as np 12 | 13 | intent_types = [ 14 | 'AddToPlaylist','BookRestaurant','GetWeather','RateBook','SearchCreativeWork','SearchScreeningEvent' 15 | ] 16 | 17 | # Function for text preprocessing 18 | def clean_str(string): 19 | """ 20 | Tokenization/string cleaning for all datasets except for SST. 21 | Original taken from https://github.com/yoonkim/CNN_sentence/blob/master/process_data.py 22 | """ 23 | string = re.sub(r"[^A-Za-z0-9(),!?\'\`]", " ", string) 24 | string = re.sub(r"\'s", " \'s", string) 25 | string = re.sub(r"\'ve", " \'ve", string) 26 | string = re.sub(r"n\'t", " n\'t", string) 27 | string = re.sub(r"\'re", " \'re", string) 28 | string = re.sub(r"\'d", " \'d", string) 29 | string = re.sub(r"\'ll", " \'ll", string) 30 | string = re.sub(r",", " , ", string) 31 | string = re.sub(r"!", " ! ", string) 32 | string = re.sub(r"\(", " \( ", string) 33 | string = re.sub(r"\)", " \) ", string) 34 | string = re.sub(r"\?", " \? ", string) 35 | string = re.sub(r"\s{2,}", " ", string) 36 | return string.strip().lower() 37 | 38 | ### Process training data 39 | 40 | def process_data(): 41 | train_list = [] 42 | train_label_list = [] 43 | for intent in range(len(intent_types)): 44 | with open("data/raw_json_data/"+intent_types[intent]+"/train_"+intent_types[intent]+"_full.json",encoding='utf-8') as f: 45 | data = json.load(f) 46 | 47 | 48 | sent_list =[] 49 | label_list =[] 50 | for i in range(len(data[intent_types[intent]])): 51 | txt = '' 52 | for j in range(len(data[intent_types[intent]][i]['data'])): 53 | txt = txt+data[intent_types[intent]][i]['data'][j]['text'] 54 | txt = clean_str(txt) 55 | sent_list.append(txt) 56 | 57 | for i in range(len(sent_list)): 58 | label_list.append(intent_types[intent]) 59 | 60 | filename = intent_types[intent]+'_train.txt' 61 | 62 | with open("data/processed_data/"+filename, mode="w", encoding='utf-8') as outfile: 63 | for s in sent_list: 64 | outfile.write("%s\n" % s) 65 | train_list.extend(sent_list) 66 | train_label_list.extend(label_list) 67 | 68 | ### Process test data 69 | test_list = [] 70 | test_label_list = [] 71 | for intent in range(len(intent_types)): 72 | with open("data/raw_json_data/"+intent_types[intent]+"/validate_"+intent_types[intent]+".json",encoding='utf-8') as f: 73 | data = json.load(f) 74 | 75 | 76 | sent_list =[] 77 | label_list =[] 78 | for i in range(len(data[intent_types[intent]])): 79 | txt = '' 80 | for j in range(len(data[intent_types[intent]][i]['data'])): 81 | txt = txt+data[intent_types[intent]][i]['data'][j]['text'] 82 | txt = clean_str(txt) 83 | sent_list.append(txt) 84 | 85 | for i in range(len(sent_list)): 86 | label_list.append(intent_types[intent]) 87 | filename = intent_types[intent]+'_test.txt' 88 | 89 | with open("data/processed_data/"+filename, mode="w", encoding='utf-8') as outfile: 90 | for s in sent_list: 91 | outfile.write("%s\n" % s) 92 | test_list.extend(sent_list) 93 | test_label_list.extend(label_list) 94 | 95 | # how to save list as array: np.array(myList).dump(open('array.npy', 'wb')) 96 | # how to load aan array: myArray = np.load(open('array.npy', 'rb')) 97 | np.array(train_list).dump(open('data/train_text.npy', 'wb')) 98 | np.array(train_label_list).dump(open('data/train_label.npy', 'wb')) 99 | np.array(test_list).dump(open('data/test_text.npy', 'wb')) 100 | np.array(test_label_list).dump(open('data/test_label.npy', 'wb')) 101 | 102 | 103 | 104 | 105 | 106 | if __name__ == '__main__': 107 | process_data() --------------------------------------------------------------------------------