├── README.md └── data └── questions.txt /README.md: -------------------------------------------------------------------------------- 1 | # CMU Advanced NLP Assignment 2: End-to-end NLP System Building 2 | 3 | Large language models (LLMs) such as Llama2 have been shown effective for question-answering ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)), however, they are often limited by their knowledge in certain domains. A common technique here is to augment LLM's knowledge with documents that are relevant to the question. In this assignment, you will *develop a retrieval augmented generation system (RAG)* ([Lewis et al., 2021](https://arxiv.org/abs/2005.11401)) that's capable of answering questions about the [Language Technology Institute](https://lti.cs.cmu.edu) (LTI) and [Carnegie Mellon University](https://www.cmu.edu) (CMU). 4 | 5 | ``` 6 | Q: Who is offering the Advanced NLP course in Spring 2024? 7 | A: Graham Neubig 8 | ``` 9 | 10 | So far in your machine learning classes, you may have experimented with standardized tasks and datasets that were easily accessible. However, in the real world, NLP practitioners often have to solve a problem from scratch (like this one!). This includes gathering and cleaning data, annotating your data, choosing a model, iterating on the model, and possibly going back to change your data. In this assignment, you'll get to experience this full process. 11 | 12 | Please note that you'll be building your own system end-to-end for this assignment, and *there is no starter code*. You must collect your own data and develop a model of your choice on the data. We will be releasing the inputs for the test set a few days before the assignment deadline, and you will run your already-constructed system over this data and submit the results. We also ask you to follow several experimental best practices, and describe the result in your report. 13 | 14 | The key checkpoints for this assignment are, 15 | 16 | - [ ] [Understand the task specification](#task-retrieval-augmented-generation-rag) 17 | - [ ] [Prepare your raw data](#preparing-raw-data) 18 | - [ ] [Annotate data for model development](#annotating-data) 19 | - [ ] [Develop a retrieval augmented generation system](#developing-your-rag-system) 20 | - [ ] [Generating results](#generating-results) 21 | - [ ] [Write a report](#writing-report) 22 | - [ ] [Submit your work](#submission--grading) 23 | 24 | All deliverables are due by **Tuesday, March 12th**. This is a group assignment, see the assignment policies for this class.[^1] 25 | 26 | ## Task: Retrieval Augmented Generation (RAG) 27 | 28 | You'll be working on the task of factual question-answering (QA). We will focus specifically on questions about various facts concerning LTI and CMU. Since existing QA systems might not have the necessary knowledge in this domain, you will need to augment each question with relevant documents. Given an input question, your system will first retrieve documents and use those documents to generate an answer. 29 | 30 | ### Data Format 31 | 32 | **Input** (`questions.txt`): A text file containing one question per line. 33 | 34 | **Output** (`system_output.txt`): A text file containing system generated answers. Each line contains a single answer string generated by your system for the corresponding question from `questions.txt`. 35 | 36 | **Reference** (`reference_answers.txt`): A text file containing reference answers. Each line contains one or more reference answer strings for the corresponding question from `questions.txt`. 37 | 38 | Read our [model and data policy](#model-and-data-policy) for this assignment. 39 | 40 | ## Preparing raw data 41 | 42 | ### Compiling a knowledge resource 43 | 44 | For your test set and the RAG systems, you will first need to compile a knowledge resource of relevant documents. You are free to use any publicly available resource, but we *highly recommend* including the following, 45 | 46 | + Faculty @ LTI 47 | - List of faculty ([LTI faculty directory](https://lti.cs.cmu.edu/people/faculty/index.html)). Limit to "Core Faculty" for this assignment. 48 | - Research papers by LTI faculty and their metadata ([Semantic Scholar API](https://www.semanticscholar.org/product/api)). Limit your resource to open access papers published in 2023. Include both the paper and its metadata. You can limit the metadata to title, abstract, authors, publication venue, year and tldr. 49 | - Teaching (see below) 50 | + Courses @ CMU 51 | - Courses offered by each department at CMU and their metadata such as instructors, locations, and credits. ([Schedule of Classes](https://enr-apps.as.cmu.edu/open/SOC/SOCServlet/completeSchedule)) 52 | - Academic calendars for 2023-2024 and 2024-2025 ([CMU calendar](https://www.cmu.edu/hub/calendar/)) 53 | + Academics @ LTI 54 | - Programs offered by LTI ([website](https://lti.cs.cmu.edu/academics/index.html)). Navigate to individual program webpages to find program overview, requirements and curriculum etc., 55 | - Program handbooks for information on curriculum, requirements and staff ([PhD](https://lti.cs.cmu.edu/academics/phd-programs/files/handbook_phd_2023-2024.pdf), [MLT](https://lti.cs.cmu.edu/academics/masters-programs/files/mlt-student-handbook-2023-2024.pdf), [MIIS](https://lti.cs.cmu.edu/academics/masters-programs/files/miis-handbook_2023-2024.pdf), [MCDS](https://lti.cs.cmu.edu/academics/masters-programs/files/mcds-student-handbook-2023_2024.pdf), [MSAII](https://lti.cs.cmu.edu/academics/masters-programs/files/handbook-msaii-2022-2023.pdf)) 56 | + Events @ CMU 57 | - Spring carnival and reunion weekend 2024 ([schedule](https://web.cvent.com/event/ab7f7aba-4e7c-4637-a1fc-dd1f608702c4/websitePage:645d57e4-75eb-4769-b2c0-f201a0bfc6ce?locale=en)) 58 | - Commencement 2024 ([schedule](https://www.cmu.edu/commencement/schedule/index.html)) 59 | + History @ SCS and CMU 60 | - School of Computer Science ([25 great things](https://www.cs.cmu.edu/scs25/25things), [history](https://www.cs.cmu.edu/scs25/history)) 61 | - [CMU fact sheet](https://www.cmu.edu/about/cmu_fact_sheet_02.pdf) and [history](https://www.cmu.edu/about/history.html) 62 | - Buggy and it's history ([article](https://www.cmu.edu/news/stories/archives/2019/april/spring-carnival-buggy.html)) 63 | - Athletics ([Tartans](https://athletics.cmu.edu/athletics/tartanfacts), [Scotty](https://athletics.cmu.edu/athletics/mascot/about), [Kiltie Band](https://athletics.cmu.edu/athletics/kiltieband/index)) 64 | 65 | ### Collecting raw data 66 | 67 | Your knowledge resource might include a mix of HTML pages, PDFs, and plain text documents. You will need to clean this data and convert it into a file format that suites your model development. Here are some tools that you could use, 68 | 69 | + For all things related to published research, you can use the [Semantic Scholar API](https://www.semanticscholar.org/product/api) to collect papers and their metadata. 70 | + To parse PDF documents into plain text, you can use [pypdf](https://github.com/py-pdf/pypdf) or [pdfplumber](https://github.com/jsvine/pdfplumber). 71 | + To process HTML pages, you can use [beautifulsoup4](https://pypi.org/project/beautifulsoup4/). 72 | 73 | By the end of this step, you will have a collection of documents that will serve as the knowledge resource for your RAG system. 74 | 75 | ## Annotating data 76 | 77 | Next, you will want to annotate question-answer pairs for two purposes: testing/analysis and training. Use the documents you compiled in the previous step to identify candidate questions for annotation. You will then use the same set of documents to identify answers for your questions. 78 | 79 | ### Test data 80 | 81 | The testing (and analysis) data will be the data that you use to make sure that your system is working properly. In order to do so, you will want to annotate enough data so that you can get an accurate estimate of how your system is doing, and if any improvements to your system are having a positive impact. Some guidelines on this, 82 | 83 | + *Domain Relevance*: Your test data should be similar to the data that you will finally be tested on (questions about LTI and CMU). Use the knowledge resources mentioned above to curate your test set. 84 | + *Diversity*: Your test data should cover a wide range of questions about LTI and CMU. 85 | + *Size*: Your test data should be large enough to distinguish between good and bad models. If you want some guidelines about this, see the lecture on experimental design and human annotation.[^2] 86 | + *Quality*: Your test data should be of high quality. We recommend that you annotate it yourself and validate your annotations within your team. 87 | 88 | To help you get started, here are some example questions, 89 | 90 | + Questions that could be answered by just prompting a LLM 91 | - When was Carnegie Mellon University founded? 92 | + Questions that can be better answered by augmenting LLM with relevant documents 93 | - Who is the president of CMU? 94 | + Questions that are likely answered only through augmentation 95 | - What courses are offered by Graham Neubig at CMU? 96 | + Questions that are sensitive to temporal signals 97 | - Who is teaching 11-711 in Spring 2024? 98 | 99 | See [Vu et al., 2023](https://arxiv.org/abs/2310.03214) for ideas about questions to prompt LLMs. For questions with multiple valid answers, you can include multiple reference answers per line in `reference_answers.txt` (separated by a semicolon `;`). As long as your system generates one of the valid answers, it will be considered correct. 100 | 101 | This test set will constitute `data/test/questions.txt` and `data/test/reference_answers.txt` in your [submission](#submission--grading). 102 | 103 | ### Training data 104 | 105 | The choice of training data is a bit more flexible, and depends on your implementation. If you are fine-tuning a model, you could possibly: 106 | 107 | + Annotate it yourself manually through the same method as the test set. 108 | + Do some sort of automatic annotation and/or data augmentation. 109 | + Use existing datasets for transfer learning. 110 | 111 | If you are using a LLM in a few-shot learning setting, you could possibly: 112 | 113 | + Annotate examples for the task using the same method as the test set. 114 | + Use existing datasets to identify examples for in-context learning. 115 | 116 | This training set will constitute `data/train/questions.txt` and `data/train/reference_answers.txt` in your [submission](#submission--grading). 117 | 118 | ### Estimating your data quality 119 | 120 | An important component of every data annotation effort is to estimate its quality. A standard approach is to measure inter-annotator agreement (IAA). To measure this, at least two members of your team should annotate a random subset of your test set. Compute IAA on this subset and report your findings. 121 | 122 | ## Developing your RAG system 123 | 124 | Unlike assignment 1, there is no starter code for this assignment. You are *free to use any open-source model and library*, just make sure you provide due credit in your report. See our [model policy](#model-and-data-policy). 125 | 126 | For your RAG system, you will need the following three components,  127 | 128 | 1. Document & query embedder 129 | 2. Document retriever 130 | 3. Document reader (aka. question-answering system) 131 | 132 | To get started, you can try langchain's RAG stack that utilizes GPT4All, Chroma and Llama2 ([langchain docs](https://python.langchain.com/docs/use_cases/question_answering/local_retrieval_qa)). 133 | 134 | Some additional resources that could be useful, 135 | 136 | + [11711 lecture notes](http://www.phontron.com/class/anlp2024/lectures/#retrieval-and-rag-feb-15) 137 | + [ACL 2023 tutorial on retrieval-augmented LMs](https://acl2023-retrieval-lm.github.io) 138 | + [llama-recipes](https://github.com/facebookresearch/llama-recipes/tree/main/demo_apps/RAG_Chatbot_example) for an example RAG chatbot with Llama2. 139 | + [Ollama](https://github.com/ollama/ollama) or [llama.cpp](https://github.com/ggerganov/llama.cpp) to run LLMs locally on your machine. 140 | 141 | All the code for your data preprocessing, model development and evaluation will be a part of your GitHub repository (see [submission](#submission--grading) for details). 142 | 143 | ## Generating results 144 | 145 | Finally, you will run your systems on our test set (questions only) and submit your results to us. This test set will be released on **Monday, March 11th**. 146 | 147 | ### Unseen test set 148 | 149 | This test set will be curated by the course staff and will evaluate your system's ability to respond to a variety of questions about LTI and CMU. Because the goal of this assignment is not to perform hyperparameter optimization on this private test set, we ask you to not overfit to this test set. You are allowed to submit up to *three* output files (`system_outputs/system_output_{1,2,3}.txt`). We will use the best performing file for grading. 150 | 151 | ### Evaluation metrics 152 | 153 | Your submissions will be evaluated on standard metrics, answer recall, exact match and F1. See section 6.1 of the [original SQuAD paper](https://arxiv.org/abs/1606.05250) for details. These metrics are token-based and measure the overlap between your system answer and the reference answer(s). Therefore, we recommend keeping your system generated responses as concise as possible. 154 | 155 | ## Writing report 156 | 157 | We ask you to write a report detailing various aspects about your end-to-end system development (see the grading criteria below). 158 | 159 | There will be a 7 page limit for the report, and there is no required template. However, we encourage you to use the [ACL template](https://github.com/acl-org/acl-style-files). 160 | 161 | > [!IMPORTANT] 162 | > Make sure you cite all your sources (open-source models, libraries, papers, blogs etc.,) in your report. 163 | 164 | ## Submission & Grading 165 | 166 | ### Submission 167 | 168 | Submit all deliverables on Canvas. Your submission checklist is below, 169 | 170 | - [ ] Your report. 171 | - [ ] A link to your GitHub repository containing your code.[^3] 172 | - [ ] A file listing contributions of each team member, 173 | - [ ] data annotation contributions from each team member (e.g. teammate A: instances 1-X; teammate B: instances X-Y, teammate C: instances Y-Z). 174 | - [ ] data collection (scraping, processing) and modeling contributions from each team member (e.g. teammate A: writing scripts to ..., implementing ...; teammate B:...; teammate C:...;) 175 | - [ ] Testing and training data you annotated for this assignment. 176 | - [ ] Your system outputs on our test set. 177 | 178 | Your submission should be a zip file with the following structure (assuming the lowercase Andrew ID is ANDREWID). Make one submission per team. 179 | 180 | ``` 181 | ANDREWID/ 182 | ├── report.pdf 183 | ├── github_url.txt 184 | ├── contributions.md 185 | ├── data/ 186 | │ ├── test/ 187 | │ │ ├── questions.txt 188 | │ │ ├── reference_answers.txt 189 | │ ├── train/ 190 | │ │ ├── questions.txt 191 | │ │ ├── reference_answers.txt 192 | ├── system_outputs/ 193 | │ ├── system_output_1.txt 194 | │ ├── system_output_2.txt (optional) 195 | │ ├── system_output_3.txt (optional) 196 | └── README.md 197 | ``` 198 | 199 | ### Grading 200 | 201 | The following points (max. 100 points) are derived from the results and your report. See course grading policy.[^4] 202 | 203 | + **Submit data** (15 points): submit testing/training data of your creation. 204 | + **Submit code** (15 points): submit your code for preprocessing and model development in the form of a GitHub repo. We may not necessarily run your code, but we will look at it. So please ensure that it contains up-to-date code with a README file outlining the steps to run it. Your repo 205 | + **Results** (30 points): points based on your system's performance on our private test set. 10 points for non-trivial performance,[^5] plus up to 20 points based on level of performance relative to other submissions from the class. 206 | + **Report**: below points are awarded based on your report. 207 | + **Data creation** (10 points): clearly describe how you created your data. Please include the following details, 208 | - How did you compile your knowledge resource, and how did you decide which documents to include? 209 | - How did you extract raw data? What tools did you use? 210 | - What data was annotated for testing and training (what kind and how much)? 211 | - How did you decide what kind and how much data to annotate? 212 | - What sort of annotation interface did you use? 213 | - How did you estimate the quality of your annotations? (IAA) 214 | - For training data that you did not annotate, did you use any extra data and in what way? 215 | + **Model details** (10 points): clearly describe your model(s). Please include the following details, 216 | - What kind of methods (including baselines) did you try? Explain at least two variations (more is welcome). This can include which model you used, which data it was trained on, training strategy, etc. 217 | - What was your justification for trying these methods? 218 | + **Results** (10 points): report raw numbers from your experiments. Please include the following details, 219 | - What was the result of each model that you tried on the testing data that you created? 220 | - Are the results statistically significant? 221 | + **Analysis** (10 points): perform quantitative/qualitative analysis and present your findings, 222 | - Perform a comparison of the outputs on a more fine-grained level than just holistic accuracy numbers, and report the results. For instance, how did your models perform across various types of questions? 223 | - Perform an analysis that evaluates the effectiveness of retrieve-and-augment strategy vs closed-book use of your models. 224 | - Show examples of outputs from at least two of the systems you created. Ideally, these examples could be representative of the quantitative differences that you found above. 225 | 226 | ## Model and Data Policy 227 | 228 | To make the assignment accessible to everyone, 229 | 230 | + You are only allowed to use models that are also accessible through [HuggingFace](https://huggingface.co/models). This means you may *not* use closed models like OpenAI models, but you *can* opt to use a hosting service for an open model (such as the Hugging Face or Together APIs). 231 | + You are only allowed to include publicly available data in your knowledge resource, test data and training data. 232 | + You are welcome to use any open-source library to assist your data annotation and model development. Make sure you check the license and provide due credit. 233 | 234 | If you have any questions about whether a model or data is allowed, please ask on Piazza. 235 | 236 | ## Acknowledgements 237 | 238 | This assignment was based on the the [Fall 2023 version of this assignment](https://github.com/cmu-anlp/nlp-from-scratch-assignment-2023/tree/main). 239 | 240 | ## References 241 | 242 | + Lewis et al., 2021. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401). 243 | + Touvron et al., 2023. [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/abs/2307.09288). 244 | + Vu et al., 2023. [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://arxiv.org/abs/2310.03214). 245 | 246 | 247 | 248 | [^1]: See the [assignment policies](http://www.phontron.com/class/anlp2024/assignments/#assignment-policies) for this class, including submission information, late day policy and more. 249 | 250 | [^2]: See the [lecture notes](http://www.phontron.com/class/anlp2024/lectures/#experimental-design-and-human-annotation-feb-13) on experimental design and human annotation for guidance on annotation, size of test/train data, and general experimental design. 251 | 252 | [^3]: Create a private GitHub repo and give access to the TAs in charge of this assignment by the deadline. See piazza announcement post for our GitHub usernames. 253 | 254 | [^4]: Grading policy: http://www.phontron.com/class/anlp2024/course_details/#grading 255 | 256 | [^5]: In general, if your system is generating answers that are relevant to the question, it would be considered non-trivial. This could be achieved with a basic RAG system. 257 | -------------------------------------------------------------------------------- /data/questions.txt: -------------------------------------------------------------------------------- 1 | What is another name for the vehicle being raced in sweepstakes? 2 | What's the course number for large language models methods and application? 3 | When will the classes begin in the Fall 2024 semester? 4 | In spring 2024, How many units is course 10315? 5 | In the TAPLoss paper, what does TAP stand for? 6 | What is the purpose of the ACL 60/60 evaluation sets? 7 | In summer 2024, What is the last day of Mini-5 classes? 8 | What number do all of the Drama classes start with? 9 | Carnegie Mellon University is home to how many members of the National Academy of Medicine (NAM)? 10 | What class room was advanced NLP taught last semester? 11 | Which LTI faculty member is an author on "Accelerating Diffusion-based Combinatorial Optimization Solvers by Progressive Distillation"? 12 | At what conference was "Fully Unsupervised Topic Clustering of Unlabelled Spoken Audio Using Self-Supervised Representation Learning and Topic Model" published? 13 | What is the full name of the conference where the paper TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement, got published? 14 | Who used the first emoticon at CMU? 15 | Who is the PI of CLAW Lab? 16 | In the BiasX paper, how much do imperfect machine-generated explanations help in correctly identifying subtly (non-)toxic content? 17 | What is Graham Neubig's job title? 18 | In fall 2023, What is the title of course 05291? 19 | How large are the trained models by the authors of SantaCoder paper? 20 | What is David Garlan's two word title; this is not the one with the word professor in it? 21 | In fall 2023, Who is the instructor for unit 02718? 22 | What is the name of the proposed approach that extends pretrained transformer models to handle unlimited input lengths? 23 | In fall 2024, What is the deadline for Mini-2 drop and withdrawal grade assigned after this date? 24 | When was aluminum first used to build buggies? 25 | Where will the Phi Beta Kappa Initiation Ceremony (not Reception) be held on May 9? 26 | In fall 2024, What is the deadline for Mini-1 drop and withdrawal grade assigned after this date? 27 | How much decrease in memory consumption (multi GPU setup) does SAMA showcase in large-scale meta learning benchmarks? 28 | When does the Spring 2025 course registeration start for masters students? 29 | In fall 2023, What is the location of course 05317? 30 | What is ValuePrism? 31 | What percentage of CMU's Computer Science first year students in 2019 were women? 32 | In spring 2024, What is the deadline for Mini-3 pass/no pass and withdrawal? 33 | What's the paper title for the paper that released a method called IPA? 34 | In the paper "Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning", on which tasks has IPA shown significant improvements? 35 | How many credits is 11824 worth? 36 | In fall 2023, What is the title of course 05391? 37 | Who is the first author on "Extracting Training Data from Diffusion Models"? 38 | In fall 2023, Who is the instructor for course 05315? 39 | How much does it cost to apply for the MLT program if an application is submitted on the day before the deadline? 40 | How many credits is the MIIS-16 program? 41 | When was the first U.S. drama degree awarded at Carnegie Tech? 42 | What can PhD students use LTI's computer cluster for? 43 | Who is teaching the Advanced Topics in Multimodal Machine Learning in Spring 2024? 44 | What language did Meloni et al (2021) achieve state-of-the-art results on for protoform reconstruction? 45 | Carnegie Mellon University (CMU) Athletics Hall of Fame was established in which year? 46 | Which department in the School of Computer Science was formed in 2006? 47 | What does SPAE stand for? 48 | In which conference was HomeRobot published? 49 | Carnegie Mellon University is home to how many members of the National Academy of Sciences (NAS)? 50 | In which year was the official Scotty costume unveiled? 51 | At which conference venue was the framework tax published? 52 | In fall 2023, What is the deadline for adding, auditing, and tuition adjustment drop for Mini-2 (deadline 1)? 53 | What year was "End-to-End Speech Recognition: A Survey" published? 54 | Does CMU discriminate based on race? 55 | According to the paper PROMPT2MODEL: Generating Deployable Models from Natural Language Instructions, what is the exact match achieved by gpt-3.5-turbo on the Squad dataset? 56 | In fall 2023, What is the location for unit 02700? 57 | By whom was Kevlar fiber invented? 58 | In spring 2024, What is the title of course 15151? 59 | When are the Spring 2024 grades due? 60 | What are the course number(s) for the courses on LLMs? 61 | In the KALE lexical expansion paper, what three datasets are evaluated? 62 | According to the MSAII handbook, what is David Garlan's office building and number? 63 | How many authors are on the paper "Multimodal Fusion Interactions: A Study of Human and Automatic Quantification"? 64 | ICML is the abbreviation for which conference? 65 | What are the two standard benchmarks used to evaluate the performance of FREDOM? 66 | Which conference was the paper Cross-Modal Fine-Tuning: Align then Refine published in? 67 | Who taught Advanced Natural Language Processing in Fall 2023? 68 | Is a valid CMU ID needed to make fitness reservations? 69 | When did the Fall Break start in 2023? 70 | What are the course number(s) for the Search Engines course? 71 | To complete the course requirements for the PhD in Language and Information Technologies degree, how many course units of graduate courses does the student have to pass? 72 | In spring 2024, What is the title of course 10701? 73 | In summer 2024, When is the first day of Mini-6 classes? 74 | Who is the first paper on the KALE paper by Jamie Callan's group? 75 | How many months is the longer track of the MIIS program? 76 | What does ICTIR stand for? 77 | What is included in the ACL 60/60 evaluation dataset? 78 | Which benchmark was used in the study? 79 | In fall 2024, When do Semester & Mini-1 Classes begin? 80 | In spring 2024, What is the title of course 17200? 81 | In spring 2024, What is the deadline for adding or dropping a Mini-4 course with tuition adjustment? 82 | What is the phone number for CMU's office of Title IX initiatives? 83 | In the paper "Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation, what is the reduction in word error rates achieved by the proposed models on LibriSpeech test-clean? 84 | Which institute did Carnegie Tech merge with in 1967? 85 | How many papers does Lori S. Levin have on Semantic Scholar? 86 | What is the language technologies institute's phone number according to the MCDS handbook? 87 | In the Convoifilter paper, what is the WER achieved by the joint fine-tuning strategy? 88 | In fall 2023, When is the deadline to drop a Mini-2 course with a withdrawal grade assigned? 89 | Does LTI offer a course on text mining? 90 | Besides Pittsburgh, where else does CMU have physical campuses? 91 | Which LTI prof co-authored the paper titled "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena"? 92 | How many authors are on the SENTECON paper? 93 | What are the last names of the professors that taught 11-711 in Fall 2023? 94 | How many test examples are included in the WebArena benchmark? 95 | How many Tony Awards have alumni and current/former faculty won so far? 96 | Does SCS Interdisciplinary offer more than 1 course in Summer 2024? 97 | What does A-LoL use to filter negative advantage (low-quality) data points during training? 98 | In spring 2024, When do classes start after the winter break? 99 | What is the task success rate of the GPT-4-based agent in WebArena? 100 | Is there a limit on the number of guests who can attend the main commencement ceremony? 101 | What is the full name of the conference where the paper The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Linkbetween Phonemes and Facial Features, got published? 102 | In the "Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research," how many participants were in the survey from the NLP community? 103 | What is the name of the benchmark that extends SUPERB to multiple languages? 104 | What model does SYNTACC use for multi-accent speech synthesis? 105 | What are the units for linguistics lab? 106 | How many courses does Abdelghany teach in Summer 2024? 107 | Which LTI faculty member works on recommender systems? 108 | What does the Plan module in the PET framework do? 109 | What is the term for the discrepancies between increases in computational throughput and reductions in floating point operations, and improvements in wall-clock inference latency? 110 | In spring 2024, What is the course number for Game Theoretic Probability, Statistics and Learning? 111 | Which Faculty from LTI Co-authored the paper Transformed Protoform Reconstruction? 112 | What is the full name of the conference where the paper ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages, got published? 113 | In the KALE paper, what evaluation metrics were reported on TREC DL 19? 114 | When are the sweepstakes finals at Spring Carnival? 115 | Where is the SENTECON paper published at? 116 | What is the framework proposed to simplify the control problem of embodied agents using LLMs? 117 | In spring carnival, Scotch'n'Soda's theatre carnival shows are on what days of the week? 118 | How many credits is the MIIS Capstone Planning Seminar worth? 119 | What are the attention dot-product scores in the Unlimiformer approach? 120 | Which LTI faculty member is an author on the COBRA Frames paper? 121 | Have Professors Bhiksha Raj and Rita Singh co-authored a paper? 122 | Who is the Director of the MSAII program? 123 | What is the full name of the conference where the paper Why do Nearest Neighbor Language Models Work?, got published? 124 | What is the mean confidence difference for the "he, she" gender-word pair in the paper "Language Models Get a Gender Makeover"? 125 | In spring 2024, When do Mini-3 faculty course evaluations open? 126 | Which ranker outperformed BM25 consistently in the InPars study? 127 | For additional information about the MIIS program, who should you contact? 128 | According to the framework tax paper, what is observed to be growing as hardware speed increases over time? 129 | In spring 2025, What is the deadline for adding, auditing, and tuition adjustment drop for Mini-3? 130 | In "Aligning Large Multimodal Models with Factually Augmented RLHF," what is the name of the method that they propose for alignment? 131 | What is the role of a chute flagger in the sweepstakes competition? 132 | What is the improvement achieved by the PET framework on the AlfWorld instruction following benchmark? 133 | In the Plan, Eliminate and Track paper, which benchmark was used in the experiments? 134 | Who should LTI PhD students contact if they have a question about their offices? 135 | Which populations were found to be predominantly aligned with by the datasets and models in the NLPositionality study? 136 | In fall 2023, Who is the instructor for unit 02701? 137 | In the paper "Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation, what is the reduction in word error rates achieved by the proposed models on LibriSpeech testother? 138 | Which LTI faculty is involved in the work Improving Factuality of Abstractive Summarization via Contrastive Reward Learning? 139 | Is there a YouTube channel for The Kiltie Band? 140 | Where is the Senior Leadership Recognition Ceremony held on May 10 2024? 141 | What is the name of Yonatan Bisk's lab? 142 | How many months does the advanced study MIIS degree typically take? 143 | Carnegie Mellon University is home to how many members of the National Academy of Engineering (NAE)? 144 | In fall 2024, When is Labor Day? 145 | Which two faculty are co-teaching the neural code generation course? 146 | What are the course numbers for question answering courses at LTI? 147 | What number do all of the Architecture classes start with? 148 | What is the full name of the conference where the paper A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech, got published? 149 | In fall 2023, What is the course number for Undergraduate Research in Computational Biology? 150 | In the BASS paper from Interspeech 2023, what is the improvement in ROUGE-L score demonstrated by the proposed block-wise training method? 151 | What is the accuracy using SHAP reduction? 152 | What are the two key factors addressed by CSurF? 153 | In spring 2024, What is the title of course 15110? 154 | What is the name of the proposed approach for fairness domain adaptation in semantic scene segmentation? 155 | What is the full name of the conference where the paper BASS: Block-wise Adaptation for Speech Summarization, got published? 156 | In spring 2024, How many units is course 15090 worth? 157 | How much does it cost to apply for the MLT program if an application is submitted on December 4th, 2023? 158 | What is the proposed learning objective to improve perceptual quality of speech? 159 | Where does the sharp right-hand turn of the buggy course occur? 160 | In spring 2024, Who is the instructor for course 15151? 161 | What LTI professor was the last author in "To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing"? 162 | What is the BartScore achieved by the CRL-COM (R) system from the paper Improving Factuality of Abstractive Summarization via Contrastive Reward Learning, on the XSUM dataset? 163 | What types of information does FiT5 integrate into a single unified model? 164 | In spring 2024, How many units is course 17214? 165 | What time in the day does SafeWalk start? 166 | In fall 2023, Where is unit 02518 held? 167 | What is the name of the novel framework introduced for learning unified multi-sensory object property representations? 168 | According to the paper ChatGPT MT, which languages does the study suggest ChatGPT is especially disadvantaged for? 169 | What can be used to attack multimodal models that allow users to provide images? 170 | In fall 2023, What are the units for unit 02614? 171 | Where was the paper titled "Computational Language Acquisition with Theory of Mind" published? 172 | In fall 2023, When is unit 02761 on Tuesdays and Thursdays? 173 | What class is taught by Eric Nyberg and Teruko Mitamura? 174 | When is the Holi celebration at the Spring Carnival? 175 | At what conference was "BASS: Block-wise Adaptation for Speech Summarization" published? 176 | What year did Andrew Carnegie die? 177 | In spring 2024, What is the location of course 10500? 178 | Which LTI prof co-authored the paper titled "AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models"? 179 | In fall 2023, Who are the instructors for unit 02512? 180 | Who taught the first freshman-level computer programming course at CMU? 181 | What is the publication venue of "Assessment of quality of life after upper extremity transplantation: Framework for patient-reported outcome scale domains"? 182 | What are the results of training 1.1B parameter models on Java, JavaScript, and Python subsets of The Stack and evaluating them on MultiPL-E? 183 | In what year was Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning published? 184 | How much does it cost to apply for the MLT program if an application is submitted a week before the deadline? 185 | Tartan Athletics Club was launched in which year? 186 | What is the full name of the conference where the paper Using counterfactual contrast to improve compositional generalization for multi-step quantitative reasonin, got published? 187 | Which was the the first U.S. school to award a degree in drama? 188 | In the paper "Modeling Empathic Similarity in Personal Narratives", what is the name of the dataset created for this task? 189 | How many action types were included in the WebArena benchmark? 190 | What is the name of the event where buggies are raced? 191 | According to the paper PWESUITE: Phonetic Word Embeddings and Tasks They Facilitate, what is the percentage accuracy for rhymes, achieved by the autoencoder model on the evaluation suite? 192 | What are the number of units for independent study: breadth? 193 | Which LTI prof co-authored the paper titled "Exploration on HuBERT with Multiple Resolutions"? 194 | What was the name of the CMU project that created its first high-speed computer network? 195 | In spring 2024, what time does the Subword Modeling class start? 196 | Who is ther first author of the paper Unlimiformer: Long-Range Transformers with Unlimited Length Input? 197 | Is the GRE optional for the Master of Science in Intelligent Information Systems application? Answer yes or no. 198 | How much does it cost to apply for the MLT program if an application is submitted a month before the deadline? 199 | What is the nickname for the sweepstakes competition? 200 | When was the first freshman-level computer programming course offered at CMU? 201 | What is the number of the HR person at LTI? 202 | In spring 2025, What is the deadline for Mini-3 pass/no pass and withdrawal? 203 | In fall 2023, What is the deadline for adding, auditing, and tuition adjustment drop for Mini-1 (deadline 1)? 204 | Which LTI faculty member is on the paper titled "Fusion-in-T5: Unifying Document Ranking Signals for Improved Information Retrieval"? 205 | What pre-trained model does MOSAIC leverage knowledge from? 206 | What are the names of the people from LTI who co-authored the paper Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms 207 | In spring 2024, What is the title of course 15122? 208 | What models does ESPnet-ST-v2 offer? 209 | In fall 2024, When is the Convocation? 210 | What are the acronmys for all LTI programs that have capstone requirements? 211 | Who is the point of contact for Naval ROTC Commissioning ceremony? 212 | In fall 2023, What are the units for unit 02712? 213 | What number do all of the Biomedical Engineering classes start with? 214 | What is the full name of the conference where the paper Rethinking Voice-Face Correlation: A Geometry View, got published? 215 | In summer 2024, When do Semester & Mini-6 Faculty Course Evaluations open? 216 | Which LTI faculty member does the most work on robots? 217 | In fall 2023, When do the Mid-semester & Mini-1 grades need to be submitted? 218 | Which Carnegie Tech School before 1973 was a college for women? 219 | What was the conclusion of the study regarding the effectiveness of query rewriting techniques using large language models for multilingual, document-grounded question-answering systems? 220 | Where did Graham Neubig get his PhD? 221 | In fall 2023, When are the Semester & Mini-2 Faculty Course Evaluations open? 222 | How many StuCo or Student Led Courses are going to be held in Spring 2024? 223 | What is the benefit of FLARE over existing retrieval augmented LMs? 224 | What is Robert Frederking's phone number according to the MCDS handbook? 225 | How many Electrical & Computer Engineering courses are going to be held in Summer 2024? 226 | In spring 2024, Who is the instructor for course 15195? 227 | The first two years of the PhD program are similar to what master's program? 228 | What is the MOS-Q achieved by the MQTTS quantizer with a code size of 1024, on the VoxCeleb test set? 229 | Where will the Buggy Showcase happen this year? 230 | What percentage of the families investigated in "SHAP-based Prediction of Mother's History of Depression to Understand the Influence on Child Behavior" were white? 231 | What loss functions are proposed in the Fairness Continual Learning approach? 232 | What was the title of paper that proposed a new task, OUTDOOR? 233 | Which stage of a model's lifecycle does the Pentathlon benchmark focus on? 234 | When is the Tartans Got Talent show at the carnival? 235 | Who is the first author of the paper BASS: Block-wise Adaptation for Speech Summarization? 236 | What is the name of the method introduced in "Semantic Pyramid AutoEncoder for Multimodal Generation"? 237 | For additional information about the MSAII program, who should you contact? 238 | What is the 5 letter abbreviation for the MS in artificial intelligence and innovation degree? 239 | What is the process of exchanging pushers during the race called? 240 | What does POMDP stand for? 241 | Which LTI prof co-authored the paper titled "Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning"? 242 | In Spring 2024, what time does "Issues of Practice" course start at in the morning? 243 | What are the protected attributes that CMU does not use in deciding the admission of PhD students? 244 | Did anyone from LTI worked on the paper Don't Take This Out of Context!: On the Need for Contextual Models and Evaluations for Stylistic Rewriting? 245 | In fall 2023, When is Democracy Day and are there any classes? 246 | What are the three main reasons why kNN-LM performs better than standard LMs? 247 | In fall 2023, When do the semester and Mini-1 classes begin? 248 | When did the MLT application period for Fall 2024 admissions start? 249 | How much increase in throughput (single GPU setup) does SAMA showcase in large-scale meta learning benchmarks? 250 | Who are the instructors for the data science seminar? 251 | What is the BigCode project about? 252 | At what conference was Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning published? 253 | At what journal was "Somatosensory and motor representations following bilateral transplants of the hands: A 6-year longitudinal case report on the first pediatric bilateral hand transplant patient" published? 254 | Does LTI offer a course on ethics? 255 | What score does the global model achieve in the 5K data NER setting in Zhisong Zhang, Emma Strubell, and Eduard Hovy's paper on data constraints and structured prediction? 256 | In the paper "COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements", what is the name of the dataset created for studying the contextual dynamics of offensiveness? 257 | What is Martial Herbert's email address? 258 | Who taught Natural Language Processing last fall? 259 | What is the Spearman Correlation of CodeBERTScore with human preference? 260 | When is CMU's main commencement ceremony in 2024? 261 | Who provides a signal for buggy drivers to start the right-hand turn from Schenley Drive onto Frew Street? 262 | Are there any authors of the paper Understanding Political Polarisation using Language Models: A dataset and method, not from CMU? 263 | At what conference was "To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing" published? 264 | What is HomeRobot? 265 | In spring 2024, What is the day and time of course 17422? 266 | The analysis in the paper Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, includes how many diverse languages? 267 | What is the main goal of event grounding? 268 | In summer 2024, When is Independence Day and what is the University's policy on classes? 269 | Who is the current director of The Kiltie Band? 270 | What is the full name of the conference where the paper CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code, got published? 271 | What is the LTI director's phone number? 272 | What are the course number(s) for the NLP course? 273 | Which LTI prof co-authored the paper titled "StyleRF: Zero-Shot 3D Style Transfer of Neural Radiance Fields"? 274 | What saying by Andrew Carnegie is now CMU's school moto? 275 | Which LTI professors wrote "BASS: Block-wise Adaptation for Speech Summarization"? 276 | What are some benefits of using a hybrid model approach for identifying hedges? 277 | In the paper titled "WebArena: A Realistic Web Environment for Building Autonomous Agents", what was the human performance on the proposed benchmark? 278 | What single letter grade do you get for an incomplete grade? 279 | What is the language technologies institute's fax number according to the MCDS handbook? 280 | What is the title of Scotch'n'Soda's performance at the Spring Carnival? 281 | In summer 2024, What is the deadline for Mini-5 pass/no pass and withdrawal? 282 | How many courses does Mechanical Engineering offer in Summer 2024? 283 | When was the Institute for Software Research formed? 284 | Where is Teruko Mitamura's Hierarchical Event Grounding published at? 285 | Pittsburgh Supercomputing Center was created as a joint effort between which entities? 286 | When is Douse-a-Dean event at this year's Spring Carnival? 287 | What time does Leading in a Lean and Six Sigma World start in Summer 2024? 288 | What street is CMU LTI located one? 289 | How many units is the MIIS Capstone Project with course number 11927 for? 290 | Which fraternity won the first race in 1920? 291 | What is the minimum GPA for the MSAII program? 292 | What are the names of the 15.5B parameter models introduced by The BigCode community? 293 | What does FLARE stand for? 294 | What is StyleRF's solution to the three-way dilemma in 3D style transfer? 295 | Where is the Center for Student Diversity and Inclusion Ceremony held on May 11, 2024? 296 | What is Fernando Diaz's job title? 297 | In spring 2024, What is the title of course 15150? 298 | What are the events on May 10 as part of the Commencement program for 2024? 299 | What is the structure attached to a buggy that a person pushes to propel it forward? 300 | If you applied to both the MIIS and MSAII programs on the day before the deadline, how much would it cost? 301 | How many papers does Alexander Hauptmann have on Semantic Scholar? 302 | In spring 2025, When do Mini-3 course drop and withdrawal grade assignment occur? 303 | How many Academy Awards have alumni and current/former faculty won so far? 304 | How many languages does GlobalBench currently cover? 305 | What does TASTE use to better characterize user behaviors? 306 | In the Value Kaleidescope paper by Maarten Sap's group, what is the name of the dataset that is introduced? 307 | HomeRobot OVMM benchmarks include two components or environments. What are they? 308 | In spring 2024, Who are the instructors for course 15122? 309 | In fall 2023, What is the deadline for adding, auditing, and tuition adjustment drop for the semester (deadline 1)? 310 | What is the name of Graham Neubig's lab? 311 | In fall 2023, What is the deadline for Mini-1 Pass/No Pass and withdrawal? 312 | What country does the Dual-Degree Ph.D. in Language and Information Technologies have a partnership with? 313 | Carnegie Mellon University is ranked #1 according to which report in 2022? 314 | In summer 2024, When do Mini-5 Faculty Course Evaluations open? 315 | What does Self-Refine use to provide feedback and refine the initial output? 316 | In November 2006, who chaired the Mascot Identity Task Force? 317 | How much decrease in memory consumption (single GPU setup) does SAMA showcase in large-scale meta learning benchmarks? 318 | What percentage of XLS-R’s performance can a vanilla HuBERT Base model maintain with only $3 \%$ of the data, 4 GPUs, and limited trials? 319 | When is the buggy showcase at the spring carnival? 320 | In summer 2024, What is the deadline for adding, auditing, and tuition adjustment drop for Mini-6? 321 | In fall 2023, When are the Semester & Mini-2 Faculty Course Evaluations closed? 322 | In fall 2023, Is there class and university operation on Labor Day? 323 | What are the codes/numbers of the distinct courses, all titled "Introduction to Computer Systems", that will be offered in the Summer of 2024? 324 | What time in the day does SafeWalk end? 325 | In Fall 2023, how many sections did Shop Skills 48104 have? 326 | When was the two-wheeled buggy eliminated? 327 | Is Yonatan the last author on the plan, eliminate and track paper? 328 | What year is "Neural Mixed Effects for Nonlinear Personalized Predictions" published in? 329 | In spring 2024, Who are the instructors for course 10615? 330 | What is the full name of the conference where the paper Learning to Ask Questions for Zero-shot Dialogue State Tracking, got published? 331 | In the paper "An Approach to Ontological Learning from Weak Labels", what dataset and ontology were used in the investigation? 332 | What is Martial Herbert's office building and number? 333 | Which LTI faculty are involved in the SPAE paper? 334 | In summer 2024, What is the deadline for adding, auditing, and tuition adjustment drop for Mini-5? 335 | Who is the PhD Academic Program Manager for the LTI PhD degree? 336 | Which LTI professor co-authored the paper titled "Text Matching Improves Sequential Recommendation by Reducing Popularity Biases"? 337 | What is the zero shot top-100 accuracy achieved by the chain-of-skills model on the dev set of HotpotQA? 338 | In the paper "Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research", what three topics did the survey investigate regarding concerns about PLMs? 339 | What two LTI professors were on the "Making Scalable Meta Learning Practical" paper? 340 | What was the previous name for the Language Technology Institute? 341 | In spring 2024, What is the title of course 10735? 342 | In spring 2024, What is the course number for Independent Study: Research? 343 | What 11-6XX courses were not taught by LTI faculty in Spring 2024? 344 | What professor was the last author on "Deriving Vocal Fold Oscillation Information from Recorded Voice Signals Using Models of Phonation"? 345 | The first doctorate at Carnegie Tech was awarded in 1919. Who was it awarded to and in what discipline was it in? 346 | In the paper "Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation, what is the reduction in word error rates achieved by the proposed models on Switchboard? 347 | What are the names of the people from LTI who co-authored the paper COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements? 348 | Which two NLP tasks were applied with the NLPositionality framework in the study? 349 | What are the number of units for 11797? 350 | In the paper "KIT’s Multilingual Speech Translation System for IWSLT 2023", what approach was used for effective adaptation in the absence of training data from the target domain? 351 | What is the title of the paper that proposes a novel re-ranker model abbreviated FiT5? 352 | Which CMU professor was on the "Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation" paper? 353 | In spring 2024, What is the deadline for withdrawing from a semester course with a withdrawal grade assigned? 354 | What are the SCS CMU classes grading standard for max GPA? 355 | How much does it cost to apply for the MLT program if an application is submitted on November 20th, 2023? 356 | In the Paaploss paper, in which speech enhancement workflows did the proposed method show improvement? 357 | Which LTI professors wrote "Rethinking Voice-Face Correlation: A Geometry View"? 358 | What city is the Language Technologies Institute at Carnegie Mellon University located in? 359 | Which LTI professor's paper introduced the TASTE algorithm that maps items and users in an embedding space and recommends items by matching their text representations? 360 | In spring 2024, Who are the instructors for course 17313? 361 | When was the final application deadline for the PhD program? Give your answer in dd/mm/yyyy. 362 | In spring 2024, What is the location of course 10716? 363 | The TASTE algorithm was introduced in what paper? 364 | When was the first three-wheeled buggy introduced? 365 | What is the director of the MSAII program's email address? 366 | Which program has an application date of September 30th? 367 | What data source do authors in "Towards Open-Domain Twitter User Profile Inference" collect their public user profiles from? 368 | For additional information about the PhD in Language and Information Technology program, who should you contact? 369 | What are the four stages of the MultiViz method? 370 | In spring 2024, When is the last day of Mini-3 classes? 371 | In spring 2024, Who are the instructors for course 15210? 372 | In fall 2023, What is the title of course 05410? 373 | How many authors contributed to the paper Generalized Glossing Guidelines: An Explicit, Human- and Machine-Readable, Item-and-Process Convention for Morphological Annotation? 374 | In spring 2024, How many units is course 15112 worth? 375 | What is the name of the new class of offline policy gradient algorithms introduced in the paper "Improving Language Models with Advantage-based Offline Policy Gradients"? 376 | How many datasets does GlobalBench currently cover? 377 | What is Carolyn Rose's email address? 378 | Which model performed the best in "Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity"? 379 | Is the Wiegand Gymnasium located in the Jared L. Cohon University Center? 380 | What's the cost in us dollars per program for the masters degrees in language technologies if you submit before the early deadline? 381 | In fall 2024, What is the deadline for Semester add, audit & tuition adjustment drop (deadline 1)? 382 | How many authors are on the paper "Pragmatic Inference with a CLIP Listener for Contrastive Captioning"? 383 | In spring 2024, What is the title of course 15195? 384 | Who generates ValuePrism's contextualized values? 385 | In fall 2024, What is the deadline for Mini-2 add, audit & tuition adjustment drop (deadline 1)? 386 | What course did Lanni teach in Spring 2023? 387 | In fall 2023, When is the Spring 2024 Registration Week? 388 | In spring 2024, Who is the instructor for Dissertation Research? 389 | What is Carolyn Penstein Rose's phone number? 390 | When was Human-Computer Interaction Institute formed? 391 | Which professors at LTI are on leave? 392 | Which LTI faculty member is an author on "Aligning Large Multimodal Models with Factually Augmented RLHF"? 393 | What does CodeBERTScore encode in addition to the generated tokens? 394 | In which month and year the Mascot Identity Task Force was formed? 395 | What's the URL for the code and data of InPars-light 396 | Is a valid CMU ID needed to use the tennis court? 397 | What city is CMU LTI located in? 398 | When did independent organizations, other than fraternities, enter Buggy for the first time? 399 | What is Mona Diab's phone number according to the MCDS handbook? 400 | When was the department of Computer Science (CSD) established at CMU? 401 | In spring 2024, Who is the instructor for course 10403? 402 | According the OUTDOOR paper, what is one of the challenges of navigating in outdoor environments compared to indoor environments? 403 | Who are the authors of the book "The Last Lecture"? 404 | Which LTI faculty member focuses on embodiment? 405 | What is the full name of the workshop where the paper Generalized Glossing Guidelines: An Explicit, Human- and Machine-Readable, Item-and-Process Convention for Morphological Annotation, got published? 406 | Who was the first director of the Robotics Institute? 407 | What are the two courses that are prerequisities for the undergraduate concentration termed the LT concentration? Include the title and course number in parentheses. 408 | What is the publicly available website for WebArena 409 | In spring 2024, Who are the instructors for course 15112? 410 | In summer 2024, When do Mini-5 Final Exams take place? 411 | Where should robots ideally exist according to the OUTDOOR paper? 412 | What number do all of the Chemical Engineering classes start with? 413 | In fall 2024, What is the deadline for Mini-1 add, audit & tuition adjustment drop (deadline 1)? 414 | The first authors of the paper NLPositionality: Characterizing Design Biases of Datasets and Models, are from which university/institute? 415 | How many people co-authored the paper Learning to Ask Questions for Zero-shot Dialogue State Tracking? 416 | What are the number of credits MCDS students must complete to graduate? 417 | Where will the The President’s Reception in honor of CMU’s Doctoral Candidates be held? 418 | How many Emmy Awards have alumni and current/former faculty won so far? 419 | What are the three concentrations in the MCDS program? 420 | How many de-biased training examples were used for fine-tuning the pre-trained model to significantly reduce the tendency to favor any gender? 421 | What is the full name of the conference where the paper NLPositionality: Characterizing Design Biases of Datasets and Models, got published? 422 | What is the office number for Joan Axelson? 423 | What sort of credentials are required to print something from an LTI printer? 424 | Which LTI faculty are involved in the framework tax paper? 425 | What is the full name of the metric used to evaluate the performance of the models on the Squad test set in the paper PROMPT2MODEL: Generating Deployable Models from Natural Language Instructions? 426 | Which conference is DIFFERENCE-MASKING published in? 427 | What is the name of the proposed recommendation model in the paper "Text Matching Improves Sequential Recommendation by Reducing Popularity Biases"? 428 | In spring 2024, What is the title of course 17416? 429 | For the MCDS degree do you need to do a capstone project? Yes/no 430 | How many turing awards recipients have been from Carnegie Mellon University? 431 | In summer 2024, When is the deadline for withdrawing from a Mini-5 course and receiving a withdrawal grade? 432 | What does SenteCon do to a given passage of text? 433 | What year was Exploration on HuBERT with Multiple Resolutions published? 434 | How many Chemical Engineering courses are going to be held in Summer 2024? 435 | When was the deadline for the MLT program applications? 436 | In spring 2025, Is there class on Martin Luther King Day? 437 | Which LTI prof co-authored the paper titled "Self-Refine: Iterative Refinement with Self-Feedback"? 438 | How many candidate documents were re-ranked using InPars-light compared to InPars? 439 | What is the room number for the Advanced Natural Language Processing course? 440 | How much increase in throughput (multi GPU setup) does SAMA showcase in large-scale meta learning benchmarks? 441 | What is the location of the courses that will be taught by Affara in Summer 2024? 442 | When should the guests be seated for the start of the student procession on May 12? 443 | Which year did The Kiltie Band began? 444 | In what year did Carnegie Tech became Carnegie Mellon University? 445 | What are the four common domains of websites in the WebArena environment? 446 | Which LLMs were used for validation of the SPAE method? 447 | What do the KALE vocabulary semantic concepts perform better than? 448 | In fall 2024, When is the Semester drop deadline and withdrawal grade assigned after this date? 449 | What does SHAP mean in SHAP-based Prediction of Mother's History of Depression to Understand the Influence on Child Behavior? 450 | When did amusing buggies like Delta Upsilon's "Fish" and Printing Management's Bathtub disappear? 451 | What type of requests will aligned text models refuse to answer? 452 | What is Yonatan Bisk's job title? 453 | Who taught human language for AI in the fall of 2023? 454 | How does CSurF address sparse lexicon-based retrieval? 455 | What does SPAE convert between? 456 | In the IWSLT 2023 paper titled "Evaluating Multilingual Speech Translation Under Realistic Conditions with Resegmentation and Terminology", authors provided a speech translation dataset covering ACL technical presentations. How many target languages were included in the final dataset? 457 | How many system submissions does GlobalBench currently cover? 458 | When was the Center for Machine Translation established at CMU? 459 | In the paper titled "WebArena: A Realistic Web Environment for Building Autonomous Agents", what is the success rate of the best performing GPT-3.5 model? 460 | At what conference was "The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Link between Phonemes and Facial Features" published at? 461 | What is the code URL for the case studies presented in the framework tax paper? 462 | In Prof. Fernando Diaz's paper on best-case retrieval evaluation, what is the name of the proposed metric for preference-based evaluation? 463 | Where was "End-to-End Speech Recognition: A Survey" published? 464 | In spring 2024, What is the title of course 17437? 465 | What number do all of the CFA Interdisciplinary classes start with? 466 | What are the names of the people from CMU who contributed to the paper RIVETER Measuring Power and Social Dynamics Between Entities? 467 | In which two task families does the document demonstrate MOSAIC's versatility? 468 | Who is the Office Manager for LTI who is listed in the LTI handbook? 469 | In spring 2024, What is the day and time of course 17445-A? 470 | What does SYNTACC stand for from Alexander Waibel's paper? 471 | In spring 2024, What is the title of course 15210? 472 | What does FACTORCL mean? 473 | What is the title for 11737? 474 | In fall 2024, When are Mid-Semester & Mini-1 grades due by 4 pm? 475 | Which paper proposed style radiance fields? 476 | What professor was the final author on the paper titled "Queer People are People First: Deconstructing Sexual Identity Stereotypes in Large Language Models"? 477 | Which invention by Professor Luis von Ahn was named Apple’s 2013 app of the year? 478 | When is the semester drop deadline for the Fall 2024 semester? 479 | What number do all of the Biological Sciences classes start with? 480 | According to the paper PWESUITE: Phonetic Word Embeddings and Tasks They Facilitate, what is the percentage accuracy for analogies, achieved by the count-based model on the evaluation suite? 481 | What is the PhD program director for LTI's phone number? 482 | Are there classes on April 11th, 2024? 483 | The MLT program is similar to the first two years of what other program? 484 | In fall 2023, What are the units for unit 02402? 485 | What are the two steps in the PaintSeg painting process? 486 | What is the course name/title for CMU 03128? 487 | Who is the Employment Processes Manager for LTI 488 | In spring 2024, What is the day and time of course 17413? 489 | Which LTI prof co-authored the "Speech collage: code-switched audio generation by collaging monolingual corpora" paper? 490 | What's the theme for the booths at Spring Carnival this year? 491 | In the paper "Improving Perceptual Quality, Intelligibility, and Acoustics on VoIP Platforms", which preprocessing methods were experimented with for audio data? 492 | Who are the instructors for the data science capstone (11632)? 493 | What is it fine-tuned on for creating StarCoder? 494 | What Psychology course in Summer 2024 will be offered at Doha, Qatar? 495 | In the paper "Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation, what is the reduction in word error rates achieved by the proposed models on CallHome? 496 | When is the annual MOBOT race? 497 | When does the Fall 2024 course registeration start for masters students? 498 | What is the tldr of the paper Multimodal Fusion Interactions: A Study of Human and Automatic Quantification? 499 | The first author of the paper Rethinking Voice-Face Correlation: A Geometry View is from which university? 500 | Fringe vehicles often start with which letter? 501 | What is the full name of the conference where the paper The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation, got published? 502 | In spring 2024, What is the day and time of course 17604-C? 503 | What is the end-to-end task success rate of the best GPT-4-based agent compared to human performance on the WebArena benchmark? 504 | When is the democracy day in 2024? 505 | When did the Fall Break end in 2023? 506 | In fall 2023, What is the title of course 05431? 507 | What is David Garlan's email address? 508 | What is the full name of the conference where the paper Riveter: Measuring Power and Social Dynamics Between Entities, got published? 509 | Which LTI prof co-authored the paper titled "Identification of Nonlinear Latent Hierarchical Models"? 510 | In spring 2024, What is the course number for Generative AI? 511 | What is the definition of dogwhistles? 512 | What are the two proposed subtasks for the DSTC11 automatic evaluation track? 513 | Who was the first dean of the School of Computer Science? 514 | When was Andrew project launched? 515 | In Fall 2023, where was 11737 taught? 516 | When are the grades due for the Fall 2024 semester? 517 | What types of prompts can PaintSeg be configured to work with? 518 | In spring 2024, When is the final deadline for withdrawing from a Mini-4 course? 519 | In spring 2024, Who are the instructors for course 17514? 520 | What is the corresponding author's emaill address for the SantaCoder paper? 521 | How many authors contributed to the work Understanding Political Polarisation using Language Models: A dataset and method? 522 | CAPTCHAs were invented by CMU researchers in 2000. What was the title of their paper? 523 | In spring 2024, What is the title of course 10301? 524 | When does the Fall 2024 course registeration start for doctoral students? 525 | In spring 2024, What is the title of course 10601? 526 | What type of models does the AV-SUPURB benchmark evaluate? 527 | Who is teaching the Multimodal Machine Learning course this semester? 528 | For LTI PhD students what room are the mailboxes and office supplies located in? 529 | Which LTI facultly were involved in the FLARE paper? 530 | What is the name of the proposed cross-modal fine-tuning framework in Graham's ICML 2023 work? 531 | Who taught Urban Design Methods and Theory in Fall 2023? 532 | What is the full name of the conference where the paper Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, got published? 533 | In summer 2024, What is the deadline for withdrawing from a Semester course and receiving a withdrawal grade? 534 | Is the GRE optional for the masters in language technologies application? Answer yes or no. 535 | In fall 2024, What is the last day of Mini-1 classes? 536 | Which two LTI professors co-authored the paper titled "Understanding Masked Autoencoders via Hierarchical Latent Variable Models"? 537 | In "Deriving Vocal Fold Oscillation Information from Recorded Voice Signals Using Models of Phonation", what is the proposed forward-backward algorithm? 538 | What tasks does ML-SUPERB consider? 539 | In the KALE paper, what evaluation metrics were reported on MSMARCO? 540 | Who all led the School of Computer Science in 1986? 541 | What is the Gates Hillman Complex at Carnegie Mellon University's 5 digit zip code? 542 | When was Campus Week discontinued and replaced with Spring Carnival? 543 | Who is CMU's first official mascot? 544 | What is the success rate of the baseline in real-world component of HomeRobot OVMM benchmark? 545 | In fall 2023, When is the deadline to drop a Mini-1 course with a withdrawal grade assigned? 546 | In spring 2024, What is the title of course 17634? 547 | Was Monica Harrison ever a member of the Carnegie Mellon Hall of Fame Selection Committee? 548 | What are the four categories of low-level acoustic descriptors used in the TAP loss? 549 | Did CMU found the world’s first university robotics department? Answer with True or False 550 | How does using random walks to estimate entity centrality on conversation entity graphs affect answer passage ranking? 551 | How many people from CMU co-authored the paper Multi-lingual and Multi-cultural Figurative Language Understanding? 552 | In fall 2024, What is the deadline for Mini-1 voucher election? 553 | What is the full name of the conference where the paper An Approach to Ontological Learning from Weak Labels, got published? 554 | What is Carolyn Penstein Rose's fax number? 555 | According to authors of the FLARE paper, what is one limitation of existing retrieval augmented LMs? 556 | In Yonatan Bisk's MOSAIC paper, what does MOSAIC stand for? 557 | What is Martial Herbert's one word title, this is not the one with the word professor in it? 558 | FLARE method from Jiang et al., was evaluated on four knowledge-intensive tasks. What are these tasks? 559 | Where can the code of OpenMatch be found? 560 | In spring 2024, Who are the instructors for course 15150? 561 | What is the role of the mapping network in the proposed model in "Generating Images with Multimodal Language Models"? 562 | Who controls the vehicles via steering and braking systems in a buggy? 563 | On which benchmarks did the authors test FiT5's performance? 564 | Are there any auditions to join The Kiltie Band? 565 | Who is the current Associate Director of Athletics, Recreational Programs? 566 | What can modeling the conversation with entity graphs be used for? 567 | Who is ther first author of the paper Cross-Modal Fine-Tuning: Align then Refine? 568 | In summer 2024, When do the May Mini-5 and Semester classes begin? 569 | What is the MOS-Q achieved by the HF-GAN on the VoxCeleb test set? 570 | Which country does LTI have a special PhD program with? 571 | How many teams participated in the IWSLT 2023 shared tasks? 572 | What is the DAE achieved by the CRL-COM (D) system from the paper Improving Factuality of Abstractive Summarization via Contrastive Reward Learning, on the XSUM dataset? 573 | Which fraternity entered a keg of beer mounted on four wheels in 1960 buggy? 574 | When was the buggy course laid out in lanes for the first time? 575 | In summer 2024, When is Juneteenth observed and what is the University's policy on classes? 576 | What is the DialDoc 2023 shared task about? 577 | What LTI professor was on "KIT’s Multilingual Speech Translation System for IWSLT 2023"? 578 | In "CONVOIFILTER: A CASE STUDY OF DOING COCKTAIL PARTY SPEECH RECOGNITION" what 3 letter metric was reduced from 80% to 26.4%? 579 | Who is the PhD Program Director for the LTI PhD degree? 580 | What is the title of LTI's text mining course? 581 | What is the procedure whereby one pusher finishes pushing a buggy and the next pusher in sequence starts to push that same buggy? 582 | Where do the rehearsals for The Kiltie Band take place? 583 | What are the multimodal capabilities of the proposed model in "Generating Images with Multimodal Language Models"? 584 | The MOSAIC framework from the paper titled "MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Perception" was evaluated on two task families. What are these task families? 585 | Scotty was officially accepted as CMU's first mascot in which year? 586 | What does KALE use to convert dense representations into a sparse set? 587 | According to the MSAII handbook, who is the associate dean for masters programs? 588 | In fall 2023, What is the last day of classes for Mini-2, Semester, and Mini-2? 589 | What number do all of the LTI classes start with? 590 | Who is the last author on WebArena? 591 | In spring 2024, When do Mini-4 faculty course evaluations close? 592 | In the IWSLT 2023 paper titled "Evaluating Multilingual Speech Translation Under Realistic Conditions with Resegmentation and Terminology", authors tackle the task of technical speech translation. What evaluation metrics were reported for translation? 593 | Which professor from LTI worked on the paper Advancing Regular Language Reasoning in Linear Recurrent Neural Networks? 594 | How many credits is Linguistics Lab worth? 595 | How do buggies move forward in the beginning of the race? 596 | In the MCDS degree, what are the two names of the tracks you can take in your plan of study? One of these is a 16 month degree and the other is a 20 month degree. Answer with the names of the two tracks with an 'and' in between. 597 | What is the title of the ethics course offered at LTI? 598 | What is ESPnet-ST-v2? 599 | What is the target duration of the LTI PhD program in years? 600 | How many courses are offered by BXA Intercollege Degree Programs in Spring 2024 (exclude the BXA Studio courses)? 601 | How does SenteCon affect predictive performance on downstream tasks? 602 | What are some of the under-served languages currently identified by GlobalBench? 603 | What is the full name of the conference where the paper GameQA: Gamified Mobile App Platform for Building Multiple-Domain Question-Answering Datasets, got published? 604 | In fall 2023, When is unit 02613 on Mondays, Wednesdays, and Fridays? 605 | Who were the instructors for 11667? 606 | Which LTI faculty were involved in the WebArena paper? 607 | In spring 2024, What is the day and time of course 17645-F? 608 | Who are all of the tenure-track associate professors in LTI? 609 | What are the components of HomeRobot? 610 | In fall 2023, What is the title of course 05360? 611 | In 2011, an IBM computer defeated human champions on the “Jeopardy!” game show. What was the name of this computer? 612 | What is the semantic notion used as a case study in "Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity"? 613 | If you wanted to take Arts & Community Development in Fall 2023, what were the course numbers of the courses offered? 614 | What are the 4 common MCDS core courses? List them in the following format: course number - title; course number - title; ... 615 | What number do all of the Computational Biology classes start with? 616 | What's the cost in us dollars per program for the masters degrees in language technologies if you submit after the early deadline? 617 | In fall 2023, When does unit 02601 take place on Fridays? 618 | What is the name of the proposed method that extends WavLM's joint prediction and denoising to 40k hours of data across 136 languages? 619 | What year was Fully Unsupervised Topic Clustering of Unlabelled Spoken Audio Using Self-Supervised Representation Learning and Topic Model published? 620 | Which instructors co-taught On-Device Machine Learning last fall? 621 | Which class did Shinji Watanabe teach in Fall 2024? 622 | In spring 2024, What is the title of course 17537? 623 | Are guests allowed to play in the tennis court? 624 | In spring 2024, What is the title of course 15050? 625 | When was the first Interfraternity Sweepstakes Race held? 626 | How many people co-authored the paper COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements? 627 | In the CSurF paper, what evaluation metrics were reported for MSMARCO? 628 | Where is Multimodal Fusion Interactions: A Study of Human and Automatic Quantification published? 629 | What version of ChatGPT is used to extract facts in the FacTool paper? 630 | Where is FACTORCL published? 631 | What is the average performance improvement of Prompt2Model over gpt-3.5-turbo LLM? 632 | What determines a buggy's aerodynamic characteristics? 633 | How many times larger was the monoT5-3B ranker compared to the MiniLM ranker used in the InPars-Light study? 634 | What are the benefits of using IPA over fine-tuning? 635 | What is StarCoderBase trained on? 636 | Who is the current head coach of men's basketball? 637 | In spring 2024, What is the deadline for adding or dropping a Mini-3 course with tuition adjustment? 638 | What was the total number of submissions for the IWSLT 2023 shared tasks? 639 | When does the Spring 2025 course registeration start for sophomores? 640 | At what conference was Exploration on HuBERT with Multiple Resolutions published? 641 | How many credits is Human Language for AI worth? 642 | Who is the LTI director? 643 | When did buggy rules change to include a permanent driver and four pushers along the course? 644 | In "Pengi: An Audio Language Model for Audio Tasks," on how many downstream tasks is the model evaluated on? 645 | In summer 2024, When is the deadline for Mini-5 vouchers? 646 | What is the name of the initiative introduced to track and incentivize the global development of equitable language technology? 647 | What was David A. Tepper School of Business's original name? 648 | In spring 2024, How many units is course 10605? 649 | Who propels a buggy via a pushbar along one of the five hills of the buggy course? 650 | Did Andy Warhol graduate from CMU? Answer with True or False 651 | What tasks does ESPnet-ST-v2 support? 652 | In the Paaploss paper, what does the neural network estimator developed in the study predict? 653 | When did Kappa Kappa Gamma enter the first all-women’s team in buggy history? 654 | In fall 2023, What is the title of course 05318? 655 | When does the Spring Carnival start in Spring 2025 semester? 656 | Which languages are included in the dataset released by "Multi-lingual and Multi-cultural Figurative Language Understanding"? 657 | Which faculty were involved in the CSurF paper? 658 | In fall 2023, Who are the instructors for course 05380? 659 | What kind of vulnerabilities do diffusion models have according to the paper "Extracting Training Data from Diffusion Models"? 660 | In fall, who were the instructors for the Introduction to Deep Learning course at LTI? 661 | In spring 2024, What is the title of course 17356? 662 | How many courses is Sindi teaching in Spring 2024? 663 | Which two institutes merged together to form the current day Carnegie Mellon University? 664 | Who is the current assistnat coach of women's basketball? 665 | Who is teaching the question answering course at LTI? 666 | Which LTI class is offered in Kigali, Rwanda? 667 | How many languages does ML-SUPERB cover? 668 | Who is teaching the "Ethics and Decision Making in Architecture" in Spring 2024? 669 | What are some of the metrics incorporated in Pentathlon for efficiency evaluation? 670 | How many parameters does the chain-of-skills model have? 671 | What LTI professor co-authored "CONVOIFILTER: A CASE STUDY OF DOING COCKTAIL PARTY SPEECH RECOGNITION"? 672 | What is the GitHub URL where MultiViz is available? 673 | What number do all of the Chemistry classes start with? 674 | How many authors from FACTORCL are from Carnegie Mellon University? 675 | In spring 2024, When are Mid-Semester and Mini-3 grades due? 676 | In the BASS paper from Interspeech 2023, what is the solution proposed to address the issue with training end-to-end speech summarization models on very large inputs? 677 | What is the mechanism that is critical to language learning in young children? 678 | How many authors are listed on the SPAE paper? 679 | What is the BERTScore achieved by BASS-adapt on the How-2 test set? 680 | What was the model used for early buggies in the 1930s? 681 | Which two LLMs were explored in the SPAE paper? 682 | What is the contact number of the Fitness Operations Manager? 683 | How many components or phases are in the MultiBench toolkit pipeline? 684 | How many scientific challenges in spoken language translation did the IWSLT 2023 shared tasks address? 685 | Who is the first author of "Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System"? 686 | According to ChatGPT MT, what is the most important feature in determining ChatGPT's relative ability to translate a language? 687 | What is the proposed approach in the paper "Rethinking Voice-Face Correlation: A Geometry View"? 688 | Which datasets were used in the evaluation of KALE method? 689 | Who is the main instructor for the search engines course? 690 | In spring 2024, When do Mini-4 classes begin? 691 | Has Professor Carolyn Rose worked on Automatic Essay Scoring? 692 | In the paper "Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations", what does ILL leverage for modeling the imprecise label information? 693 | What is the target duration of the LTI PhD program in months? 694 | In spring 2024, What is the voucher deadline for Mini-3? 695 | For additional information about the LT concentration for undergraduates, who should you contact? 696 | In spring 2024, When is the final deadline for withdrawing from a Mini-3 course? 697 | In which year did The Kiltie Band have their first official performance? 698 | What is the name of the open-scientific collaboration working on the responsible development of Large Language Models for Code? 699 | Which shared task does "Language-Agnostic Transformers and Assessing ChatGPT-Based Query Rewriting for Multilingual Document-Grounded QA" focus on? 700 | In the paper "A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech", what type of data did the authors use to train their TTS systems? 701 | In spring 2024, When is Martin Luther King Day observed? 702 | Which LTI faculty members are authors on the WebArena paper? 703 | Where is advanced NLP taught this semester? 704 | In spring 2024, When are final examinations for the semester and Mini-4? 705 | What is the outer structure or covering of a buggy called? 706 | In fall 2023, When are the final grades due for the semester? 707 | What is one limitation of lexical exact-match systems? 708 | In fall 2023, Who is the instructor for unit 02261 on Wednesdays? 709 | In which types of benchmarks has SoftMatch shown substantial improvements? 710 | In spring 2024, Who is the instructor for Advanced Deep Learning? 711 | Where is the President's Graduates Toast for bachelor's students going to be held? 712 | What does CLIP stand for? 713 | Which independent organization set a course record of 2:06.20 in 1988 buggy? 714 | Does LTI offer a course on large language models? 715 | What's the title for 11700? 716 | In the study "On the Interactions of Structural Constraints and Data Resources for Structured Prediction", what are the three structured prediction tasks evaluated? 717 | In Justine Cassell's recent SIGDIAL paper, what feature does the study find to have a significant impact on hedge prediction? 718 | What site can I visit for more information about CMU's COVID policies? 719 | What is the name of the author of the paper The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Linkbetween Phonemes and Facial Features who is from Max Planck Institute? 720 | In which semester do the Buggy Races happen? 721 | When did Andrew Carnegie emigrate from Scotland to Pittsburgh? 722 | What issue is reciprocal rank found to have? 723 | What is the proposed method for grounding pre-trained text-only language models to the visual domain? 724 | In fall 2023, What is the course title for unit 02090? 725 | How many months is the shorter track of the MIIS program? 726 | Is the university open on January 15th 2024? 727 | What are some diffusion models mentioned in the document, "Extracting Training Data from Diffusion Models"? 728 | In fall 2023, Who are the instructors for course 05430? 729 | In fall 2024, When are Fall Deans' Lists Posted? 730 | In the ICTIR paper, what does KALE stand for? 731 | What time in eastern time was the final application deadline for the PhD program in language and information technology? Give your answer in 12h time format with either an am or pm label. 732 | Which LTI faculty was a contributor on the HomeRobot paper? 733 | When did CMU get its first IBM 650 computer? 734 | In the Plan, Eliminate and Track paper, what percentage gain did the proposed framework achieve over the state-of-the-art? 735 | How many scenes were included in the HomeRobot OVMM benchmark? 736 | In the paper "Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization, what are the three unseen tasks investigated for Whisper model? 737 | When was the last day of classes for the Fall 2023 semester? 738 | What is reciprocal rank used to measure? 739 | How many authors are on SantaCoder paper? 740 | In fall 2024, What is the deadline for Mini-1 Pass/no pass & withdrawal? 741 | What number do all of the Integrated Innovation Institute classes start with in Summer 2024? 742 | What number do all of the Civil & Environmental Engineering classes start with? 743 | In spring 2025, When do the first day of classes for the winter semester take place? 744 | How many authors contributed to the paper CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code? 745 | Which section of the freeroll portion of the buggy course do buggies make a sharp right-hand turn? 746 | What was H. John Heinz III College previously called? 747 | What is the full name of the conference where the paper Transformed Protoform Reconstruction, got published? 748 | When is the buggy bash at the spring carnival? 749 | When will the Spring Break end in 2024? 750 | What are the two innovative designs of StyleRF? 751 | What is the technique used by Pengi to leverage Transfer Learning? 752 | In the paper "Quantifying & Modeling Feature Interactions: An Information Decomposition Framework", in which areas are the real-world applicability of the proposed approach demonstrated? 753 | In the paper "Accelerating Diffusion-based Combinatorial Optimization Solvers by Progressive Distillation", what is the performance degradation of the progressively distilled model on the TSP-50 dataset? 754 | In fall 2023, Who is the instructor for course 05432? 755 | When will the Spring Break start in 2024? 756 | In fall 2023, What is the course title for unit 02801? 757 | What does the MLT program prepare students for? 758 | How many authors does the WebArena paper have? 759 | How many modalities does the MultiBench benchmark include? 760 | How many datasets does the MultiBench benchmark include? 761 | What computation is offloaded to a k-nearest-neighbor (kNN) index in the Unlimiformer approach? 762 | Is chalk permitted in the Fitness Centre at the Jared L. Cohon University Center? 763 | What is the novel architecture introduced in the paper "Efficient Sequence Transduction by Jointly Predicting Tokens and Durations"? 764 | What is the effect of training speakers with a highly weighted ToM listener component? 765 | When are the Spring 2024 grades due for graduating students? 766 | What dataset does the BASS paper by Bhiksha Raj's group evaluate on? 767 | What are the three aspects assessed by the holistic evaluation in MultiZoo & MultiBench? 768 | What LTI professor was on "SYNTACC : Synthesizing Multi-Accent Speech By Weight Factorization"? 769 | What is the contact number of the Director of Sports Medicine? 770 | When were Simon and Newell of CMU awarded the Turing award? 771 | --------------------------------------------------------------------------------