├── README.md ├── images └── RLHF.png ├── tutorials ├── RLHF_with_Custom_Datasets.ipynb ├── ls_input_data.json └── ls_output_data.json └── utils ├── __init__.py └── elo.py /README.md: -------------------------------------------------------------------------------- 1 | # Aligning ML Models with Human Feedback 2 | 3 | - [What this repository contains:](#what-this-repository-contains) 4 | - [This repo may be of interest to you if:](#this-repo-may-be-of-interest-to-you-if) 5 | - [If you have any questions about this repo, or need a hand:](#if-you-have-any-questions-about-this-repo-or-need-a-hand) 6 | - [Establishing Supervised Model Baseline](#establishing-supervised-model-baseline) 7 | - [Gathering and Incorporating Human Feedback](#gathering-and-incorporating-human-feedback) 8 | - [Training and Assessing the Final Model with Reinforcement Learning](#training-and-assessing-the-final-model-with-reinforcement-learning) 9 | 10 | ![Stages of model alignment](./images/RLHF.png) 11 | 12 | ## What this repository contains: 13 | This repository contains a collection of tutorials, best practices, and references for developers, data scientists, and machine learning professionals of all skill levels. 14 | 15 | ### This repo may be of interest to you if: 16 | - You've attended the PyData Berlin Talk by [Nikolai Liubimov](https://www.linkedin.com/in/liubimov/) and [Erin Mikail Staples](https://www.linkedin.com/in/erinmikail/) 17 | - You're interested in data-centric model development 18 | - You're using [Label Studio](https://labelstud.io) to improve your ML process and models. 19 | 20 | ### If you have any questions about this repo, or need a hand: 21 | - This repo is maintained by the [Label Studio](https://labelstud.io) team. 22 | - Reach out to the [Label Studio Community Team](mailto:community@labelstud.io) or open an issue with any questions. 23 | - Join the [Label Studio Community Slack](https://slack.labelstud.io/?source=github-RLHF) 24 | - Check out more resources around Human in the Loop / RHLF in our [Awesome List](https://github.com/heartexlabs/awesome-human-in-the-loop/tree/master) 25 | 26 | 27 | ## Establishing Supervised Model Baseline 28 | In this step, we collect labeled text data to train an initial Large Language Model (LLM), focusing on task-specific performance improvements. This stage involves gathering instructions and responses to adapt the base model to a broad range of tasks, enhancing its ability to generate accurate and contextually relevant responses. 29 | Typically this step involves: 30 | - Selecting baseline Foundational Model (FM) that can perform fairly well on general tasks (like [GPT2](https://huggingface.co/gpt2) or [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6b)) 31 | - Generate dataset of pairs prompt input followed by response. You can manually label dataset or generate the data like it is [provided in the example](https://github.com/tatsu-lab/stanford_alpaca) 32 | - Peform supervised model finetuning. 33 | 34 | ## Gathering and Incorporating Human Feedback 35 | This stage involves collecting comparison data to establish human preferences for the responses generated by the supervised model. By ranking multiple responses based on quality, we can train a reward model that effectively captures human preferences. This reward model plays a crucial role in reinforcement learning, optimizing the performance of the fine-tuned foundational model. 36 | 37 | [Gathering Human Feedback Tutorial](tutorials/RLHF_with_Custom_Datasets.ipynb) - This Jupyter Notebook tutorial will guide you through the process of collecting comparison data, establishing human preferences, and incorporating this feedback into the reward model training. 38 | 39 | ## Training and Assessing the Final Model with Reinforcement Learning 40 | The training stage is a challenging process, and the final model assessment is a critical component in evaluating your model's quality. It is essential to determine whether the model adheres to the provided instructions, avoids biases, and maintains a high standard of performance. 41 | -------------------------------------------------------------------------------- /images/RLHF.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HumanSignal/RLHF/cc3fa781049fee796ace5a22c7ae51abfe87fb67/images/RLHF.png -------------------------------------------------------------------------------- /tutorials/ls_input_data.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "prompt": "What is the latest news on the stock market?", 4 | "answer1": "Let the spotlight shine on something big, something that matters. If you haven't picked up on this year's stocks market (which will likely be over for a few months), then you may be missing", 5 | "answer2": "Here are 5 things you should know about the market's stock performance this past week.\n\nMARCH 1 UPDATE\n\nA few weeks ago, The Wall Street Journal said that U." 6 | }, 7 | { 8 | "prompt": "What is the current state of the economy?", 9 | "answer1": "I'm seeing some of the data back on here, about how much we need to increase our business expenditures. In a recent report, the Congressional Budget Office's Bureau of Economic Analysis estimated that the", 10 | "answer2": "I think we've seen some progress on this. When I came to politics, I heard a lot about how you talk about you and how you were \"taking the fight to Wall Street.\" I" 11 | }, 12 | { 13 | "prompt": "What are the latest developments in technology?", 14 | "answer1": "Will it ever be able to be incorporated into existing devices?\n\nThis survey started as a conversation about how technology might help people better understand their own personal data. When there's a large body of data,", 15 | "answer2": "This has been the most difficult aspect of my career, due to multiple sources I have already spoken with, the amount of people who knew and had close relationships with the developers and the way they dealt with" 16 | }, 17 | { 18 | "prompt": "What is the political situation in the Middle East?", 19 | "answer1": "The Arab League, of course, is very concerned about what's going on in the Middle East, and for that, the West is not necessarily going to keep to the model they have,", 20 | "answer2": "On the one hand, we are dealing here with a crisis of the way, a period of uncertainty. But on the other hand, this crisis comes with a price. Our relationship with some" 21 | }, 22 | { 23 | "prompt": "What are the latest trends in fashion and beauty?", 24 | "answer1": "", 25 | "answer2": "We believe the new trend we tend to see is the beauty trend and the rise of celebrity. A huge amount of young men are getting into fashion because they like the work and the money and" 26 | }, 27 | { 28 | "prompt": "What are the top travel destinations for this year?", 29 | "answer1": "The travel categories are not broken up by country, but are rather organized into three segments: Australia (where your passport is valid), Germany (where it is valid), and Taiwan, where it", 30 | "answer2": "1. LA, California 2. Phoenix, Arizona 3. Toronto, Ontario 4. Las Vegas, Nevada 5. Atlanta, Georgia 6. New York, New York\n\nIf you found" 31 | }, 32 | { 33 | "prompt": "What are some healthy recipes for a vegan diet?", 34 | "answer1": "This post is really long, so I wanted to stick to a quick little guide for what you need, so please refrain from commenting!\n\nIf you really want to know what's going", 35 | "answer2": "1. Take a lot of water.\n\n2. Mix flour, vegetable oil, almond extract and cayenne pepper together with a mix bit more lightly. Mix well." 36 | }, 37 | { 38 | "prompt": "What are the most important events happening in the world today?", 39 | "answer1": "1. The fall of the Roman Empire\n\nOne of the most important events in humanity's history occurred the fall of Rome on December 31, 1446 and during that time the", 40 | "answer2": "A global financial crisis triggered massive international recession, global economic turmoil, unprecedented financial crises in the 1990s, and a global financial meltdown, resulting in a global financial meltdown of unparalleled systemic" 41 | }, 42 | { 43 | "prompt": "What are some tips for improving mental health?", 44 | "answer1": "One of the many things that people are good at and are working hard on is helping themselves and others get better. It's helpful to know your options.\n\nIn the past, mental health", 45 | "answer2": "1) Get out there and have a conversation. Maybe some people won't talk. Maybe you have friends who don't agree with your mental health. The best thing is if they have experienced anything" 46 | }, 47 | { 48 | "prompt": "What are the best ways to save money for retirement?", 49 | "answer1": "What types of investments need to have their own annual accounts?\n\nDo you find it essential to set up a account plan to help you make your own decisions on what to invest in", 50 | "answer2": "First, there is a lot of great information online about how to pay for your retirement. This post has lots of links, as well as some great tips that will help you decide if" 51 | }, 52 | { 53 | "prompt": "What are some popular new books or movies?", 54 | "answer1": "A very low number of books are sold digitally (as well as on social media sites) but most TV and radio shows (including many documentaries on film) have been available for a couple of years", 55 | "answer2": "You've got to make them in real time. Just keep repeating the idea -- if a book gets more hits then it's getting more books to make. Let yourself get a few books to read by the" 56 | }, 57 | { 58 | "prompt": "What are some effective ways to reduce stress?", 59 | "answer1": "(What you'll need\n\nIn short - Stress is something you're all about because of the actions and thinking that your system takes. The easiest way to reduce stress is to avoid it, take steps", 60 | "answer2": "Well, it's easy to find those simple, but powerful ways that stress can actually work well in the face of adversity \u2014 things like that:\n\nWhen I was in high school, I" 61 | }, 62 | { 63 | "prompt": "What are the latest developments in artificial intelligence?", 64 | "answer1": "How are the companies in the field working to improve our understanding about the nature of information?\n\nIn our research, we focused on research and development by researchers, especially those who may be more", 65 | "answer2": "On the frontline of AI and beyond, Artificial Intelligence has been on the offensive for decades. Some of the most important advances in this field were first introduced in the 1960s and have been observed" 66 | }, 67 | { 68 | "prompt": "What are some top-rated restaurants in your city?", 69 | "answer1": "In order to choose the right restaurant, we rely on the following criteria:\n\nYour area's reputation;\n\nYour staff's skills;\n\nIn all cases, you'll", 70 | "answer2": "Let us know in the comments!" 71 | }, 72 | { 73 | "prompt": "What are the best ways to stay fit and healthy?", 74 | "answer1": "In an article of the \"What Are You Doing at Work?\" category, FitIQ explained how to keep up with your body when it comes to exercise:\n\n\"To help you", 75 | "answer2": "Find out with our simple answers, designed by a fitness enthusiast who doesn't hate anything but fitness!\n\nIf you know what exercises you'll need, you should try. Here on the Fit" 76 | }, 77 | { 78 | "prompt": "What are some tips for successful entrepreneurship?", 79 | "answer1": "When your own business is first entering the market you might receive a few tips, especially if you are setting a goal.\n\nWhat are some tips for successful entrepreneurship? The sooner your goal is reached", 80 | "answer2": "1. Avoid marketing scams\n\nIt has been said that you should NEVER advertise in print. It's pretty obvious that you need a professional if you don't want to get your clients to write about" 81 | }, 82 | { 83 | "prompt": "What are some effective ways to improve productivity?", 84 | "answer1": "What makes a good productivity manager? It is a decision made from within your organization that is made more often than not based on your experience rather than simply based on your own ideas, which are generally", 85 | "answer2": "The productivity gains of the following three items can help people gain more productivity:\n\nIncreased focus on the task at hand. By increasing the amount of time in which they spend focused thinking about the" 86 | }, 87 | { 88 | "prompt": "What are the latest developments in climate change research?", 89 | "answer1": "Read more\n\nDavid Brown, chairman of the IPCC, said some recent changes to the science, such as its analysis of the role of ocean warming due to human greenhouse effects, \"can only be", 90 | "answer2": "Are we going to change that? What is the link between the various science communities being informed about the climate crisis? What are the causes of the problems we're facing? What needs to be done to" 91 | }, 92 | { 93 | "prompt": "What are some top-rated TV shows or movies on streaming services?", 94 | "answer1": "We love sports sports, but we're always looking for more good sports programming. It's usually a combination of great sports programming and great entertainment. The best sports programming has", 95 | "answer2": "Comedy Central (CBS)\n\nComedy Central is the only cable channel in North America where you can catch up on current episode time and date from various networks." 96 | }, 97 | { 98 | "prompt": "What are some fun activities to do on weekends?", 99 | "answer1": "I work in the morning in the coffee kitchen, watching as girls and boys come to pick up their tea.\n\nWhere can I get my supplies?\n\nThere is no direct bus", 100 | "answer2": "Share them with us at Facebook.com/#!/TravisBurdette @WUWTBruz\n\nRead more: [FULL LATEST ARTICLE]" 101 | }, 102 | { 103 | "prompt": "What are some effective ways to manage time and prioritize tasks?", 104 | "answer1": "Do you have any tips on how that might help with scheduling and/or time management in a business? If so, what strategies, actions or tips do you use? Let us know in", 105 | "answer2": "Where do we start, etc.? Where do we begin, etc.?\n\nIt's a complex question, so there are a couple important things to consider:\n\nTime\n\nTime" 106 | }, 107 | { 108 | "prompt": "What are the latest trends in home decor and design?", 109 | "answer1": "Let us know.\n\n\nThe Best: For its long-standing mission of supporting home furnishings, The Woodlands is home furnishings creator, design editor for the website. Woodland's", 110 | "answer2": "" 111 | }, 112 | { 113 | "prompt": "What are the best ways to develop a successful career?", 114 | "answer1": "Many career paths consist of an individualization of a career process. There are two approaches which may help you to get there\u2026\n\nYou can pick out specific opportunities or take an optional", 115 | "answer2": "And when can I start building more? The process starts with a few basic questions. Do you want to learn anything new? If not, what do you prefer to learn or do when you do" 116 | }, 117 | { 118 | "prompt": "What are some popular new products or gadgets?", 119 | "answer1": "There has been a long talk at CES 2014 that the future of devices is in the hands of new developers. The point being that, as in many industries, one of the primary benefits from being", 120 | "answer2": "There are some popular new devices that we are working on that bring something new to the Tablets. For example, as mentioned previously:\n\n\nIn order to connect tablets to the mobile gaming app" 121 | }, 122 | { 123 | "prompt": "What are some effective ways to improve communication skills?", 124 | "answer1": "The following is a list of examples of a general topic that all should address. Be patient when speaking about individual communication skills.\n\nPractice Your Own Businesses\n\nYour career and", 125 | "answer2": "How to: Learn to focus on your communication when the world around you is constantly shifting, and at times you are constantly learning so you don't need to stop.\n\nOne or more" 126 | }, 127 | { 128 | "prompt": "What are some tips for successful relationships?", 129 | "answer1": "One of the most important things you should know when choosing to have a relationship with someone is that you should never be a \"couple who just loves each other for a day\".\n\nIf you have an", 130 | "answer2": "We all know that men are often the best men, but there's no doubt that women are most successful when they're good at their jobs. The job that pays the highest paying job can give you" 131 | }, 132 | { 133 | "prompt": "What are the latest developments in space exploration?", 134 | "answer1": "The U.S. has spent more than $4 trillion in space exploration spending over the last 40 years, which includes the use of commercial spacecraft, space stations and space shuttles. Space", 135 | "answer2": "I am thrilled that the ISS is on the move to the U.S., along with the SES and ISS. We are doing the most exciting things possible using space. NASA has shown that" 136 | }, 137 | { 138 | "prompt": "What are some top-rated online courses or certifications?", 139 | "answer1": "The major online programs for a bachelor's degree typically provide a degree from a private institution\u2014usually in finance. Most programs, especially those funded by federal funding, offer only one online", 140 | "answer2": "If you want to learn the business of finance, you need to have a good one.\n\nHere are seven of the best things you can do in the finance business in 6 months." 141 | }, 142 | { 143 | "prompt": "What are some effective ways to improve public speaking skills?", 144 | "answer1": "There are many ways to improve public speaking skills. To start, ask yourself when you have used a word or phrase. Can I use it for the context you like better than your preferred", 145 | "answer2": "There are many ways an entrepreneur could improve their public speaking skills. For example, start early with a good public speaking strategy that includes:\n\nUsing a wide range of media" 146 | }, 147 | { 148 | "prompt": "What are the latest trends in digital marketing?", 149 | "answer1": "\"I'm talking about the business models\"\n\nFor most industries, media consumption is the main drivers of advertising revenue. However, not many companies focus just on targeting new audiences. Advertis", 150 | "answer2": "The data is based on hundreds of studies and several thousand individual surveys. Here's a brief update on today:\n\nWhen it comes to digital marketing, it seems that consumers are increasingly consuming data from a" 151 | }, 152 | { 153 | "prompt": "What are some fun and creative DIY projects?", 154 | "answer1": "Check out this post.", 155 | "answer2": "Let's start with the DIY project of creating a simple 3D printed wooden bar. A bar is created by placing a wooden handle and a wire, then placing a wire in the bottom of the" 156 | }, 157 | { 158 | "prompt": "What are some effective ways to improve leadership skills?", 159 | "answer1": "1. Encourage people to stay accountable to themselves\n\nWhen leaders behave themselves in a negative way, people often don't want to do things the way they wanted to do, they want", 160 | "answer2": "To get to the bottom of leadership skills questions, in this post, we're going to use the most basic definitions for leadership in a career path (based on my personal experience):" 161 | } 162 | ] -------------------------------------------------------------------------------- /tutorials/ls_output_data.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "id": 401635, 4 | "annotations": [ 5 | { 6 | "id": 68045, 7 | "completed_by": 2, 8 | "result": [ 9 | { 10 | "value": { 11 | "selected": "left" 12 | }, 13 | "id": "Sba4F4tuVh", 14 | "from_name": "comparison", 15 | "to_name": "comparison", 16 | "type": "pairwise", 17 | "origin": "manual" 18 | } 19 | ], 20 | "was_cancelled": false, 21 | "ground_truth": false, 22 | "created_at": "2023-04-17T21:54:20.457543Z", 23 | "updated_at": "2023-04-17T21:54:20.457580Z", 24 | "lead_time": 12.51, 25 | "prediction": {}, 26 | "result_count": 0, 27 | "unique_id": "80f751c6-c26a-47ef-9eea-8208d33a172a", 28 | "last_action": null, 29 | "task": 401635, 30 | "project": 183, 31 | "updated_by": 2, 32 | "parent_prediction": null, 33 | "parent_annotation": null, 34 | "last_created_by": null 35 | } 36 | ], 37 | "file_upload": "65af2932-ls_input_data.json", 38 | "drafts": [], 39 | "predictions": [], 40 | "data": { 41 | "prompt": "What is the latest news on the stock market?", 42 | "answer1": "Let the spotlight shine on something big, something that matters. If you haven't picked up on this year's stocks market (which will likely be over for a few months), then you may be missing", 43 | "answer2": "Here are 5 things you should know about the market's stock performance this past week.\n\nMARCH 1 UPDATE\n\nA few weeks ago, The Wall Street Journal said that U." 44 | }, 45 | "meta": {}, 46 | "created_at": "2023-04-17T21:53:56.689509Z", 47 | "updated_at": "2023-04-17T21:54:20.494417Z", 48 | "inner_id": 1, 49 | "total_annotations": 1, 50 | "cancelled_annotations": 0, 51 | "total_predictions": 0, 52 | "comment_count": 0, 53 | "unresolved_comment_count": 0, 54 | "last_comment_updated_at": null, 55 | "project": 183, 56 | "updated_by": 2, 57 | "comment_authors": [] 58 | }, 59 | { 60 | "id": 401636, 61 | "annotations": [ 62 | { 63 | "id": 68046, 64 | "completed_by": 2, 65 | "result": [ 66 | { 67 | "value": { 68 | "selected": "left" 69 | }, 70 | "id": "ctzsT_2BwS", 71 | "from_name": "comparison", 72 | "to_name": "comparison", 73 | "type": "pairwise", 74 | "origin": "manual" 75 | } 76 | ], 77 | "was_cancelled": false, 78 | "ground_truth": false, 79 | "created_at": "2023-04-17T21:54:25.905373Z", 80 | "updated_at": "2023-04-17T21:54:25.905420Z", 81 | "lead_time": 5.228, 82 | "prediction": {}, 83 | "result_count": 0, 84 | "unique_id": "d89dbfb4-a81a-4914-8a1f-adeda6143633", 85 | "last_action": null, 86 | "task": 401636, 87 | "project": 183, 88 | "updated_by": 2, 89 | "parent_prediction": null, 90 | "parent_annotation": null, 91 | "last_created_by": null 92 | } 93 | ], 94 | "file_upload": "65af2932-ls_input_data.json", 95 | "drafts": [], 96 | "predictions": [], 97 | "data": { 98 | "prompt": "What is the current state of the economy?", 99 | "answer1": "I'm seeing some of the data back on here, about how much we need to increase our business expenditures. In a recent report, the Congressional Budget Office's Bureau of Economic Analysis estimated that the", 100 | "answer2": "I think we've seen some progress on this. When I came to politics, I heard a lot about how you talk about you and how you were \"taking the fight to Wall Street.\" I" 101 | }, 102 | "meta": {}, 103 | "created_at": "2023-04-17T21:53:56.689600Z", 104 | "updated_at": "2023-04-17T21:54:25.943155Z", 105 | "inner_id": 2, 106 | "total_annotations": 1, 107 | "cancelled_annotations": 0, 108 | "total_predictions": 0, 109 | "comment_count": 0, 110 | "unresolved_comment_count": 0, 111 | "last_comment_updated_at": null, 112 | "project": 183, 113 | "updated_by": 2, 114 | "comment_authors": [] 115 | }, 116 | { 117 | "id": 401637, 118 | "annotations": [ 119 | { 120 | "id": 68047, 121 | "completed_by": 2, 122 | "result": [ 123 | { 124 | "value": { 125 | "selected": "left" 126 | }, 127 | "id": "zesEg6XXxp", 128 | "from_name": "comparison", 129 | "to_name": "comparison", 130 | "type": "pairwise", 131 | "origin": "manual" 132 | } 133 | ], 134 | "was_cancelled": false, 135 | "ground_truth": false, 136 | "created_at": "2023-04-17T21:54:29.067807Z", 137 | "updated_at": "2023-04-17T21:54:29.067834Z", 138 | "lead_time": 3.002, 139 | "prediction": {}, 140 | "result_count": 0, 141 | "unique_id": "c21bd5c4-d424-4202-aac5-e495997b754e", 142 | "last_action": null, 143 | "task": 401637, 144 | "project": 183, 145 | "updated_by": 2, 146 | "parent_prediction": null, 147 | "parent_annotation": null, 148 | "last_created_by": null 149 | } 150 | ], 151 | "file_upload": "65af2932-ls_input_data.json", 152 | "drafts": [], 153 | "predictions": [], 154 | "data": { 155 | "prompt": "What are the latest developments in technology?", 156 | "answer1": "Will it ever be able to be incorporated into existing devices?\n\nThis survey started as a conversation about how technology might help people better understand their own personal data. When there's a large body of data,", 157 | "answer2": "This has been the most difficult aspect of my career, due to multiple sources I have already spoken with, the amount of people who knew and had close relationships with the developers and the way they dealt with" 158 | }, 159 | "meta": {}, 160 | "created_at": "2023-04-17T21:53:56.689662Z", 161 | "updated_at": "2023-04-17T21:54:29.101730Z", 162 | "inner_id": 3, 163 | "total_annotations": 1, 164 | "cancelled_annotations": 0, 165 | "total_predictions": 0, 166 | "comment_count": 0, 167 | "unresolved_comment_count": 0, 168 | "last_comment_updated_at": null, 169 | "project": 183, 170 | "updated_by": 2, 171 | "comment_authors": [] 172 | }, 173 | { 174 | "id": 401638, 175 | "annotations": [ 176 | { 177 | "id": 68048, 178 | "completed_by": 2, 179 | "result": [ 180 | { 181 | "value": { 182 | "selected": "left" 183 | }, 184 | "id": "w-z3BSG09V", 185 | "from_name": "comparison", 186 | "to_name": "comparison", 187 | "type": "pairwise", 188 | "origin": "manual" 189 | } 190 | ], 191 | "was_cancelled": false, 192 | "ground_truth": false, 193 | "created_at": "2023-04-17T21:54:32.730112Z", 194 | "updated_at": "2023-04-17T21:54:32.730135Z", 195 | "lead_time": 3.513, 196 | "prediction": {}, 197 | "result_count": 0, 198 | "unique_id": "295fa97a-06f1-46f8-8bba-422fb11c6980", 199 | "last_action": null, 200 | "task": 401638, 201 | "project": 183, 202 | "updated_by": 2, 203 | "parent_prediction": null, 204 | "parent_annotation": null, 205 | "last_created_by": null 206 | } 207 | ], 208 | "file_upload": "65af2932-ls_input_data.json", 209 | "drafts": [], 210 | "predictions": [], 211 | "data": { 212 | "prompt": "What is the political situation in the Middle East?", 213 | "answer1": "The Arab League, of course, is very concerned about what's going on in the Middle East, and for that, the West is not necessarily going to keep to the model they have,", 214 | "answer2": "On the one hand, we are dealing here with a crisis of the way, a period of uncertainty. But on the other hand, this crisis comes with a price. Our relationship with some" 215 | }, 216 | "meta": {}, 217 | "created_at": "2023-04-17T21:53:56.689719Z", 218 | "updated_at": "2023-04-17T21:54:32.759669Z", 219 | "inner_id": 4, 220 | "total_annotations": 1, 221 | "cancelled_annotations": 0, 222 | "total_predictions": 0, 223 | "comment_count": 0, 224 | "unresolved_comment_count": 0, 225 | "last_comment_updated_at": null, 226 | "project": 183, 227 | "updated_by": 2, 228 | "comment_authors": [] 229 | }, 230 | { 231 | "id": 401639, 232 | "annotations": [ 233 | { 234 | "id": 68049, 235 | "completed_by": 2, 236 | "result": [ 237 | { 238 | "value": { 239 | "selected": "right" 240 | }, 241 | "id": "lDkNZt7C6c", 242 | "from_name": "comparison", 243 | "to_name": "comparison", 244 | "type": "pairwise", 245 | "origin": "manual" 246 | } 247 | ], 248 | "was_cancelled": false, 249 | "ground_truth": false, 250 | "created_at": "2023-04-17T21:54:34.106777Z", 251 | "updated_at": "2023-04-17T21:54:34.106810Z", 252 | "lead_time": 1.225, 253 | "prediction": {}, 254 | "result_count": 0, 255 | "unique_id": "7c4855ea-8a9e-415a-94fe-1c805f3af941", 256 | "last_action": null, 257 | "task": 401639, 258 | "project": 183, 259 | "updated_by": 2, 260 | "parent_prediction": null, 261 | "parent_annotation": null, 262 | "last_created_by": null 263 | } 264 | ], 265 | "file_upload": "65af2932-ls_input_data.json", 266 | "drafts": [], 267 | "predictions": [], 268 | "data": { 269 | "prompt": "What are the latest trends in fashion and beauty?", 270 | "answer1": "", 271 | "answer2": "We believe the new trend we tend to see is the beauty trend and the rise of celebrity. A huge amount of young men are getting into fashion because they like the work and the money and" 272 | }, 273 | "meta": {}, 274 | "created_at": "2023-04-17T21:53:56.689778Z", 275 | "updated_at": "2023-04-17T21:54:34.142130Z", 276 | "inner_id": 5, 277 | "total_annotations": 1, 278 | "cancelled_annotations": 0, 279 | "total_predictions": 0, 280 | "comment_count": 0, 281 | "unresolved_comment_count": 0, 282 | "last_comment_updated_at": null, 283 | "project": 183, 284 | "updated_by": 2, 285 | "comment_authors": [] 286 | }, 287 | { 288 | "id": 401640, 289 | "annotations": [ 290 | { 291 | "id": 68050, 292 | "completed_by": 2, 293 | "result": [ 294 | { 295 | "value": { 296 | "selected": "left" 297 | }, 298 | "id": "A7VZ-4IPkV", 299 | "from_name": "comparison", 300 | "to_name": "comparison", 301 | "type": "pairwise", 302 | "origin": "manual" 303 | } 304 | ], 305 | "was_cancelled": false, 306 | "ground_truth": false, 307 | "created_at": "2023-04-17T21:54:40.054708Z", 308 | "updated_at": "2023-04-17T21:54:40.054744Z", 309 | "lead_time": 5.783, 310 | "prediction": {}, 311 | "result_count": 0, 312 | "unique_id": "ed87356c-4273-49bb-b35d-c10210a420fc", 313 | "last_action": null, 314 | "task": 401640, 315 | "project": 183, 316 | "updated_by": 2, 317 | "parent_prediction": null, 318 | "parent_annotation": null, 319 | "last_created_by": null 320 | } 321 | ], 322 | "file_upload": "65af2932-ls_input_data.json", 323 | "drafts": [], 324 | "predictions": [], 325 | "data": { 326 | "prompt": "What are the top travel destinations for this year?", 327 | "answer1": "The travel categories are not broken up by country, but are rather organized into three segments: Australia (where your passport is valid), Germany (where it is valid), and Taiwan, where it", 328 | "answer2": "1. LA, California 2. Phoenix, Arizona 3. Toronto, Ontario 4. Las Vegas, Nevada 5. Atlanta, Georgia 6. New York, New York\n\nIf you found" 329 | }, 330 | "meta": {}, 331 | "created_at": "2023-04-17T21:53:56.689850Z", 332 | "updated_at": "2023-04-17T21:54:40.088359Z", 333 | "inner_id": 6, 334 | "total_annotations": 1, 335 | "cancelled_annotations": 0, 336 | "total_predictions": 0, 337 | "comment_count": 0, 338 | "unresolved_comment_count": 0, 339 | "last_comment_updated_at": null, 340 | "project": 183, 341 | "updated_by": 2, 342 | "comment_authors": [] 343 | }, 344 | { 345 | "id": 401641, 346 | "annotations": [ 347 | { 348 | "id": 68051, 349 | "completed_by": 2, 350 | "result": [ 351 | { 352 | "value": { 353 | "selected": "right" 354 | }, 355 | "id": "wqZRfajJrS", 356 | "from_name": "comparison", 357 | "to_name": "comparison", 358 | "type": "pairwise", 359 | "origin": "manual" 360 | } 361 | ], 362 | "was_cancelled": false, 363 | "ground_truth": false, 364 | "created_at": "2023-04-17T21:54:42.792855Z", 365 | "updated_at": "2023-04-17T21:54:42.792880Z", 366 | "lead_time": 2.544, 367 | "prediction": {}, 368 | "result_count": 0, 369 | "unique_id": "e24f975d-9cb3-4304-bec9-79f4dbf0a726", 370 | "last_action": null, 371 | "task": 401641, 372 | "project": 183, 373 | "updated_by": 2, 374 | "parent_prediction": null, 375 | "parent_annotation": null, 376 | "last_created_by": null 377 | } 378 | ], 379 | "file_upload": "65af2932-ls_input_data.json", 380 | "drafts": [], 381 | "predictions": [], 382 | "data": { 383 | "prompt": "What are some healthy recipes for a vegan diet?", 384 | "answer1": "This post is really long, so I wanted to stick to a quick little guide for what you need, so please refrain from commenting!\n\nIf you really want to know what's going", 385 | "answer2": "1. Take a lot of water.\n\n2. Mix flour, vegetable oil, almond extract and cayenne pepper together with a mix bit more lightly. Mix well." 386 | }, 387 | "meta": {}, 388 | "created_at": "2023-04-17T21:53:56.689938Z", 389 | "updated_at": "2023-04-17T21:54:42.825539Z", 390 | "inner_id": 7, 391 | "total_annotations": 1, 392 | "cancelled_annotations": 0, 393 | "total_predictions": 0, 394 | "comment_count": 0, 395 | "unresolved_comment_count": 0, 396 | "last_comment_updated_at": null, 397 | "project": 183, 398 | "updated_by": 2, 399 | "comment_authors": [] 400 | }, 401 | { 402 | "id": 401642, 403 | "annotations": [ 404 | { 405 | "id": 68052, 406 | "completed_by": 2, 407 | "result": [ 408 | { 409 | "value": { 410 | "selected": "right" 411 | }, 412 | "id": "AjGrm4_GlY", 413 | "from_name": "comparison", 414 | "to_name": "comparison", 415 | "type": "pairwise", 416 | "origin": "manual" 417 | } 418 | ], 419 | "was_cancelled": false, 420 | "ground_truth": false, 421 | "created_at": "2023-04-17T21:54:47.833039Z", 422 | "updated_at": "2023-04-17T21:54:47.833069Z", 423 | "lead_time": 4.883, 424 | "prediction": {}, 425 | "result_count": 0, 426 | "unique_id": "6ff073ba-4fae-4d11-9d6a-5cf62a150ea2", 427 | "last_action": null, 428 | "task": 401642, 429 | "project": 183, 430 | "updated_by": 2, 431 | "parent_prediction": null, 432 | "parent_annotation": null, 433 | "last_created_by": null 434 | } 435 | ], 436 | "file_upload": "65af2932-ls_input_data.json", 437 | "drafts": [], 438 | "predictions": [], 439 | "data": { 440 | "prompt": "What are the most important events happening in the world today?", 441 | "answer1": "1. The fall of the Roman Empire\n\nOne of the most important events in humanity's history occurred the fall of Rome on December 31, 1446 and during that time the", 442 | "answer2": "A global financial crisis triggered massive international recession, global economic turmoil, unprecedented financial crises in the 1990s, and a global financial meltdown, resulting in a global financial meltdown of unparalleled systemic" 443 | }, 444 | "meta": {}, 445 | "created_at": "2023-04-17T21:53:56.689994Z", 446 | "updated_at": "2023-04-17T21:54:47.868204Z", 447 | "inner_id": 8, 448 | "total_annotations": 1, 449 | "cancelled_annotations": 0, 450 | "total_predictions": 0, 451 | "comment_count": 0, 452 | "unresolved_comment_count": 0, 453 | "last_comment_updated_at": null, 454 | "project": 183, 455 | "updated_by": 2, 456 | "comment_authors": [] 457 | }, 458 | { 459 | "id": 401643, 460 | "annotations": [ 461 | { 462 | "id": 68053, 463 | "completed_by": 2, 464 | "result": [ 465 | { 466 | "value": { 467 | "selected": "right" 468 | }, 469 | "id": "Bt4sdtZjtL", 470 | "from_name": "comparison", 471 | "to_name": "comparison", 472 | "type": "pairwise", 473 | "origin": "manual" 474 | } 475 | ], 476 | "was_cancelled": false, 477 | "ground_truth": false, 478 | "created_at": "2023-04-17T21:54:49.688670Z", 479 | "updated_at": "2023-04-17T21:54:49.688699Z", 480 | "lead_time": 1.703, 481 | "prediction": {}, 482 | "result_count": 0, 483 | "unique_id": "27896962-7eac-407b-aefc-e4ea0cbb4c61", 484 | "last_action": null, 485 | "task": 401643, 486 | "project": 183, 487 | "updated_by": 2, 488 | "parent_prediction": null, 489 | "parent_annotation": null, 490 | "last_created_by": null 491 | } 492 | ], 493 | "file_upload": "65af2932-ls_input_data.json", 494 | "drafts": [], 495 | "predictions": [], 496 | "data": { 497 | "prompt": "What are some tips for improving mental health?", 498 | "answer1": "One of the many things that people are good at and are working hard on is helping themselves and others get better. It's helpful to know your options.\n\nIn the past, mental health", 499 | "answer2": "1) Get out there and have a conversation. Maybe some people won't talk. Maybe you have friends who don't agree with your mental health. The best thing is if they have experienced anything" 500 | }, 501 | "meta": {}, 502 | "created_at": "2023-04-17T21:53:56.690051Z", 503 | "updated_at": "2023-04-17T21:54:49.717007Z", 504 | "inner_id": 9, 505 | "total_annotations": 1, 506 | "cancelled_annotations": 0, 507 | "total_predictions": 0, 508 | "comment_count": 0, 509 | "unresolved_comment_count": 0, 510 | "last_comment_updated_at": null, 511 | "project": 183, 512 | "updated_by": 2, 513 | "comment_authors": [] 514 | }, 515 | { 516 | "id": 401644, 517 | "annotations": [ 518 | { 519 | "id": 68054, 520 | "completed_by": 2, 521 | "result": [ 522 | { 523 | "value": { 524 | "selected": "right" 525 | }, 526 | "id": "3BjKbmtwaN", 527 | "from_name": "comparison", 528 | "to_name": "comparison", 529 | "type": "pairwise", 530 | "origin": "manual" 531 | } 532 | ], 533 | "was_cancelled": false, 534 | "ground_truth": false, 535 | "created_at": "2023-04-17T21:54:56.464129Z", 536 | "updated_at": "2023-04-17T21:54:56.464151Z", 537 | "lead_time": 6.641, 538 | "prediction": {}, 539 | "result_count": 0, 540 | "unique_id": "fbc5faf6-b2c4-4860-959e-49832c32855a", 541 | "last_action": null, 542 | "task": 401644, 543 | "project": 183, 544 | "updated_by": 2, 545 | "parent_prediction": null, 546 | "parent_annotation": null, 547 | "last_created_by": null 548 | } 549 | ], 550 | "file_upload": "65af2932-ls_input_data.json", 551 | "drafts": [], 552 | "predictions": [], 553 | "data": { 554 | "prompt": "What are the best ways to save money for retirement?", 555 | "answer1": "What types of investments need to have their own annual accounts?\n\nDo you find it essential to set up a account plan to help you make your own decisions on what to invest in", 556 | "answer2": "First, there is a lot of great information online about how to pay for your retirement. This post has lots of links, as well as some great tips that will help you decide if" 557 | }, 558 | "meta": {}, 559 | "created_at": "2023-04-17T21:53:56.690111Z", 560 | "updated_at": "2023-04-17T21:54:56.495936Z", 561 | "inner_id": 10, 562 | "total_annotations": 1, 563 | "cancelled_annotations": 0, 564 | "total_predictions": 0, 565 | "comment_count": 0, 566 | "unresolved_comment_count": 0, 567 | "last_comment_updated_at": null, 568 | "project": 183, 569 | "updated_by": 2, 570 | "comment_authors": [] 571 | }, 572 | { 573 | "id": 401645, 574 | "annotations": [ 575 | { 576 | "id": 68055, 577 | "completed_by": 2, 578 | "result": [ 579 | { 580 | "value": { 581 | "selected": "left" 582 | }, 583 | "id": "gi-GTT9QvO", 584 | "from_name": "comparison", 585 | "to_name": "comparison", 586 | "type": "pairwise", 587 | "origin": "manual" 588 | } 589 | ], 590 | "was_cancelled": false, 591 | "ground_truth": false, 592 | "created_at": "2023-04-17T21:54:58.470877Z", 593 | "updated_at": "2023-04-17T21:54:58.470905Z", 594 | "lead_time": 1.854, 595 | "prediction": {}, 596 | "result_count": 0, 597 | "unique_id": "ff8a9ab5-1c9e-4de4-bd43-24b54de0d464", 598 | "last_action": null, 599 | "task": 401645, 600 | "project": 183, 601 | "updated_by": 2, 602 | "parent_prediction": null, 603 | "parent_annotation": null, 604 | "last_created_by": null 605 | } 606 | ], 607 | "file_upload": "65af2932-ls_input_data.json", 608 | "drafts": [], 609 | "predictions": [], 610 | "data": { 611 | "prompt": "What are some popular new books or movies?", 612 | "answer1": "A very low number of books are sold digitally (as well as on social media sites) but most TV and radio shows (including many documentaries on film) have been available for a couple of years", 613 | "answer2": "You've got to make them in real time. Just keep repeating the idea -- if a book gets more hits then it's getting more books to make. Let yourself get a few books to read by the" 614 | }, 615 | "meta": {}, 616 | "created_at": "2023-04-17T21:53:56.690172Z", 617 | "updated_at": "2023-04-17T21:54:58.504871Z", 618 | "inner_id": 11, 619 | "total_annotations": 1, 620 | "cancelled_annotations": 0, 621 | "total_predictions": 0, 622 | "comment_count": 0, 623 | "unresolved_comment_count": 0, 624 | "last_comment_updated_at": null, 625 | "project": 183, 626 | "updated_by": 2, 627 | "comment_authors": [] 628 | }, 629 | { 630 | "id": 401646, 631 | "annotations": [ 632 | { 633 | "id": 68056, 634 | "completed_by": 2, 635 | "result": [ 636 | { 637 | "value": { 638 | "selected": "left" 639 | }, 640 | "id": "xf_ZYmU8dZ", 641 | "from_name": "comparison", 642 | "to_name": "comparison", 643 | "type": "pairwise", 644 | "origin": "manual" 645 | } 646 | ], 647 | "was_cancelled": false, 648 | "ground_truth": false, 649 | "created_at": "2023-04-17T21:55:00.076099Z", 650 | "updated_at": "2023-04-17T21:55:00.076121Z", 651 | "lead_time": 1.465, 652 | "prediction": {}, 653 | "result_count": 0, 654 | "unique_id": "83ceeb10-a8dc-4b6f-a96b-48a678b77d89", 655 | "last_action": null, 656 | "task": 401646, 657 | "project": 183, 658 | "updated_by": 2, 659 | "parent_prediction": null, 660 | "parent_annotation": null, 661 | "last_created_by": null 662 | } 663 | ], 664 | "file_upload": "65af2932-ls_input_data.json", 665 | "drafts": [], 666 | "predictions": [], 667 | "data": { 668 | "prompt": "What are some effective ways to reduce stress?", 669 | "answer1": "(What you'll need\n\nIn short - Stress is something you're all about because of the actions and thinking that your system takes. The easiest way to reduce stress is to avoid it, take steps", 670 | "answer2": "Well, it's easy to find those simple, but powerful ways that stress can actually work well in the face of adversity — things like that:\n\nWhen I was in high school, I" 671 | }, 672 | "meta": {}, 673 | "created_at": "2023-04-17T21:53:56.690235Z", 674 | "updated_at": "2023-04-17T21:55:00.108446Z", 675 | "inner_id": 12, 676 | "total_annotations": 1, 677 | "cancelled_annotations": 0, 678 | "total_predictions": 0, 679 | "comment_count": 0, 680 | "unresolved_comment_count": 0, 681 | "last_comment_updated_at": null, 682 | "project": 183, 683 | "updated_by": 2, 684 | "comment_authors": [] 685 | }, 686 | { 687 | "id": 401647, 688 | "annotations": [ 689 | { 690 | "id": 68057, 691 | "completed_by": 2, 692 | "result": [ 693 | { 694 | "value": { 695 | "selected": "right" 696 | }, 697 | "id": "PpeT2nKeuG", 698 | "from_name": "comparison", 699 | "to_name": "comparison", 700 | "type": "pairwise", 701 | "origin": "manual" 702 | } 703 | ], 704 | "was_cancelled": false, 705 | "ground_truth": false, 706 | "created_at": "2023-04-17T21:55:01.338344Z", 707 | "updated_at": "2023-04-17T21:55:01.338372Z", 708 | "lead_time": 1.11, 709 | "prediction": {}, 710 | "result_count": 0, 711 | "unique_id": "9535594d-6d44-4ecb-b597-5a28965dde3b", 712 | "last_action": null, 713 | "task": 401647, 714 | "project": 183, 715 | "updated_by": 2, 716 | "parent_prediction": null, 717 | "parent_annotation": null, 718 | "last_created_by": null 719 | } 720 | ], 721 | "file_upload": "65af2932-ls_input_data.json", 722 | "drafts": [], 723 | "predictions": [], 724 | "data": { 725 | "prompt": "What are the latest developments in artificial intelligence?", 726 | "answer1": "How are the companies in the field working to improve our understanding about the nature of information?\n\nIn our research, we focused on research and development by researchers, especially those who may be more", 727 | "answer2": "On the frontline of AI and beyond, Artificial Intelligence has been on the offensive for decades. Some of the most important advances in this field were first introduced in the 1960s and have been observed" 728 | }, 729 | "meta": {}, 730 | "created_at": "2023-04-17T21:53:56.690295Z", 731 | "updated_at": "2023-04-17T21:55:01.372145Z", 732 | "inner_id": 13, 733 | "total_annotations": 1, 734 | "cancelled_annotations": 0, 735 | "total_predictions": 0, 736 | "comment_count": 0, 737 | "unresolved_comment_count": 0, 738 | "last_comment_updated_at": null, 739 | "project": 183, 740 | "updated_by": 2, 741 | "comment_authors": [] 742 | }, 743 | { 744 | "id": 401648, 745 | "annotations": [ 746 | { 747 | "id": 68058, 748 | "completed_by": 2, 749 | "result": [ 750 | { 751 | "value": { 752 | "selected": "left" 753 | }, 754 | "id": "AnHQcR02Ml", 755 | "from_name": "comparison", 756 | "to_name": "comparison", 757 | "type": "pairwise", 758 | "origin": "manual" 759 | } 760 | ], 761 | "was_cancelled": false, 762 | "ground_truth": false, 763 | "created_at": "2023-04-17T21:55:05.729432Z", 764 | "updated_at": "2023-04-17T21:55:05.729468Z", 765 | "lead_time": 4.235, 766 | "prediction": {}, 767 | "result_count": 0, 768 | "unique_id": "ae845e47-905d-45fb-847b-f7db7718405b", 769 | "last_action": null, 770 | "task": 401648, 771 | "project": 183, 772 | "updated_by": 2, 773 | "parent_prediction": null, 774 | "parent_annotation": null, 775 | "last_created_by": null 776 | } 777 | ], 778 | "file_upload": "65af2932-ls_input_data.json", 779 | "drafts": [], 780 | "predictions": [], 781 | "data": { 782 | "prompt": "What are some top-rated restaurants in your city?", 783 | "answer1": "In order to choose the right restaurant, we rely on the following criteria:\n\nYour area's reputation;\n\nYour staff's skills;\n\nIn all cases, you'll", 784 | "answer2": "Let us know in the comments!" 785 | }, 786 | "meta": {}, 787 | "created_at": "2023-04-17T21:53:56.690366Z", 788 | "updated_at": "2023-04-17T21:55:05.762851Z", 789 | "inner_id": 14, 790 | "total_annotations": 1, 791 | "cancelled_annotations": 0, 792 | "total_predictions": 0, 793 | "comment_count": 0, 794 | "unresolved_comment_count": 0, 795 | "last_comment_updated_at": null, 796 | "project": 183, 797 | "updated_by": 2, 798 | "comment_authors": [] 799 | }, 800 | { 801 | "id": 401649, 802 | "annotations": [ 803 | { 804 | "id": 68059, 805 | "completed_by": 2, 806 | "result": [ 807 | { 808 | "value": { 809 | "selected": "right" 810 | }, 811 | "id": "AK4GUxYo9o", 812 | "from_name": "comparison", 813 | "to_name": "comparison", 814 | "type": "pairwise", 815 | "origin": "manual" 816 | } 817 | ], 818 | "was_cancelled": false, 819 | "ground_truth": false, 820 | "created_at": "2023-04-17T21:55:07.665882Z", 821 | "updated_at": "2023-04-17T21:55:07.665914Z", 822 | "lead_time": 1.786, 823 | "prediction": {}, 824 | "result_count": 0, 825 | "unique_id": "525fac20-82dc-4150-b71a-bc2c970c3078", 826 | "last_action": null, 827 | "task": 401649, 828 | "project": 183, 829 | "updated_by": 2, 830 | "parent_prediction": null, 831 | "parent_annotation": null, 832 | "last_created_by": null 833 | } 834 | ], 835 | "file_upload": "65af2932-ls_input_data.json", 836 | "drafts": [], 837 | "predictions": [], 838 | "data": { 839 | "prompt": "What are the best ways to stay fit and healthy?", 840 | "answer1": "In an article of the \"What Are You Doing at Work?\" category, FitIQ explained how to keep up with your body when it comes to exercise:\n\n\"To help you", 841 | "answer2": "Find out with our simple answers, designed by a fitness enthusiast who doesn't hate anything but fitness!\n\nIf you know what exercises you'll need, you should try. Here on the Fit" 842 | }, 843 | "meta": {}, 844 | "created_at": "2023-04-17T21:53:56.690550Z", 845 | "updated_at": "2023-04-17T21:55:07.697648Z", 846 | "inner_id": 15, 847 | "total_annotations": 1, 848 | "cancelled_annotations": 0, 849 | "total_predictions": 0, 850 | "comment_count": 0, 851 | "unresolved_comment_count": 0, 852 | "last_comment_updated_at": null, 853 | "project": 183, 854 | "updated_by": 2, 855 | "comment_authors": [] 856 | }, 857 | { 858 | "id": 401650, 859 | "annotations": [ 860 | { 861 | "id": 68060, 862 | "completed_by": 2, 863 | "result": [ 864 | { 865 | "value": { 866 | "selected": "left" 867 | }, 868 | "id": "1-Xc3y6U7f", 869 | "from_name": "comparison", 870 | "to_name": "comparison", 871 | "type": "pairwise", 872 | "origin": "manual" 873 | } 874 | ], 875 | "was_cancelled": false, 876 | "ground_truth": false, 877 | "created_at": "2023-04-17T21:55:08.937173Z", 878 | "updated_at": "2023-04-17T21:55:08.937206Z", 879 | "lead_time": 1.131, 880 | "prediction": {}, 881 | "result_count": 0, 882 | "unique_id": "87a3cddb-7d4e-4535-b1b6-e7a26971f7c9", 883 | "last_action": null, 884 | "task": 401650, 885 | "project": 183, 886 | "updated_by": 2, 887 | "parent_prediction": null, 888 | "parent_annotation": null, 889 | "last_created_by": null 890 | } 891 | ], 892 | "file_upload": "65af2932-ls_input_data.json", 893 | "drafts": [], 894 | "predictions": [], 895 | "data": { 896 | "prompt": "What are some tips for successful entrepreneurship?", 897 | "answer1": "When your own business is first entering the market you might receive a few tips, especially if you are setting a goal.\n\nWhat are some tips for successful entrepreneurship? The sooner your goal is reached", 898 | "answer2": "1. Avoid marketing scams\n\nIt has been said that you should NEVER advertise in print. It's pretty obvious that you need a professional if you don't want to get your clients to write about" 899 | }, 900 | "meta": {}, 901 | "created_at": "2023-04-17T21:53:56.690633Z", 902 | "updated_at": "2023-04-17T21:55:08.967158Z", 903 | "inner_id": 16, 904 | "total_annotations": 1, 905 | "cancelled_annotations": 0, 906 | "total_predictions": 0, 907 | "comment_count": 0, 908 | "unresolved_comment_count": 0, 909 | "last_comment_updated_at": null, 910 | "project": 183, 911 | "updated_by": 2, 912 | "comment_authors": [] 913 | }, 914 | { 915 | "id": 401651, 916 | "annotations": [ 917 | { 918 | "id": 68061, 919 | "completed_by": 2, 920 | "result": [ 921 | { 922 | "value": { 923 | "selected": "left" 924 | }, 925 | "id": "0Yow6k1iK9", 926 | "from_name": "comparison", 927 | "to_name": "comparison", 928 | "type": "pairwise", 929 | "origin": "manual" 930 | } 931 | ], 932 | "was_cancelled": false, 933 | "ground_truth": false, 934 | "created_at": "2023-04-17T21:55:11.785361Z", 935 | "updated_at": "2023-04-17T21:55:11.785388Z", 936 | "lead_time": 1.643, 937 | "prediction": {}, 938 | "result_count": 0, 939 | "unique_id": "32a4cb78-b4e6-47f6-8001-8163857d6733", 940 | "last_action": null, 941 | "task": 401651, 942 | "project": 183, 943 | "updated_by": 2, 944 | "parent_prediction": null, 945 | "parent_annotation": null, 946 | "last_created_by": null 947 | } 948 | ], 949 | "file_upload": "65af2932-ls_input_data.json", 950 | "drafts": [], 951 | "predictions": [], 952 | "data": { 953 | "prompt": "What are some effective ways to improve productivity?", 954 | "answer1": "What makes a good productivity manager? It is a decision made from within your organization that is made more often than not based on your experience rather than simply based on your own ideas, which are generally", 955 | "answer2": "The productivity gains of the following three items can help people gain more productivity:\n\nIncreased focus on the task at hand. By increasing the amount of time in which they spend focused thinking about the" 956 | }, 957 | "meta": {}, 958 | "created_at": "2023-04-17T21:53:56.690691Z", 959 | "updated_at": "2023-04-17T21:55:11.815198Z", 960 | "inner_id": 17, 961 | "total_annotations": 1, 962 | "cancelled_annotations": 0, 963 | "total_predictions": 0, 964 | "comment_count": 0, 965 | "unresolved_comment_count": 0, 966 | "last_comment_updated_at": null, 967 | "project": 183, 968 | "updated_by": 2, 969 | "comment_authors": [] 970 | }, 971 | { 972 | "id": 401652, 973 | "annotations": [ 974 | { 975 | "id": 68062, 976 | "completed_by": 2, 977 | "result": [ 978 | { 979 | "value": { 980 | "selected": "right" 981 | }, 982 | "id": "Usdl7tWlWN", 983 | "from_name": "comparison", 984 | "to_name": "comparison", 985 | "type": "pairwise", 986 | "origin": "manual" 987 | } 988 | ], 989 | "was_cancelled": false, 990 | "ground_truth": false, 991 | "created_at": "2023-04-17T21:55:14.312547Z", 992 | "updated_at": "2023-04-17T21:55:14.312573Z", 993 | "lead_time": 2.382, 994 | "prediction": {}, 995 | "result_count": 0, 996 | "unique_id": "87cc9a8b-5fae-4488-bd2e-44eb94ebbaf3", 997 | "last_action": null, 998 | "task": 401652, 999 | "project": 183, 1000 | "updated_by": 2, 1001 | "parent_prediction": null, 1002 | "parent_annotation": null, 1003 | "last_created_by": null 1004 | } 1005 | ], 1006 | "file_upload": "65af2932-ls_input_data.json", 1007 | "drafts": [], 1008 | "predictions": [], 1009 | "data": { 1010 | "prompt": "What are the latest developments in climate change research?", 1011 | "answer1": "Read more\n\nDavid Brown, chairman of the IPCC, said some recent changes to the science, such as its analysis of the role of ocean warming due to human greenhouse effects, \"can only be", 1012 | "answer2": "Are we going to change that? What is the link between the various science communities being informed about the climate crisis? What are the causes of the problems we're facing? What needs to be done to" 1013 | }, 1014 | "meta": {}, 1015 | "created_at": "2023-04-17T21:53:56.690748Z", 1016 | "updated_at": "2023-04-17T21:55:14.344947Z", 1017 | "inner_id": 18, 1018 | "total_annotations": 1, 1019 | "cancelled_annotations": 0, 1020 | "total_predictions": 0, 1021 | "comment_count": 0, 1022 | "unresolved_comment_count": 0, 1023 | "last_comment_updated_at": null, 1024 | "project": 183, 1025 | "updated_by": 2, 1026 | "comment_authors": [] 1027 | }, 1028 | { 1029 | "id": 401653, 1030 | "annotations": [ 1031 | { 1032 | "id": 68063, 1033 | "completed_by": 2, 1034 | "result": [ 1035 | { 1036 | "value": { 1037 | "selected": "right" 1038 | }, 1039 | "id": "JovbZHeb8M", 1040 | "from_name": "comparison", 1041 | "to_name": "comparison", 1042 | "type": "pairwise", 1043 | "origin": "manual" 1044 | } 1045 | ], 1046 | "was_cancelled": false, 1047 | "ground_truth": false, 1048 | "created_at": "2023-04-17T21:55:15.796686Z", 1049 | "updated_at": "2023-04-17T21:55:15.796713Z", 1050 | "lead_time": 1.337, 1051 | "prediction": {}, 1052 | "result_count": 0, 1053 | "unique_id": "ab36e09a-ac0f-4021-b87d-c0a00cf65159", 1054 | "last_action": null, 1055 | "task": 401653, 1056 | "project": 183, 1057 | "updated_by": 2, 1058 | "parent_prediction": null, 1059 | "parent_annotation": null, 1060 | "last_created_by": null 1061 | } 1062 | ], 1063 | "file_upload": "65af2932-ls_input_data.json", 1064 | "drafts": [], 1065 | "predictions": [], 1066 | "data": { 1067 | "prompt": "What are some top-rated TV shows or movies on streaming services?", 1068 | "answer1": "We love sports sports, but we're always looking for more good sports programming. It's usually a combination of great sports programming and great entertainment. The best sports programming has", 1069 | "answer2": "Comedy Central (CBS)\n\nComedy Central is the only cable channel in North America where you can catch up on current episode time and date from various networks." 1070 | }, 1071 | "meta": {}, 1072 | "created_at": "2023-04-17T21:53:56.690803Z", 1073 | "updated_at": "2023-04-17T21:55:15.827660Z", 1074 | "inner_id": 19, 1075 | "total_annotations": 1, 1076 | "cancelled_annotations": 0, 1077 | "total_predictions": 0, 1078 | "comment_count": 0, 1079 | "unresolved_comment_count": 0, 1080 | "last_comment_updated_at": null, 1081 | "project": 183, 1082 | "updated_by": 2, 1083 | "comment_authors": [] 1084 | }, 1085 | { 1086 | "id": 401654, 1087 | "annotations": [ 1088 | { 1089 | "id": 68064, 1090 | "completed_by": 2, 1091 | "result": [ 1092 | { 1093 | "value": { 1094 | "selected": "left" 1095 | }, 1096 | "id": "g4bcBpMnWM", 1097 | "from_name": "comparison", 1098 | "to_name": "comparison", 1099 | "type": "pairwise", 1100 | "origin": "manual" 1101 | } 1102 | ], 1103 | "was_cancelled": false, 1104 | "ground_truth": false, 1105 | "created_at": "2023-04-17T21:55:17.330213Z", 1106 | "updated_at": "2023-04-17T21:55:17.330247Z", 1107 | "lead_time": 1.387, 1108 | "prediction": {}, 1109 | "result_count": 0, 1110 | "unique_id": "a66f9fe6-996f-47a0-b23f-afb385b87159", 1111 | "last_action": null, 1112 | "task": 401654, 1113 | "project": 183, 1114 | "updated_by": 2, 1115 | "parent_prediction": null, 1116 | "parent_annotation": null, 1117 | "last_created_by": null 1118 | } 1119 | ], 1120 | "file_upload": "65af2932-ls_input_data.json", 1121 | "drafts": [], 1122 | "predictions": [], 1123 | "data": { 1124 | "prompt": "What are some fun activities to do on weekends?", 1125 | "answer1": "I work in the morning in the coffee kitchen, watching as girls and boys come to pick up their tea.\n\nWhere can I get my supplies?\n\nThere is no direct bus", 1126 | "answer2": "Share them with us at Facebook.com\/#!\/TravisBurdette @WUWTBruz\n\nRead more: [FULL LATEST ARTICLE]" 1127 | }, 1128 | "meta": {}, 1129 | "created_at": "2023-04-17T21:53:56.690859Z", 1130 | "updated_at": "2023-04-17T21:55:17.360784Z", 1131 | "inner_id": 20, 1132 | "total_annotations": 1, 1133 | "cancelled_annotations": 0, 1134 | "total_predictions": 0, 1135 | "comment_count": 0, 1136 | "unresolved_comment_count": 0, 1137 | "last_comment_updated_at": null, 1138 | "project": 183, 1139 | "updated_by": 2, 1140 | "comment_authors": [] 1141 | }, 1142 | { 1143 | "id": 401655, 1144 | "annotations": [ 1145 | { 1146 | "id": 68065, 1147 | "completed_by": 2, 1148 | "result": [ 1149 | { 1150 | "value": { 1151 | "selected": "left" 1152 | }, 1153 | "id": "fjLOUCL-e_", 1154 | "from_name": "comparison", 1155 | "to_name": "comparison", 1156 | "type": "pairwise", 1157 | "origin": "manual" 1158 | } 1159 | ], 1160 | "was_cancelled": false, 1161 | "ground_truth": false, 1162 | "created_at": "2023-04-17T21:55:18.728529Z", 1163 | "updated_at": "2023-04-17T21:55:18.728556Z", 1164 | "lead_time": 1.255, 1165 | "prediction": {}, 1166 | "result_count": 0, 1167 | "unique_id": "a918f550-5060-4a16-8cc4-4d506058c3f4", 1168 | "last_action": null, 1169 | "task": 401655, 1170 | "project": 183, 1171 | "updated_by": 2, 1172 | "parent_prediction": null, 1173 | "parent_annotation": null, 1174 | "last_created_by": null 1175 | } 1176 | ], 1177 | "file_upload": "65af2932-ls_input_data.json", 1178 | "drafts": [], 1179 | "predictions": [], 1180 | "data": { 1181 | "prompt": "What are some effective ways to manage time and prioritize tasks?", 1182 | "answer1": "Do you have any tips on how that might help with scheduling and\/or time management in a business? If so, what strategies, actions or tips do you use? Let us know in", 1183 | "answer2": "Where do we start, etc.? Where do we begin, etc.?\n\nIt's a complex question, so there are a couple important things to consider:\n\nTime\n\nTime" 1184 | }, 1185 | "meta": {}, 1186 | "created_at": "2023-04-17T21:53:56.690913Z", 1187 | "updated_at": "2023-04-17T21:55:18.761339Z", 1188 | "inner_id": 21, 1189 | "total_annotations": 1, 1190 | "cancelled_annotations": 0, 1191 | "total_predictions": 0, 1192 | "comment_count": 0, 1193 | "unresolved_comment_count": 0, 1194 | "last_comment_updated_at": null, 1195 | "project": 183, 1196 | "updated_by": 2, 1197 | "comment_authors": [] 1198 | }, 1199 | { 1200 | "id": 401656, 1201 | "annotations": [ 1202 | { 1203 | "id": 68066, 1204 | "completed_by": 2, 1205 | "result": [ 1206 | { 1207 | "value": { 1208 | "selected": "left" 1209 | }, 1210 | "id": "ZRYApDokfY", 1211 | "from_name": "comparison", 1212 | "to_name": "comparison", 1213 | "type": "pairwise", 1214 | "origin": "manual" 1215 | } 1216 | ], 1217 | "was_cancelled": false, 1218 | "ground_truth": false, 1219 | "created_at": "2023-04-17T21:55:19.854068Z", 1220 | "updated_at": "2023-04-17T21:55:19.854109Z", 1221 | "lead_time": 0.97, 1222 | "prediction": {}, 1223 | "result_count": 0, 1224 | "unique_id": "057cf84f-6551-493e-b7e6-7a52fbe753a4", 1225 | "last_action": null, 1226 | "task": 401656, 1227 | "project": 183, 1228 | "updated_by": 2, 1229 | "parent_prediction": null, 1230 | "parent_annotation": null, 1231 | "last_created_by": null 1232 | } 1233 | ], 1234 | "file_upload": "65af2932-ls_input_data.json", 1235 | "drafts": [], 1236 | "predictions": [], 1237 | "data": { 1238 | "prompt": "What are the latest trends in home decor and design?", 1239 | "answer1": "Let us know.\n\n\nThe Best: For its long-standing mission of supporting home furnishings, The Woodlands is home furnishings creator, design editor for the website. Woodland's", 1240 | "answer2": "" 1241 | }, 1242 | "meta": {}, 1243 | "created_at": "2023-04-17T21:53:56.690968Z", 1244 | "updated_at": "2023-04-17T21:55:19.885554Z", 1245 | "inner_id": 22, 1246 | "total_annotations": 1, 1247 | "cancelled_annotations": 0, 1248 | "total_predictions": 0, 1249 | "comment_count": 0, 1250 | "unresolved_comment_count": 0, 1251 | "last_comment_updated_at": null, 1252 | "project": 183, 1253 | "updated_by": 2, 1254 | "comment_authors": [] 1255 | }, 1256 | { 1257 | "id": 401657, 1258 | "annotations": [ 1259 | { 1260 | "id": 68067, 1261 | "completed_by": 2, 1262 | "result": [ 1263 | { 1264 | "value": { 1265 | "selected": "right" 1266 | }, 1267 | "id": "e-7vM_tvxw", 1268 | "from_name": "comparison", 1269 | "to_name": "comparison", 1270 | "type": "pairwise", 1271 | "origin": "manual" 1272 | } 1273 | ], 1274 | "was_cancelled": false, 1275 | "ground_truth": false, 1276 | "created_at": "2023-04-17T21:55:21.643783Z", 1277 | "updated_at": "2023-04-17T21:55:21.643806Z", 1278 | "lead_time": 1.623, 1279 | "prediction": {}, 1280 | "result_count": 0, 1281 | "unique_id": "5216c95c-6890-4dd7-abf9-3a54c7d3840d", 1282 | "last_action": null, 1283 | "task": 401657, 1284 | "project": 183, 1285 | "updated_by": 2, 1286 | "parent_prediction": null, 1287 | "parent_annotation": null, 1288 | "last_created_by": null 1289 | } 1290 | ], 1291 | "file_upload": "65af2932-ls_input_data.json", 1292 | "drafts": [], 1293 | "predictions": [], 1294 | "data": { 1295 | "prompt": "What are the best ways to develop a successful career?", 1296 | "answer1": "Many career paths consist of an individualization of a career process. There are two approaches which may help you to get there…\n\nYou can pick out specific opportunities or take an optional", 1297 | "answer2": "And when can I start building more? The process starts with a few basic questions. Do you want to learn anything new? If not, what do you prefer to learn or do when you do" 1298 | }, 1299 | "meta": {}, 1300 | "created_at": "2023-04-17T21:53:56.691024Z", 1301 | "updated_at": "2023-04-17T21:55:21.675914Z", 1302 | "inner_id": 23, 1303 | "total_annotations": 1, 1304 | "cancelled_annotations": 0, 1305 | "total_predictions": 0, 1306 | "comment_count": 0, 1307 | "unresolved_comment_count": 0, 1308 | "last_comment_updated_at": null, 1309 | "project": 183, 1310 | "updated_by": 2, 1311 | "comment_authors": [] 1312 | }, 1313 | { 1314 | "id": 401658, 1315 | "annotations": [ 1316 | { 1317 | "id": 68068, 1318 | "completed_by": 2, 1319 | "result": [ 1320 | { 1321 | "value": { 1322 | "selected": "left" 1323 | }, 1324 | "id": "fORB2aH2-4", 1325 | "from_name": "comparison", 1326 | "to_name": "comparison", 1327 | "type": "pairwise", 1328 | "origin": "manual" 1329 | } 1330 | ], 1331 | "was_cancelled": false, 1332 | "ground_truth": false, 1333 | "created_at": "2023-04-17T21:55:23.254890Z", 1334 | "updated_at": "2023-04-17T21:55:23.254915Z", 1335 | "lead_time": 1.471, 1336 | "prediction": {}, 1337 | "result_count": 0, 1338 | "unique_id": "fb1ff0f7-4748-4e10-a53b-61fd4c89c882", 1339 | "last_action": null, 1340 | "task": 401658, 1341 | "project": 183, 1342 | "updated_by": 2, 1343 | "parent_prediction": null, 1344 | "parent_annotation": null, 1345 | "last_created_by": null 1346 | } 1347 | ], 1348 | "file_upload": "65af2932-ls_input_data.json", 1349 | "drafts": [], 1350 | "predictions": [], 1351 | "data": { 1352 | "prompt": "What are some popular new products or gadgets?", 1353 | "answer1": "There has been a long talk at CES 2014 that the future of devices is in the hands of new developers. The point being that, as in many industries, one of the primary benefits from being", 1354 | "answer2": "There are some popular new devices that we are working on that bring something new to the Tablets. For example, as mentioned previously:\n\n\nIn order to connect tablets to the mobile gaming app" 1355 | }, 1356 | "meta": {}, 1357 | "created_at": "2023-04-17T21:53:56.691080Z", 1358 | "updated_at": "2023-04-17T21:55:23.284868Z", 1359 | "inner_id": 24, 1360 | "total_annotations": 1, 1361 | "cancelled_annotations": 0, 1362 | "total_predictions": 0, 1363 | "comment_count": 0, 1364 | "unresolved_comment_count": 0, 1365 | "last_comment_updated_at": null, 1366 | "project": 183, 1367 | "updated_by": 2, 1368 | "comment_authors": [] 1369 | }, 1370 | { 1371 | "id": 401659, 1372 | "annotations": [ 1373 | { 1374 | "id": 68069, 1375 | "completed_by": 2, 1376 | "result": [ 1377 | { 1378 | "value": { 1379 | "selected": "left" 1380 | }, 1381 | "id": "tJia5Rd782", 1382 | "from_name": "comparison", 1383 | "to_name": "comparison", 1384 | "type": "pairwise", 1385 | "origin": "manual" 1386 | } 1387 | ], 1388 | "was_cancelled": false, 1389 | "ground_truth": false, 1390 | "created_at": "2023-04-17T21:55:24.989877Z", 1391 | "updated_at": "2023-04-17T21:55:24.989903Z", 1392 | "lead_time": 1.587, 1393 | "prediction": {}, 1394 | "result_count": 0, 1395 | "unique_id": "c9fc7bc1-a1be-41b4-8338-308bd12df9db", 1396 | "last_action": null, 1397 | "task": 401659, 1398 | "project": 183, 1399 | "updated_by": 2, 1400 | "parent_prediction": null, 1401 | "parent_annotation": null, 1402 | "last_created_by": null 1403 | } 1404 | ], 1405 | "file_upload": "65af2932-ls_input_data.json", 1406 | "drafts": [], 1407 | "predictions": [], 1408 | "data": { 1409 | "prompt": "What are some effective ways to improve communication skills?", 1410 | "answer1": "The following is a list of examples of a general topic that all should address. Be patient when speaking about individual communication skills.\n\nPractice Your Own Businesses\n\nYour career and", 1411 | "answer2": "How to: Learn to focus on your communication when the world around you is constantly shifting, and at times you are constantly learning so you don't need to stop.\n\nOne or more" 1412 | }, 1413 | "meta": {}, 1414 | "created_at": "2023-04-17T21:53:56.691137Z", 1415 | "updated_at": "2023-04-17T21:55:25.019795Z", 1416 | "inner_id": 25, 1417 | "total_annotations": 1, 1418 | "cancelled_annotations": 0, 1419 | "total_predictions": 0, 1420 | "comment_count": 0, 1421 | "unresolved_comment_count": 0, 1422 | "last_comment_updated_at": null, 1423 | "project": 183, 1424 | "updated_by": 2, 1425 | "comment_authors": [] 1426 | }, 1427 | { 1428 | "id": 401660, 1429 | "annotations": [ 1430 | { 1431 | "id": 68070, 1432 | "completed_by": 2, 1433 | "result": [ 1434 | { 1435 | "value": { 1436 | "selected": "right" 1437 | }, 1438 | "id": "D9ep5Be1in", 1439 | "from_name": "comparison", 1440 | "to_name": "comparison", 1441 | "type": "pairwise", 1442 | "origin": "manual" 1443 | } 1444 | ], 1445 | "was_cancelled": false, 1446 | "ground_truth": false, 1447 | "created_at": "2023-04-17T21:55:26.322713Z", 1448 | "updated_at": "2023-04-17T21:55:26.322738Z", 1449 | "lead_time": 1.18, 1450 | "prediction": {}, 1451 | "result_count": 0, 1452 | "unique_id": "ce932a8a-aa6a-436e-b27d-21ff5f2e51cd", 1453 | "last_action": null, 1454 | "task": 401660, 1455 | "project": 183, 1456 | "updated_by": 2, 1457 | "parent_prediction": null, 1458 | "parent_annotation": null, 1459 | "last_created_by": null 1460 | } 1461 | ], 1462 | "file_upload": "65af2932-ls_input_data.json", 1463 | "drafts": [], 1464 | "predictions": [], 1465 | "data": { 1466 | "prompt": "What are some tips for successful relationships?", 1467 | "answer1": "One of the most important things you should know when choosing to have a relationship with someone is that you should never be a \"couple who just loves each other for a day\".\n\nIf you have an", 1468 | "answer2": "We all know that men are often the best men, but there's no doubt that women are most successful when they're good at their jobs. The job that pays the highest paying job can give you" 1469 | }, 1470 | "meta": {}, 1471 | "created_at": "2023-04-17T21:53:56.691193Z", 1472 | "updated_at": "2023-04-17T21:55:26.360442Z", 1473 | "inner_id": 26, 1474 | "total_annotations": 1, 1475 | "cancelled_annotations": 0, 1476 | "total_predictions": 0, 1477 | "comment_count": 0, 1478 | "unresolved_comment_count": 0, 1479 | "last_comment_updated_at": null, 1480 | "project": 183, 1481 | "updated_by": 2, 1482 | "comment_authors": [] 1483 | }, 1484 | { 1485 | "id": 401661, 1486 | "annotations": [ 1487 | { 1488 | "id": 68071, 1489 | "completed_by": 2, 1490 | "result": [ 1491 | { 1492 | "value": { 1493 | "selected": "right" 1494 | }, 1495 | "id": "CutnzASmKs", 1496 | "from_name": "comparison", 1497 | "to_name": "comparison", 1498 | "type": "pairwise", 1499 | "origin": "manual" 1500 | } 1501 | ], 1502 | "was_cancelled": false, 1503 | "ground_truth": false, 1504 | "created_at": "2023-04-17T21:55:28.917100Z", 1505 | "updated_at": "2023-04-17T21:55:28.917126Z", 1506 | "lead_time": 2.45, 1507 | "prediction": {}, 1508 | "result_count": 0, 1509 | "unique_id": "287ce9e4-67a4-4488-82b9-dd7cb9378e95", 1510 | "last_action": null, 1511 | "task": 401661, 1512 | "project": 183, 1513 | "updated_by": 2, 1514 | "parent_prediction": null, 1515 | "parent_annotation": null, 1516 | "last_created_by": null 1517 | } 1518 | ], 1519 | "file_upload": "65af2932-ls_input_data.json", 1520 | "drafts": [], 1521 | "predictions": [], 1522 | "data": { 1523 | "prompt": "What are the latest developments in space exploration?", 1524 | "answer1": "The U.S. has spent more than $4 trillion in space exploration spending over the last 40 years, which includes the use of commercial spacecraft, space stations and space shuttles. Space", 1525 | "answer2": "I am thrilled that the ISS is on the move to the U.S., along with the SES and ISS. We are doing the most exciting things possible using space. NASA has shown that" 1526 | }, 1527 | "meta": {}, 1528 | "created_at": "2023-04-17T21:53:56.691280Z", 1529 | "updated_at": "2023-04-17T21:55:28.947885Z", 1530 | "inner_id": 27, 1531 | "total_annotations": 1, 1532 | "cancelled_annotations": 0, 1533 | "total_predictions": 0, 1534 | "comment_count": 0, 1535 | "unresolved_comment_count": 0, 1536 | "last_comment_updated_at": null, 1537 | "project": 183, 1538 | "updated_by": 2, 1539 | "comment_authors": [] 1540 | }, 1541 | { 1542 | "id": 401662, 1543 | "annotations": [ 1544 | { 1545 | "id": 68072, 1546 | "completed_by": 2, 1547 | "result": [ 1548 | { 1549 | "value": { 1550 | "selected": "left" 1551 | }, 1552 | "id": "kgx38uCM0g", 1553 | "from_name": "comparison", 1554 | "to_name": "comparison", 1555 | "type": "pairwise", 1556 | "origin": "manual" 1557 | } 1558 | ], 1559 | "was_cancelled": false, 1560 | "ground_truth": false, 1561 | "created_at": "2023-04-17T21:55:30.216583Z", 1562 | "updated_at": "2023-04-17T21:55:30.216617Z", 1563 | "lead_time": 1.149, 1564 | "prediction": {}, 1565 | "result_count": 0, 1566 | "unique_id": "6c2bb7d8-422b-4188-8ee4-6227b20e407f", 1567 | "last_action": null, 1568 | "task": 401662, 1569 | "project": 183, 1570 | "updated_by": 2, 1571 | "parent_prediction": null, 1572 | "parent_annotation": null, 1573 | "last_created_by": null 1574 | } 1575 | ], 1576 | "file_upload": "65af2932-ls_input_data.json", 1577 | "drafts": [], 1578 | "predictions": [], 1579 | "data": { 1580 | "prompt": "What are some top-rated online courses or certifications?", 1581 | "answer1": "The major online programs for a bachelor's degree typically provide a degree from a private institution—usually in finance. Most programs, especially those funded by federal funding, offer only one online", 1582 | "answer2": "If you want to learn the business of finance, you need to have a good one.\n\nHere are seven of the best things you can do in the finance business in 6 months." 1583 | }, 1584 | "meta": {}, 1585 | "created_at": "2023-04-17T21:53:56.691420Z", 1586 | "updated_at": "2023-04-17T21:55:30.250052Z", 1587 | "inner_id": 28, 1588 | "total_annotations": 1, 1589 | "cancelled_annotations": 0, 1590 | "total_predictions": 0, 1591 | "comment_count": 0, 1592 | "unresolved_comment_count": 0, 1593 | "last_comment_updated_at": null, 1594 | "project": 183, 1595 | "updated_by": 2, 1596 | "comment_authors": [] 1597 | }, 1598 | { 1599 | "id": 401663, 1600 | "annotations": [ 1601 | { 1602 | "id": 68073, 1603 | "completed_by": 2, 1604 | "result": [ 1605 | { 1606 | "value": { 1607 | "selected": "left" 1608 | }, 1609 | "id": "hSCq5Q3xX4", 1610 | "from_name": "comparison", 1611 | "to_name": "comparison", 1612 | "type": "pairwise", 1613 | "origin": "manual" 1614 | } 1615 | ], 1616 | "was_cancelled": false, 1617 | "ground_truth": false, 1618 | "created_at": "2023-04-17T21:55:31.508164Z", 1619 | "updated_at": "2023-04-17T21:55:31.508191Z", 1620 | "lead_time": 1.144, 1621 | "prediction": {}, 1622 | "result_count": 0, 1623 | "unique_id": "27846daf-1423-4545-b434-69bffc627005", 1624 | "last_action": null, 1625 | "task": 401663, 1626 | "project": 183, 1627 | "updated_by": 2, 1628 | "parent_prediction": null, 1629 | "parent_annotation": null, 1630 | "last_created_by": null 1631 | } 1632 | ], 1633 | "file_upload": "65af2932-ls_input_data.json", 1634 | "drafts": [], 1635 | "predictions": [], 1636 | "data": { 1637 | "prompt": "What are some effective ways to improve public speaking skills?", 1638 | "answer1": "There are many ways to improve public speaking skills. To start, ask yourself when you have used a word or phrase. Can I use it for the context you like better than your preferred", 1639 | "answer2": "There are many ways an entrepreneur could improve their public speaking skills. For example, start early with a good public speaking strategy that includes:\n\nUsing a wide range of media" 1640 | }, 1641 | "meta": {}, 1642 | "created_at": "2023-04-17T21:53:56.691541Z", 1643 | "updated_at": "2023-04-17T21:55:31.538016Z", 1644 | "inner_id": 29, 1645 | "total_annotations": 1, 1646 | "cancelled_annotations": 0, 1647 | "total_predictions": 0, 1648 | "comment_count": 0, 1649 | "unresolved_comment_count": 0, 1650 | "last_comment_updated_at": null, 1651 | "project": 183, 1652 | "updated_by": 2, 1653 | "comment_authors": [] 1654 | }, 1655 | { 1656 | "id": 401664, 1657 | "annotations": [ 1658 | { 1659 | "id": 68074, 1660 | "completed_by": 2, 1661 | "result": [ 1662 | { 1663 | "value": { 1664 | "selected": "right" 1665 | }, 1666 | "id": "7HbWkmQ5sx", 1667 | "from_name": "comparison", 1668 | "to_name": "comparison", 1669 | "type": "pairwise", 1670 | "origin": "manual" 1671 | } 1672 | ], 1673 | "was_cancelled": false, 1674 | "ground_truth": false, 1675 | "created_at": "2023-04-17T21:55:33.479101Z", 1676 | "updated_at": "2023-04-17T21:55:33.479151Z", 1677 | "lead_time": 1.814, 1678 | "prediction": {}, 1679 | "result_count": 0, 1680 | "unique_id": "afde71ec-02e0-4381-a13c-92ed8323d245", 1681 | "last_action": null, 1682 | "task": 401664, 1683 | "project": 183, 1684 | "updated_by": 2, 1685 | "parent_prediction": null, 1686 | "parent_annotation": null, 1687 | "last_created_by": null 1688 | } 1689 | ], 1690 | "file_upload": "65af2932-ls_input_data.json", 1691 | "drafts": [], 1692 | "predictions": [], 1693 | "data": { 1694 | "prompt": "What are the latest trends in digital marketing?", 1695 | "answer1": "\"I'm talking about the business models\"\n\nFor most industries, media consumption is the main drivers of advertising revenue. However, not many companies focus just on targeting new audiences. Advertis", 1696 | "answer2": "The data is based on hundreds of studies and several thousand individual surveys. Here's a brief update on today:\n\nWhen it comes to digital marketing, it seems that consumers are increasingly consuming data from a" 1697 | }, 1698 | "meta": {}, 1699 | "created_at": "2023-04-17T21:53:56.691654Z", 1700 | "updated_at": "2023-04-17T21:55:33.510871Z", 1701 | "inner_id": 30, 1702 | "total_annotations": 1, 1703 | "cancelled_annotations": 0, 1704 | "total_predictions": 0, 1705 | "comment_count": 0, 1706 | "unresolved_comment_count": 0, 1707 | "last_comment_updated_at": null, 1708 | "project": 183, 1709 | "updated_by": 2, 1710 | "comment_authors": [] 1711 | }, 1712 | { 1713 | "id": 401665, 1714 | "annotations": [ 1715 | { 1716 | "id": 68075, 1717 | "completed_by": 2, 1718 | "result": [ 1719 | { 1720 | "value": { 1721 | "selected": "right" 1722 | }, 1723 | "id": "ab7Zy36b5G", 1724 | "from_name": "comparison", 1725 | "to_name": "comparison", 1726 | "type": "pairwise", 1727 | "origin": "manual" 1728 | } 1729 | ], 1730 | "was_cancelled": false, 1731 | "ground_truth": false, 1732 | "created_at": "2023-04-17T21:55:34.483377Z", 1733 | "updated_at": "2023-04-17T21:55:34.483404Z", 1734 | "lead_time": 0.855, 1735 | "prediction": {}, 1736 | "result_count": 0, 1737 | "unique_id": "f30e22f4-d3b5-4961-b99d-01760c0062da", 1738 | "last_action": null, 1739 | "task": 401665, 1740 | "project": 183, 1741 | "updated_by": 2, 1742 | "parent_prediction": null, 1743 | "parent_annotation": null, 1744 | "last_created_by": null 1745 | } 1746 | ], 1747 | "file_upload": "65af2932-ls_input_data.json", 1748 | "drafts": [], 1749 | "predictions": [], 1750 | "data": { 1751 | "prompt": "What are some fun and creative DIY projects?", 1752 | "answer1": "Check out this post.", 1753 | "answer2": "Let's start with the DIY project of creating a simple 3D printed wooden bar. A bar is created by placing a wooden handle and a wire, then placing a wire in the bottom of the" 1754 | }, 1755 | "meta": {}, 1756 | "created_at": "2023-04-17T21:53:56.691759Z", 1757 | "updated_at": "2023-04-17T21:55:34.517308Z", 1758 | "inner_id": 31, 1759 | "total_annotations": 1, 1760 | "cancelled_annotations": 0, 1761 | "total_predictions": 0, 1762 | "comment_count": 0, 1763 | "unresolved_comment_count": 0, 1764 | "last_comment_updated_at": null, 1765 | "project": 183, 1766 | "updated_by": 2, 1767 | "comment_authors": [] 1768 | }, 1769 | { 1770 | "id": 401666, 1771 | "annotations": [ 1772 | { 1773 | "id": 68076, 1774 | "completed_by": 2, 1775 | "result": [ 1776 | { 1777 | "value": { 1778 | "selected": "left" 1779 | }, 1780 | "id": "k5M8iU0hHw", 1781 | "from_name": "comparison", 1782 | "to_name": "comparison", 1783 | "type": "pairwise", 1784 | "origin": "manual" 1785 | } 1786 | ], 1787 | "was_cancelled": false, 1788 | "ground_truth": false, 1789 | "created_at": "2023-04-17T21:55:35.850690Z", 1790 | "updated_at": "2023-04-17T21:55:35.850713Z", 1791 | "lead_time": 1.228, 1792 | "prediction": {}, 1793 | "result_count": 0, 1794 | "unique_id": "d6bd06b1-dcb9-4d7a-baf3-3d3c58e2ec6b", 1795 | "last_action": null, 1796 | "task": 401666, 1797 | "project": 183, 1798 | "updated_by": 2, 1799 | "parent_prediction": null, 1800 | "parent_annotation": null, 1801 | "last_created_by": null 1802 | } 1803 | ], 1804 | "file_upload": "65af2932-ls_input_data.json", 1805 | "drafts": [], 1806 | "predictions": [], 1807 | "data": { 1808 | "prompt": "What are some effective ways to improve leadership skills?", 1809 | "answer1": "1. Encourage people to stay accountable to themselves\n\nWhen leaders behave themselves in a negative way, people often don't want to do things the way they wanted to do, they want", 1810 | "answer2": "To get to the bottom of leadership skills questions, in this post, we're going to use the most basic definitions for leadership in a career path (based on my personal experience):" 1811 | }, 1812 | "meta": {}, 1813 | "created_at": "2023-04-17T21:53:56.691863Z", 1814 | "updated_at": "2023-04-17T21:55:35.881548Z", 1815 | "inner_id": 32, 1816 | "total_annotations": 1, 1817 | "cancelled_annotations": 0, 1818 | "total_predictions": 0, 1819 | "comment_count": 0, 1820 | "unresolved_comment_count": 0, 1821 | "last_comment_updated_at": null, 1822 | "project": 183, 1823 | "updated_by": 2, 1824 | "comment_authors": [] 1825 | } 1826 | ] -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HumanSignal/RLHF/cc3fa781049fee796ace5a22c7ae51abfe87fb67/utils/__init__.py -------------------------------------------------------------------------------- /utils/elo.py: -------------------------------------------------------------------------------- 1 | def elo_probability(rating1, rating2, k=10): 2 | """ 3 | Calculate the probability of winning for two players based on their Elo ratings. 4 | 5 | :param rating1: Elo rating of player 1 6 | :param rating2: Elo rating of player 2 7 | :param k: scaling factor 8 | :return: probability of player 1 winning 9 | """ 10 | prob = 1 / (1 + 10 ** ((rating2 - rating1) / (k * 1.0))) 11 | return prob 12 | 13 | 14 | def elo_update(rating, opponent_rating, result, k=32): 15 | """ 16 | Update the Elo rating for a player based on the result of a game. 17 | 18 | :param rating: player's current Elo rating 19 | :param opponent_rating: opponent's Elo rating 20 | :param result: result of the game (1 for win, 0.5 for draw, 0 for loss) 21 | :param k: scaling factor 22 | :return: updated Elo rating 23 | """ 24 | expected_win_probability = elo_probability(rating, opponent_rating, k) 25 | updated_rating = rating + k * (result - expected_win_probability) 26 | return updated_rating 27 | 28 | 29 | def elo_process_comparison_matrix(matrix, initial_rating=1500, k=32): 30 | n = len(matrix) 31 | ratings = [initial_rating] * n 32 | 33 | for i in range(n): 34 | for j in range(i + 1, n): 35 | result = matrix[i][j] 36 | 37 | if result is not None: 38 | ratings[i] = elo_update(ratings[i], ratings[j], result, k) 39 | ratings[j] = elo_update(ratings[j], ratings[i], 1 - result, k) 40 | 41 | # Sort strings by their rating, descending 42 | sorted_indices = sorted(range(n), key=lambda i: ratings[i], reverse=True) 43 | 44 | return sorted_indices, [ratings[i] for i in sorted_indices] 45 | 46 | 47 | if __name__ == "__main__": 48 | # Example usage 49 | matrix = [ 50 | [None, 1, 0, None], 51 | [0, None, 1, 1], 52 | [1, 0, None, 0], 53 | [None, 0, 1, None] 54 | ] 55 | 56 | sorted_indices, sorted_ratings = elo_process_comparison_matrix(matrix) 57 | 58 | print("Sorted indices:", sorted_indices) 59 | print("Sorted ratings:", sorted_ratings) 60 | --------------------------------------------------------------------------------