├── README.md ├── gpt-comment-detector.js └── screenshot ├── screenshot1.png ├── screenshot2.png └── screenshot3.png /README.md: -------------------------------------------------------------------------------- 1 | # GPTCommentDetector 2 | 3 | A UserScript to detect GPT generated comments on Hackernews. 4 | 5 | ![image](https://github.com/chryzsh/GPTCommentDetector/raw/main/screenshot/screenshot1.png) 6 | 7 | This is a browser UserScript that tries to detect comments on [Hacker News](https://news.ycombinator.com/) that are GPT/OpenAI generated [ChatGPT](https://chatgpt.com/). It is based on [GPT output detector](https://huggingface.co/openai-detector). 8 | 9 | # How it works 10 | 11 | It puts every comment into the [GPT output detector](https://huggingface.co/openai-detector) and colors and writes a short comment on the HN comment based on its assessment, like you see in the screenshot. This is based on a threshold set by me. >0.7 is probably AI, >0.9 is definitely AI. Lower than that is most likely human. Most comments on HN still appear to be human. 12 | 13 | It only becomes reliable after about 50 tokens (one token is around 4 characters) so I mark the comments that are too short with gray and make no assessment on those. 14 | 15 | Remember that the point is to detect comments obviously written completely by GPT, not detecting that a human a wrote it. You have to discern between the two, as the scale is not really utilized here. What I've found over the last few days of playing with this is that most AI generated text will almost always be identified with 99.97% probability , while human text ranges somewhere between 70-98. So I could move the threshold for my script quite a lot. I just wanted to see how it works with these thresholds for a while. 16 | 17 | # Disclaimer 18 | 19 | I know pretty much nothing about Javascript. This is shitty code largely written by ChatGPT itself, untested beyond Chrome on MacOS with Violenmonkey. I have no plans for maintenance or extending it. If you want features or support, you have to ask ChatGPT, not me. 20 | 21 | It's just sending a HTTP request to https://huggingface.co/openai-detector/ with the contents of a comment. This have some privacy, performance and data usage concerns. Additionally I am guessing that there is some rate limiting to using huggingface's service for free. 22 | 23 | # Install 24 | 25 | ## Prerequisites 26 | 27 | Install **[Violentmonkey](https://violentmonkey.github.io/)** 28 | 29 | Alternative: **Tampermonkey** **（[Chrome](https://www.tampermonkey.net/)** / **[Firefox](https://addons.mozilla.org/firefox/addon/tampermonkey/)** / **[Edge](https://microsoftedge.microsoft.com/addons/detail/tampermonkey/iikmkjmpaadaobahmlepeloendndfphd?hl=zh-CN)）** 30 | 31 | 32 | ## UserScript 33 | 34 | | Greasyfork | GitHub | 35 | | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | 36 | | [GreasyFork](https://greasyfork.org/en/scripts/456398-gpt-comment-detector) | [Github](https://github.com/chryzsh/GPTCommentDetector/raw/main/gpt-comment-detector.js) 37 | 38 | # Example 39 | 40 | ## Screenshots 41 | ![Detected](https://github.com/chryzsh/GPTCommentDetector/raw/main/screenshot/screenshot3.png) 42 | ![Detected](https://github.com/chryzsh/GPTCommentDetector/raw/main/screenshot/screenshot2.png) 43 | **** -------------------------------------------------------------------------------- /gpt-comment-detector.js: -------------------------------------------------------------------------------- 1 | // ==UserScript== 2 | // @name GPT Comment Detector 3 | // @namespace https://example.com/ 4 | // @version 0.1 5 | // @description Identify AI-generated comments on Hacker News 6 | // @author chryzsh 7 | // @match https://news.ycombinator.com/* 8 | // @grant none 9 | // @connect huggingface.co 10 | // @grant GM.xmlHttpRequest 11 | // @grant unsafeWindow 12 | // @sandbox JavaScript 13 | // @license MIT 14 | // ==/UserScript== 15 | 16 | // Set a threshold for the minimum probability that a comment is AI-generated 17 | const AI_THRESHOLD = 0.99; 18 | const PROBABLE_THRESHOLD = 0.7; 19 | const TOKEN_THRESHOLD = 200; //4 characters are about 50 tokens, which is when the detector model because reliable 20 | 21 | // put all comments into a NodeList 22 | const comments = document.querySelectorAll('.comment'); 23 | 24 | // regex to remove whitespaces 25 | const whitespaceRegex = /\s+/g; 26 | 27 | //regex to remove "reply" at the end of comments 28 | const replyRegex = /\breply\b\s*/g; 29 | 30 | // kick off the program 31 | main(); 32 | 33 | function main() { 34 | // loop over each comment 35 | comments.forEach((comment) => { 36 | // add an onclick for each comment 37 | comment.addEventListener('click', function () { 38 | // trim commments for whitespaces and test length for each comment 39 | testLength(comment); 40 | }); 41 | }); 42 | }; 43 | 44 | // Trim and test the length of comments 45 | function testLength(comment) { 46 | var text = comment.textContent; 47 | 48 | // trim whitespaces and shit from the end first - dont do this on the comment object itself, just a variable 49 | text = comment.textContent.trim(); 50 | text = comment.textContent.replace(whitespaceRegex, " "); 51 | 52 | // remove "reply" from the end 53 | text = comment.textContent.replace(replyRegex, ""); 54 | 55 | // test the length first so we don't run detection on too short comments 56 | if (text.length < TOKEN_THRESHOLD) { 57 | comment.style.border = '1px solid gray'; 58 | comment.innerHTML += `
Insufficient data to assess if AI generated
`; 59 | } 60 | else { 61 | // run AI detection and add comment based on the result 62 | detectAI(comment.textContent, function (result) { 63 | addComment(result, comment); 64 | }); 65 | 66 | } 67 | } 68 | 69 | // Add comment 70 | function addComment(result, comment) { 71 | var json = JSON.parse(result); 72 | var fakeProbability = json.fake_probability; 73 | var realProbability = json.real_probability; 74 | var formattedNumberAI = (fakeProbability * 100).toFixed(3); 75 | var formattedNumberHuman = (realProbability * 100).toFixed(3); 76 | 77 | // most comments will not be AI (hopefully) so start with testing that 78 | if (fakeProbability < PROBABLE_THRESHOLD) { //assume not AI if lower than 0.7 on the AI-meter 79 | comment.style.border = '1px solid green'; 80 | comment.innerHTML += `
Definitely human - human probability ${formattedNumberHuman}
`; 81 | } 82 | // If the probability is above the threshold, label the comment as AI-generated 83 | else if (fakeProbability > PROBABLE_THRESHOLD) { // possibly AI 84 | if (fakeProbability > AI_THRESHOLD) { //definitely AI 85 | 86 | comment.style.border = '1px solid red'; 87 | comment.innerHTML += `
Definitely AI - AI probability ${formattedNumberAI}
`; 88 | } 89 | else { 90 | comment.style.border = '1px solid yellow'; 91 | comment.innerHTML += `
Possibly AI - AI probability ${formattedNumberAI}
`; 92 | } 93 | } 94 | }; 95 | 96 | // Run AI detection 97 | function detectAI(input, callback) { 98 | var detectorUrlWithInput = "https://huggingface.co/openai-detector?" + input 99 | GM.xmlHttpRequest({ 100 | method: "GET", 101 | url: detectorUrlWithInput, 102 | headers: { 103 | "User-Agent": "Mozilla/5.0", // If not specified, navigator.userAgent will be used. 104 | "Content-Type": "application/x-www-form-urlencoded" 105 | }, 106 | 107 | onload: function (response) { 108 | var responseParsed = [response.responseText].join("\n"); 109 | callback(responseParsed); 110 | } 111 | }); 112 | } 113 | -------------------------------------------------------------------------------- /screenshot/screenshot1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chryzsh/GPTCommentDetector/6fd6fc1daf22dc200b63e4a57a3a035d35b411a7/screenshot/screenshot1.png -------------------------------------------------------------------------------- /screenshot/screenshot2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chryzsh/GPTCommentDetector/6fd6fc1daf22dc200b63e4a57a3a035d35b411a7/screenshot/screenshot2.png -------------------------------------------------------------------------------- /screenshot/screenshot3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chryzsh/GPTCommentDetector/6fd6fc1daf22dc200b63e4a57a3a035d35b411a7/screenshot/screenshot3.png --------------------------------------------------------------------------------