├── .gitattributes ├── CHANGELOG.md ├── README.md └── script.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this file. 3 | 4 | ## 0.3.0 - 19 November 2023 5 | ### Added 6 | - Overhauled the message multiplier system to be more flexible and hopefully capable of producing better results 7 | - New setting `max_messages`: This is the maximum number of recent messages that will be injected into the negative prompt 8 | - New setting `max_multiplier`: This is the maximum multiplier that will be applied to a message (in other words, the repeat count) 9 | - New setting `scaling`: This is the scaling algorithm that will be applied to the multiplier for each message, currently supports `constant`, `linear`, `exponential` and `logarithmic` 10 | 11 | ### Removed 12 | - `history_multiplier` setting 13 | - `last_message_multiplier` setting 14 | 15 | ## 0.2.0 - 18 November 2023 16 | ### Added 17 | - Default tab support 18 | - Notebook tab support 19 | - Print debug info for Blacklist 20 | - New setting `context_delimiter`: Allows you to specify expected history format for Notebook and Default tabs, defaults to `\n` 21 | 22 | ### Changed 23 | - Renamed `Message Delimiter` to `Negative Delimiter` 24 | - Blank history messages are now ignored 25 | 26 | ## 0.1.0 - 16 November 2023 27 | ### Added 28 | - New Blacklist feature: Allows you to exclude certain terms from being injected into your negative prompt, with support for * wildcard and regex 29 | - `CHANGELOG.md` to track changes 30 | 31 | ### Changed 32 | - Increased default `history_multiplier` from 0 to 1 (I'm still experimenting with different defaults; your feedback is appreciated) 33 | 34 | ## 0.0.2 - 15 November 2023 35 | ### Fixed 36 | - Fixed missing key for `enable` parameter 37 | - Fixed `history_length` datatype issue 38 | - Corrected typos in README 39 | 40 | ## 0.0.1 - 15 November 2023 41 | ### Added 42 | - Initial release -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Echoproof 2 | 3 | Echoproof is a simple extension for Ooobabooga's [text-generation-webui](https://github.com/oobabooga/text-generation-webui) that injects recent conversation history into the negative prompt with the goal of minimizing the LLM's tendency to fixate on a single word, phrase, or sentence structure. 4 | 5 | ## The problem 6 | 7 | I have observed that certain tokens will cause LLMs to exhibit an "OCD-like" behavior where future messages become progressively more repetitive. If you are not familiar with this effect, try appending a bunch of emoji 👀😲😔 to a chatbot's reply or forcing it to write in ALL CAPS - it will become a broken record very quickly. 8 | 9 | This is certainly true of quantized Llama 2 models in the 7b to 30b parameter range - I'm guessing it's less prevalent in 70b models, but I don't have the hardware to test that. 10 | 11 | Existing solutions to address this problem, such as `repetition_penalty`, have shown limited success. 12 | 13 | This issue can derail a conversation well before the context window is exhausted, so I believe it is unrelated to another known phenomenon where a model will descend into a "word salad" state once the chat has gone on for too long. 14 | 15 | ## The solution (?) 16 | 17 | What if we just inject the last thing the chatbot said into the negative prompt for its next message? That was the main idea behind Echoproof, and it seems to work pretty well. 18 | 19 | A few weeks of testing have led me to refine this approach with these additional features: 20 | 21 | ### Max Messages 22 | The maximum number of recent messages that will be injected into the negative prompt. This is a hard limit, so if you set it to 5 and your chat history is 10 messages long, only the last 5 messages will be injected. 23 | 24 | ### Max Multiplier 25 | The maximum multiplier that will be applied to a message (in other words, the repeat count). I have found that passing a message into the negative prompt only once is not enough to offset the OCD effect, but repeating it 3-5 times makes a noticeable difference. 26 | 27 | ### Scaling 28 | This is the scaling algorithm that will be applied to the multiplier for each message. You can choose between `constant`, `linear`, `exponential` and `logarithmic`. 29 | 30 | ### Notebook and Default Tab Support 31 | 32 | Originally, this extension only supported Chat mode, but it has since been updated to support other modes as well. The `context_delimiter` setting allows you to specify the expected message separator for Notebook and Default tabs, which defaults to `\n`. 33 | 34 | ### Blacklist 35 | 36 | This feature allows you to exclude certain terms from being injected into your negative prompt, with support for `*` wildcard and regex. This is useful if you want to preserve the LLM's understanding of certain words or phrases, for example in complex multi-character scenarios. 37 | 38 | ## How to install 39 | 40 | Paste the URL of this repo into the "Session" tab of the webui and press enter. Go get a cup of coffee. 41 | 42 | ## How to use 43 | 44 | Load a model with `cfg-cache` enabled and set your `guidance_scale` to a value above 1 in the "Parameters" tab. Otherwise, your negative prompt will not have an effect. -------------------------------------------------------------------------------- /script.py: -------------------------------------------------------------------------------- 1 | """ 2 | Simple extension that injects recent conversation into the negative prompt in order to reduce OCD-like behavior of the assistant fixating on a single word, phrase, or sentence structure. 3 | """ 4 | 5 | import gradio as gr 6 | import torch, math 7 | from transformers import LogitsProcessor 8 | 9 | from modules import chat, shared 10 | from modules.text_generation import ( 11 | decode, 12 | encode, 13 | generate_reply, 14 | ) 15 | 16 | params = { 17 | "display_name": "Echoproof", 18 | "is_tab": False, 19 | "debug": False, 20 | "context_delimiter":"\\n", 21 | "negative_delimiter":" ", 22 | "max_messages":10, 23 | "max_multiplier":4, 24 | "scaling":"exponential", 25 | "enable": True, 26 | "blacklist":"", 27 | "tab":"chat" 28 | } 29 | 30 | VERSION = "0.3.0" 31 | 32 | def state_modifier(state): 33 | """ 34 | Modifies the state variable, which is a dictionary containing the input 35 | values in the UI like sliders and checkboxes. 36 | """ 37 | 38 | # print("Call to `state_modifier()`") 39 | # print(f"State: {state}") 40 | 41 | if params["enable"]: 42 | if params["tab"]=="chat": 43 | internal_history = state["history"]["internal"] 44 | elif params["tab"]=="notebook": 45 | internal_history = state["textbox-notebook"].split("\n") 46 | elif params["tab"]=="default": 47 | internal_history = state["textbox-default"].split("\n") 48 | 49 | if params["blacklist"]: 50 | import re 51 | blacklist = params["blacklist"].split("\n") 52 | for term in blacklist: 53 | # Convert glob-style wildcard to regex-style 54 | term = term.replace("*", ".*") 55 | for idx, msg in enumerate(internal_history): 56 | if params["tab"]=="chat": msg = msg[1] 57 | new_msg = re.sub(term, "", msg) 58 | if params["debug"] and msg != new_msg: print(f"Replaced `{msg}` with `{new_msg}`") 59 | msg = new_msg 60 | 61 | if params["tab"]=="chat": internal_history[idx] = (internal_history[idx][0], msg) 62 | else: internal_history[idx] = msg 63 | 64 | # Remove empty strings from internal_history 65 | internal_history = list(filter(None, internal_history)) 66 | 67 | history_length = len(internal_history) 68 | 69 | _min = max(0,history_length - params["max_messages"]) if params["max_messages"] else 0 70 | 71 | extra_neg = "" 72 | total_messages = history_length - _min 73 | for idx in range(_min,history_length): 74 | msg = internal_history[idx] 75 | if params["tab"]=="chat": msg = msg[1] 76 | 77 | # Linear scaled repeats 78 | relative_idx = idx - _min + 1 79 | 80 | if params["scaling"]=="constant": 81 | multiplier = params["max_multiplier"] 82 | elif params["scaling"]=="linear": 83 | multiplier = round(relative_idx / total_messages * params["max_multiplier"]) 84 | elif params["scaling"]=="exponential": 85 | multiplier = round(params["max_multiplier"] * math.pow((relative_idx / total_messages), 2)) 86 | elif params["scaling"]=="logarithmic": 87 | multiplier = round(params["max_multiplier"] * math.log(relative_idx + 1) / math.log(total_messages + 1)) 88 | 89 | if params["debug"]: print(f"Multiplier for message #{relative_idx}/{total_messages}: {multiplier}") 90 | 91 | extra_neg += (msg + params["negative_delimiter"]) * multiplier 92 | 93 | if params["debug"]: 94 | print(f"Value of `extra_neg`: {extra_neg}") 95 | 96 | state["negative_prompt"] += extra_neg 97 | 98 | return state 99 | 100 | def ui(): 101 | """ 102 | Gets executed when the UI is drawn. Custom gradio elements and 103 | their corresponding event handlers should be defined here. 104 | 105 | To learn about gradio components, check out the docs: 106 | https://gradio.app/docs/ 107 | """ 108 | with gr.Accordion(f"Echoproof v{VERSION}", open=False): 109 | 110 | gr.Markdown("**Note:** You must load a model with `cfg-cache` enabled and set `guidance_scale` to a value > 1 for Echoproof to take effect.") 111 | 112 | with gr.Row(): 113 | enable = gr.Checkbox(value=True,label="Enable") 114 | enable.change(lambda x: params.update({"enable": x}), enable, None) 115 | 116 | debug = gr.Checkbox(value=False,label="Debug") 117 | debug.change(lambda x: params.update({"debug": x}), debug, None) 118 | 119 | 120 | with gr.Row(): 121 | max_messages = gr.Slider(1, 50, label="Max Messages", info="Adds the most recent x messages to your negative prompt.", value=10, step=1) 122 | max_messages.change(lambda x: params.update({"max_messages": x}), max_messages, None) 123 | 124 | max_multiplier = gr.Slider(0, 50, label="Max Multiplier", info="The maximum number of times a message will be added to the negative prompt.", value=4, step=1) 125 | max_multiplier.change(lambda x: params.update({"max_multiplier": x}), max_multiplier, None) 126 | 127 | scaling = gr.Radio(value="exponential",label="Scaling",info="How the multiplier scales with the number of messages.",choices=["constant","linear","exponential","logarithmic"]) 128 | scaling.change(lambda x: params.update({"scaling": x}), scaling, None) 129 | 130 | 131 | with gr.Row(): 132 | context_delimiter = gr.Textbox(value="\\n",label="Context Delimiter",info="Expected history format for Default and Notebook tabs.") 133 | context_delimiter.change(lambda x: params.update({"context_delimiter": x}), context_delimiter, None) 134 | 135 | delimiter = gr.Textbox(value=" ",label="Negative Delimiter",info="String that separates each message in the negative prompt.") 136 | delimiter.change(lambda x: params.update({"negative_delimiter": x}), delimiter, None) 137 | 138 | blacklist = gr.Textbox(value="",lines=3,max_lines=100,label="Blacklist",info="These terms will be excluded from being injected into your negative prompt. Enter one term per line. Supports `*` wildcard as well as regex.") 139 | blacklist.change(lambda x: params.update({"blacklist": x}), blacklist, None) 140 | 141 | tab = gr.Radio(value="chat",label="Tab",info="Due to a limitation of the extension framework, you must specify the current WebUI tab here.",choices=["chat","default","notebook"]) 142 | tab.change(lambda x: params.update({"tab": x}), tab, None) 143 | 144 | gr.Markdown("If you find this project useful, consider [supporting my work here](https://github.com/sponsors/ThereforeGames). Thank you. ❤️") --------------------------------------------------------------------------------