├── src ├── parsers │ ├── utils │ │ ├── __init__.py │ │ ├── utils.py │ │ ├── mlc_utils.py │ │ └── llamacpp_utils.py │ ├── README.md │ ├── scripts │ │ └── logcat_getter.sh │ ├── parser.py │ └── notebooks │ │ └── per_op_plotting.ipynb ├── model_evaluation │ ├── .gitignore │ ├── src │ │ ├── evaluate_awq.sh │ │ ├── evaluate_gptq.sh │ │ ├── evaluate_hf.sh │ │ ├── download_gptq_awq_models.sh │ │ ├── README.md │ │ ├── evaluate_llamacpp.sh │ │ └── eval_requirements.txt │ ├── README.md │ └── results │ │ ├── model_specs.csv │ │ ├── visualization.ipynb │ │ └── results.csv ├── models │ ├── convert_utils │ │ ├── __init__.py │ │ ├── awq_utils.py │ │ ├── mlc_utils.py │ │ └── llama_utils.py │ ├── requirements.txt │ ├── scripts │ │ ├── replace_link_with_model.sh │ │ ├── download_all_models.sh │ │ ├── convert_new.sh │ │ └── convert_legacy.sh │ ├── download.py │ ├── README.md │ └── convert.py ├── prompts │ ├── .gitignore │ ├── requirements.txt │ ├── README.md │ ├── notebooks │ │ ├── oasst_prompts.ipynb │ │ └── prompt_statistics.ipynb │ ├── input_sustained.json │ └── conversations.json └── configs │ ├── README.md │ ├── llama-2-7b-hf.yaml │ ├── stablelm-zephyr-3b.yaml │ ├── gemma-2b-it.yaml │ ├── gemma-7b-it.yaml │ ├── example_config.yaml │ ├── tinyllama-1.1b-chat-v0.5.yaml │ ├── llama-2-13b-chat-hf.yaml │ ├── llama-2-7b-chat-hf.yaml │ └── mistral-7b-instruct-v0.1.yaml ├── .gitmodules ├── LICENSE ├── .gitignore └── README.md /src/parsers/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/model_evaluation/.gitignore: -------------------------------------------------------------------------------- 1 | logs/ -------------------------------------------------------------------------------- /src/models/convert_utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/prompts/.gitignore: -------------------------------------------------------------------------------- 1 | results/ 2 | -------------------------------------------------------------------------------- /src/models/requirements.txt: -------------------------------------------------------------------------------- 1 | numpy==1.24 2 | huggingface_hub 3 | sentencepiece 4 | transformers -------------------------------------------------------------------------------- /src/prompts/requirements.txt: -------------------------------------------------------------------------------- 1 | datasets 2 | notebook 3 | nltk 4 | pandas 5 | numpy 6 | matplotlib 7 | seaborn -------------------------------------------------------------------------------- /src/models/scripts/replace_link_with_model.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to replace softlinks with models. 4 | # Author: Stefanos Laskaridis (stefanos@brave.com) 5 | 6 | TARGET_DIR=$1 7 | 8 | pushd $TARGET_DIR 9 | for file in $(ls);do 10 | if [ -h $file ]; then 11 | # replace link with model 12 | echo "Replacing softlink for file $file" 13 | cp --remove-destination `readlink $file` $file 14 | fi 15 | done 16 | popd -------------------------------------------------------------------------------- /src/configs/README.md: -------------------------------------------------------------------------------- 1 | # Model Config Files 2 | 3 | These are configuration files that describe various model and execution level parameters for how to run the models. You can think of them as hyperparameters in some cases. 4 | 5 | They are typically initialised by what is found in `frameworks/MLC/mlc-llm/cpp/conv_templates.cc`, so as to provide a common ground between frameworks. 6 | 7 | These are later used by jetsonlab and phonelab to be translated into target specific configuration parameters, such as CLI arguments in the case of llama.cpp or JSON configs in the case of LLMFarm. -------------------------------------------------------------------------------- /src/prompts/README.md: -------------------------------------------------------------------------------- 1 | # MELT Prompts 2 | 3 | This directory contains the conversational data used to interact later on with the LLMs (i.e. in macro-experiments of the paper). 4 | 5 | 6 | ## Structure 7 | 8 | ```bash 9 | ├── conversations.json # The conversation used for regular macro experiments 10 | ├── input_sustained.json # The conversation used for sustained inference experiments 11 | ├── notebooks # Notebooks for extracting and analysing the src dataset 12 | ``` 13 | 14 | ## How to run 15 | 16 | ```bash 17 | pip install -r requirements.txt 18 | cd notebooks/ 19 | jupyter notebook 20 | ``` 21 | -------------------------------------------------------------------------------- /src/configs/llama-2-7b-hf.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | model: llama-2-7b-hf 4 | url: https://huggingface.co/meta-llama/Llama-2-7b-hf 5 | sampling: 6 | temperature: 0.7 7 | repetition_penalty: 1.0 8 | top_p: 0.95 9 | top_k: 40 # Applicable to llama.cpp only 10 | repeat_last_n: 64 # Applicable to llama.cpp only 11 | n_batch: 512 # Applicable to llama.cpp only 12 | generation: 13 | mean_gen_len: 128 # Applicable to mlc only 14 | max_gen_len: 256 15 | max_window_size: 4096 16 | vocab_size: 32000 # Applicable to mlc only 17 | prompt: # Used in llama.cpp, taken from mlc 18 | text: "" 19 | in_prefix: "" 20 | in_suffix: "" 21 | reverse: "" -------------------------------------------------------------------------------- /src/configs/stablelm-zephyr-3b.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | model: stablelm-zephyr-3b 4 | url: https://huggingface.co/stabilityai/stablelm-zephyr-3b 5 | sampling: 6 | temperature: 0.7 7 | repetition_penalty: 1.0 8 | top_p: 0.95 9 | top_k: 40 # Applicable to llama.cpp only 10 | repeat_last_n: 64 # Applicable to llama.cpp only 11 | n_batch: 512 # Applicable to llama.cpp only 12 | generation: 13 | mean_gen_len: 128 # Applicable to mlc only 14 | max_gen_len: 2048 15 | max_window_size: 512 16 | vocab_size: 50304 # Applicable to mlc only 17 | prompt: # Used in llama.cpp, taken from mlc 18 | conv_template: stablelm_zephyr_3b 19 | text: "\"\"" 20 | in_prefix: "\"<|user|>\\n\"" 21 | in_suffix: "\"<|endoftext|>\\n<|assistant|>\"" -------------------------------------------------------------------------------- /src/configs/gemma-2b-it.yaml: -------------------------------------------------------------------------------- 1 | model: google_gemma-2b-it 2 | url: https://huggingface.co/google/gemma-2b-it 3 | sampling: 4 | temperature: 0.7 5 | repetition_penalty: 1.0 6 | top_p: 0.95 7 | top_k: 40 # Applicable to llama.cpp only 8 | repeat_last_n: 64 # Applicable to llama.cpp only 9 | n_batch: 512 # Applicable to llama.cpp only 10 | generation: 11 | mean_gen_len: 128 # Applicable to mlc only 12 | max_gen_len: 256 13 | max_window_size: 4096 14 | vocab_size: 256000 # Applicable to mlc only 15 | prompt: # Used in llama.cpp, taken from mlc 16 | conv_template: "gemma_instruction" 17 | text: "\"\"" 18 | in_prefix: "\"user\\n\"" 19 | in_suffix: "\"\\n\"" 20 | reverse: "\"model\"" -------------------------------------------------------------------------------- /src/configs/gemma-7b-it.yaml: -------------------------------------------------------------------------------- 1 | model: google_gemma-7b-it 2 | url: https://huggingface.co/google/gemma-7b-it 3 | sampling: 4 | temperature: 0.7 5 | repetition_penalty: 1.0 6 | top_p: 0.95 7 | top_k: 40 # Applicable to llama.cpp only 8 | repeat_last_n: 64 # Applicable to llama.cpp only 9 | n_batch: 512 # Applicable to llama.cpp only 10 | generation: 11 | mean_gen_len: 128 # Applicable to mlc only 12 | max_gen_len: 256 13 | max_window_size: 4096 14 | vocab_size: 256000 # Applicable to mlc only 15 | prompt: # Used in llama.cpp, taken from mlc 16 | conv_template: "gemma_instruction" 17 | text: "\"\"" 18 | in_prefix: "\"user\\n\"" 19 | in_suffix: "\"\\n\"" 20 | reverse: "\"model\"" -------------------------------------------------------------------------------- /src/configs/example_config.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | model: model_alias 4 | url: https://huggingface.co/org/model_name 5 | sampling: 6 | temperature: 0.7 7 | repetition_penalty: 1.0 8 | top_p: 0.95 9 | top_k: 40 # Applicable to llama.cpp only 10 | repeat_last_n: 64 # Applicable to llama.cpp only 11 | n_batch: 512 # Applicable to llama.cpp only 12 | generation: 13 | mean_gen_len: 128 # Applicable to mlc only 14 | max_gen_len: 256 15 | max_window_size: 4096 16 | vocab_size: 32000 # Applicable to mlc only 17 | prompt: # Used in llama.cpp, taken from mlc 18 | conv_template: "template key from frameworks/MLC/mlc-llm/cpp/conv_templates.cc" 19 | text: "Your prompt text here" 20 | in_prefix: "" 21 | in_suffix: "" 22 | reverse: "" -------------------------------------------------------------------------------- /src/configs/tinyllama-1.1b-chat-v0.5.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | model: TinyLlama-1.1B-Chat-v0.5 4 | url: https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.5 5 | sampling: 6 | temperature: 0.7 7 | repetition_penalty: 1.0 8 | top_p: 0.95 9 | top_k: 40 # Applicable to llama.cpp only 10 | repeat_last_n: 64 # Applicable to llama.cpp only 11 | n_batch: 512 # Applicable to llama.cpp only 12 | generation: 13 | mean_gen_len: 128 # Applicable to mlc only 14 | max_gen_len: 2048 15 | max_window_size: 2048 16 | vocab_size: 32003 # Applicable to mlc only 17 | prompt: # Used in llama.cpp, taken from mlc 18 | conv_template: "chatml" 19 | text: "\"A conversation between a user and an LLM-based AI assistant. The assistant gives helpful and honest answers.\"" 20 | chatml: '' 21 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "frameworks/MLC/tvm-unity"] 2 | path = frameworks/MLC/tvm-unity 3 | url = git@github.com:brave-experiments/tvm-public 4 | [submodule "frameworks/MLC/mlc-llm"] 5 | path = frameworks/MLC/mlc-llm 6 | url = git@github.com:brave-experiments/mlc-llm-public 7 | [submodule "frameworks/llama.cpp/llama.cpp"] 8 | path = frameworks/llama.cpp/llama.cpp 9 | url = git@github.com:brave-experiments/llama.cpp-public 10 | [submodule "frameworks/llama.cpp/LLMFarmEval"] 11 | path = frameworks/llama.cpp/LLMFarmEval 12 | url = git@github.com:brave-experiments/LLMFarmEval-public 13 | [submodule "jetsonlab"] 14 | path = jetsonlab 15 | url = git@github.com:brave-experiments/jetsonlab-public 16 | [submodule "blade"] 17 | path = blade 18 | url = git@github.com:brave-experiments/blade-public 19 | -------------------------------------------------------------------------------- /src/configs/llama-2-13b-chat-hf.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | model: llama-2-13b-chat-hf 4 | url: https://huggingface.co/meta-llama/Llama-2-13b-chat-hf 5 | sampling: 6 | temperature: 0.7 7 | repetition_penalty: 1.0 8 | top_p: 0.95 9 | top_k: 40 # Applicable to llama.cpp only 10 | repeat_last_n: 64 # Applicable to llama.cpp only 11 | n_batch: 512 # Applicable to llama.cpp only 12 | generation: 13 | mean_gen_len: 128 # Applicable to mlc only 14 | max_gen_len: 4096 15 | max_window_size: 4096 16 | vocab_size: 32000 # Applicable to mlc only 17 | prompt: # Used in llama.cpp, taken from mlc 18 | conv_template: "llama-2" 19 | text: "\"<>\\n\\nYou are a helpful, respectful and honest assistant.\\n<>\\n\\n\"" 20 | in_prefix: "\"[INST]\"" 21 | in_suffix: "\"[/INST]\"" 22 | reverse: "\"[INST]\"" -------------------------------------------------------------------------------- /src/configs/llama-2-7b-chat-hf.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | model: llama-2-7b-chat-hf 4 | url: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf 5 | sampling: 6 | temperature: 0.7 7 | repetition_penalty: 1.0 8 | top_p: 0.95 9 | top_k: 40 # Applicable to llama.cpp only 10 | repeat_last_n: 64 # Applicable to llama.cpp only 11 | n_batch: 512 # Applicable to llama.cpp only 12 | generation: 13 | mean_gen_len: 128 # Applicable to mlc only 14 | max_gen_len: 4096 15 | max_window_size: 4096 16 | vocab_size: 32000 # Applicable to mlc only 17 | prompt: # Used in llama.cpp, taken from mlc 18 | conv_template: "llama-2" 19 | text: "\"<>\\n\\nYou are a helpful, respectful and honest assistant.\\n<>\\n\\n\"" 20 | in_prefix: "\"[INST]\"" 21 | in_suffix: "\"[/INST]\"" 22 | reverse: "\"[INST]\"" -------------------------------------------------------------------------------- /src/model_evaluation/src/evaluate_awq.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to evaluate the performance of the AWQ models with lm-eval suite. 4 | # Author: Stefanos Laskaridis 5 | 6 | MODEL_DIR=${MODEL_DIR:-"../../../melt_models_converted"} 7 | 8 | MODELS=( 9 | TheBloke_Llama-2-7B 10 | TheBloke_Llama-2-13B 11 | TheBloke_Mistral-7B-Instruct-v0.1 12 | TheBloke_TinyLlama-1.1B-Chat-v1.0 13 | TheBloke_stablelm-zephyr-3b 14 | ) 15 | 16 | for MODEL in ${MODELS[@]};do 17 | CUDA_VISIBLE_DEVICES=0 lm-eval \ 18 | --model vllm \ 19 | --model_args pretrained=${MODEL_DIR/${MODEL}-AWQ/,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=1 \ 20 | --tasks winogrande,arc_easy,arc_challenge,truthfulqa,hellaswag \ 21 | --batch_size auto |& tee ${MODEL}-AWQ.log 22 | done 23 | -------------------------------------------------------------------------------- /src/model_evaluation/src/evaluate_gptq.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to evaluate the performance of the GPTQ models with lm-eval suite. 4 | # Author: Stefanos Laskaridis 5 | 6 | MODEL_DIR=${MODEL_DIR:-"../../../melt_models_converted"} 7 | 8 | MODELS=( 9 | TheBloke_Llama-2-7B 10 | TheBloke_Llama-2-13B 11 | TheBloke_Mistral-7B-Instruct-v0.1 12 | TheBloke_TinyLlama-1.1B-Chat-v1.0 13 | TheBloke_stablelm-zephyr-3b 14 | ) 15 | 16 | for MODEL in ${MODELS[@]};do 17 | CUDA_VISIBLE_DEVICES=0 lm-eval \ 18 | --model hf \ 19 | --model_args pretrained=${MODEL_DIR}/${MODEL}-GPTQ/,autogptq=model.safetensors,gptq_use_triton=True \ 20 | --tasks winogrande,arc_easy,arc_challenge,truthfulqa,hellaswag \ 21 | --device cuda:0 \ 22 | --batch_size auto |& tee ${MODEL}-GPTQ.log 23 | done 24 | -------------------------------------------------------------------------------- /src/model_evaluation/src/evaluate_hf.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to evaluate the performance of the HF models with lm-eval suite. 4 | # Author: Stefanos Laskaridis 5 | 6 | MODEL_DIR=${MODEL_DIR:-"../../../melt_models/"} 7 | MODELS=( 8 | TinyLlama_TinyLlama-1.1B-Chat-v0.5 9 | stabilityai_stablelm-zephyr-3b 10 | mistralai_Mistral-7B-Instruct-v0.1 11 | meta-llama_Llama-2-7b-hf 12 | meta-llama_Llama-2-13b-hf 13 | google_gemma-2b 14 | google_gemma-7b 15 | ) 16 | TASKS=(winogrande arc_easy arc_challenge truthfulqa hellaswag) 17 | 18 | 19 | for MODEL in ${MODELS[@]}; do 20 | for TASK in ${TASKS[@]}; do 21 | echo "Evaluating $MODEL on task $TASK" 22 | (lm_eval --model hf --model_args pretrained="$MODEL_DIR/$MODEL" --tasks $TASK --batch_size auto |& tee eval_${MODEL}_${TASK}.log) & 23 | done 24 | done -------------------------------------------------------------------------------- /src/models/scripts/download_all_models.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to easily download all models from HF. 4 | # Author: Stefanos Laskaridis (stefanos@brave.com) 5 | 6 | # Models 7 | # 1. TinyLlama 8 | # 2. Zephyr 3b 9 | # 3. Mistral 7b 10 | # 4. Llama 7b 11 | # 5. Llama 13b 12 | # 6. Starcoder 13 | 14 | DOWNLOAD_SCRIPT_PATH=${DOWNLOAD_SCRIPT_PATH:-"./"} 15 | OUTPUT_PATH=${OUTPUT_PATH:-"../../melt_models"} 16 | HUGGINGFACE_TOKEN=$(cat ~/.hf_token) 17 | 18 | MODELS=( 19 | "TinyLlama/TinyLlama-1.1B-Chat-v0.5" 20 | "stabilityai/stablelm-zephyr-3b" 21 | "mistralai/Mistral-7B-Instruct-v0.1" 22 | "meta-llama/Llama-2-7b-chat-hf" 23 | "meta-llama/Llama-2-13b-chat-hf" 24 | "google/gemma-2b-it" 25 | "google/gemma-7b-it" 26 | ) 27 | 28 | pushd ../ 29 | for MODEL in ${MODELS[@]}; do 30 | python ${DOWNLOAD_SCRIPT_PATH}/download.py -m ${MODEL} -d ${OUTPUT_PATH} -t ${HUGGINGFACE_TOKEN} 31 | done 32 | popd -------------------------------------------------------------------------------- /src/configs/mistral-7b-instruct-v0.1.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | model: mistral-7b-instruct-v0.1 4 | url: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 5 | sampling: 6 | temperature: 0.7 7 | repetition_penalty: 1.0 8 | top_p: 0.95 9 | top_k: 40 # Applicable to llama.cpp only 10 | repeat_last_n: 64 # Applicable to llama.cpp only 11 | n_batch: 512 # Applicable to llama.cpp only 12 | generation: 13 | mean_gen_len: 128 # Applicable to mlc only 14 | max_gen_len: 16384 15 | max_window_size: 16384 16 | vocab_size: 32000 # Applicable to mlc only 17 | prompt: # Used in llama.cpp, taken from mlc 18 | conv_template: "mistral_default" 19 | text: "\"Always assist with care, respect, and truth. Respond with utmost utility yet securely. Avoid harmful, unethical, prejudiced, or negative content. Ensure replies promote fairness and positivity.\\n\"" 20 | in_prefix: "\"[INST]\"" 21 | in_suffix: "\"[/INST]\"" 22 | reverse: "\"[INST]\"" 23 | -------------------------------------------------------------------------------- /src/parsers/README.md: -------------------------------------------------------------------------------- 1 | # Per op output parser 2 | 3 | This is a parser for the JSONs output when profiling per op. It supports MLC and llama.cpp. 4 | In order to generate the parsable lines, the user needs to have build the respective custom backends with the appropriate flag (`BENCHMARK_PER_LAYER=1`). 5 | 6 | * For llama.cpp, we support parsing from the logs of the CLI logs (applicable on Android and Jetsons). 7 | * For MLC, we support parsing from the CLI and logcat logs. 8 | 9 | ## How to run 10 | 11 | ### For local logs 12 | 13 | ```bash 14 | python parser.py -i test_files/dummy_file.mlc.log [-o CSVs] [--merge ] -b mlc 15 | python parser.py -i test_files/dummy_file.llama_cpp.log [-o CSVs] [--merge ] -b llama.cpp 16 | ``` 17 | 18 | ### For Android residing logs 19 | 20 | ```bash 21 | cd scripts/ 22 | [ADB_RUNNER= DEVICE_ID= LOGFILE_ROOT=] ./logcat_getter.sh 23 | ``` 24 | -------------------------------------------------------------------------------- /src/model_evaluation/src/download_gptq_awq_models.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to download GPTQ and AWQ models from Hugging Face. 4 | # Author: Stefanos Laskaridis 5 | 6 | # Models 7 | # 1. TinyLlama 8 | # 2. Zephyr 3b 9 | # 3. Mistral 7b 10 | # 4. Llama 7b 11 | # 5. Llama 13b 12 | 13 | DOWNLOAD_SCRIPT_PATH=${DOWNLOAD_SCRIPT_PATH:-"../../models/"} 14 | OUTPUT_PATH=${OUTPUT_PATH:-"../../../melt_models_converted"} 15 | HUGGINGFACE_TOKEN=$(cat .hf_token) 16 | 17 | MODELS=( 18 | TheBloke/TinyLlama-1.1B-Chat-v1.0-GPTQ # GPTQ models 19 | TheBloke/stablelm-zephyr-3b-GPTQ 20 | TheBloke/Mistral-7B-Instruct-v0.1-GPTQ 21 | TheBloke/Llama-2-7B-GPTQ 22 | TheBloke/Llama-2-13B-GPTQ 23 | 24 | TheBloke/TinyLlama-1.1B-Chat-v1.0-AWQ # AWQ models 25 | TheBloke/stablelm-zephyr-3b-AWQ 26 | TheBloke/Mistral-7B-Instruct-v0.1-AWQ 27 | TheBloke/Llama-2-7B-AWQ 28 | TheBloke/Llama-2-13B-AWQ 29 | ) 30 | 31 | for MODEL in ${MODELS[@]};do 32 | python ${DOWNLOAD_SCRIPT_PATH}/download.py -m $MODEL -d ${OUTPUT_PATH} -t ${HUGGINGFACE_TOKEN} 33 | done 34 | -------------------------------------------------------------------------------- /src/parsers/scripts/logcat_getter.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to acquire MLC logs from Android device. 4 | # Author: Stefanos Laskaridis (stefanos@brave.com) 5 | 6 | 7 | ADB_RUNNER=${ADB_RUNNER:-adb} 8 | DEVICE_ID=${DEVICE_ID:-$($ADB_RUNNER devices | grep -v List | cut -f 1 | head -n 1)} 9 | LOGFILE_ROOT=${LOGFILE_ROOT:-"/data/local/tmp/"} 10 | OUTFILE_ROOT=${OUTFILE_ROOT:-"logs/"} 11 | NOW=$(date +"%Y%m%d%H%M%S") 12 | OUTFILE=${OUTFILE:-"logcat_${NOW}.txt"} 13 | 14 | # This script is used to get logcat from android device 15 | echo "Running logcat on device, persisting to $LOGFILE_ROOT/logcat_$NOW.txt" 16 | $ADB_RUNNER -s $DEVICE_ID logcat -f ${LOGFILE_ROOT}/logcat_$NOW.txt -d -v raw "TVM_RUNTIME:I" "*:S" 17 | 18 | echo "Getting log file from device" 19 | $ADB_RUNNER -s $DEVICE_ID pull $LOGFILE_ROOT/logcat_$NOW.txt $OUTFILE 20 | 21 | echo "Filtering lines" 22 | grep "Report" -A1 $OUTFILE > ${OUTFILE}_ 23 | rm ${OUTFILE} 24 | mv ${OUTFILE}_ ${OUTFILE} 25 | 26 | echo "Removing log file from device" 27 | $ADB_RUNNER -s $DEVICE_ID shell rm ${LOGFILE_ROOT}/logcat_$NOW.txt -------------------------------------------------------------------------------- /src/model_evaluation/README.md: -------------------------------------------------------------------------------- 1 | # MELT Model Evaluation 2 | 3 | This directory includes the accuracy vs. model size plots for various quantisation methods in the paper. 4 | 5 | Specifically, we support: 6 | 7 | * HF models 8 | * GPTQ-quantised models 9 | * AWQ-quantised models 10 | * llama.cpp-quantised models (k-quants) 11 | 12 | ## Structure 13 | 14 | ```bash 15 | ├── results # includes the plotting logic for what was used in the paper 16 | └── src # includes the code to run evaluations on models 17 | ``` 18 | 19 | ## How to run 20 | 21 | ```bash 22 | pushd src 23 | # Install python requirements 24 | conda create python=3.9 -n melt_evaluations 25 | pip install -r eval_requirements.txt 26 | pip install git+https://github.com/PanQiWei/AutoGPTQ@d2662b18bb91e1864b29e4e05862712382b8a076 27 | 28 | # Download models 29 | ./download_gptq_awq_models.sh 30 | 31 | # Run evaluations 32 | ./evaluate_awq.sh 33 | ./evaluate_gptq.sh 34 | ./evaluate_hf.sh # Needs access to models, as downloaded from (../../models) 35 | ./evaluate_llamacpp.sh # Needs separate compilation of llama.cpp and downloading of models (see ../../models) 36 | popd 37 | ``` -------------------------------------------------------------------------------- /src/parsers/utils/utils.py: -------------------------------------------------------------------------------- 1 | # Note: Utility functions for parsing and merging the logs from the MLC and llama.cpp backends. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import argparse 5 | 6 | from .llamacpp_utils import parse_logfile as llamacpp_parse_logfile 7 | from .mlc_utils import parse_logfile as mlc_parse_logfile 8 | 9 | 10 | def parse_args(): 11 | args = argparse.ArgumentParser() 12 | args.add_argument( 13 | "-b", "--backend", type=str, choices=["mlc", "llama.cpp"], required=True 14 | ) 15 | args.add_argument("-i", "--input", type=str) 16 | args.add_argument("-o", "--output", type=str) 17 | args.add_argument("-v", "--verbose", action="store_true", default=False) 18 | args.add_argument( 19 | "--merge", type=str, choices=["per_module", "per_op"], default=None 20 | ) 21 | 22 | return args.parse_args() 23 | 24 | 25 | def parse_file(log_file, backend, verbose): 26 | if backend == "mlc": 27 | return mlc_parse_logfile(log_file, verbose) 28 | 29 | if backend == "llama.cpp": 30 | return llamacpp_parse_logfile(log_file, verbose) 31 | 32 | raise ValueError(f"Invalid backend: {backend}") 33 | 34 | 35 | def merge_ops(df_merged, group_cols=["Name", "Device"], drop_cols=[]): 36 | df_grouped = df_merged.groupby(group_cols) 37 | df_merged = df_grouped.sum() 38 | 39 | df_merged.drop(columns=drop_cols, inplace=True) 40 | 41 | return df_merged 42 | -------------------------------------------------------------------------------- /src/model_evaluation/src/README.md: -------------------------------------------------------------------------------- 1 | # Model Evaluation 2 | 3 | To evaluate quantised models, we are leveraging the power of [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). 4 | We provide scripts for evaluating models in the following formats: 5 | * [x] [llama.cpp](https://github.com/ggerganov/llama.cpp) quantised models 6 | * [ ] [mlc](https://github.com/mlc-ai/mlc-llm) quantised models 7 | * [x] HF unquantised models 8 | * [x] [autogptq](https://github.com/PanQiWei/AutoGPTQ) quantised models 9 | * [x] [autoawq](https://github.com/casper-hansen/AutoAWQ) quantised models 10 | 11 | ## How to run 12 | 13 | 1. Download the models: 14 | 15 | For HF models, you can use the code in `src/models/download.py`. 16 | 17 | For llama.cpp models, you need to convert them using the script in `src/models/convert.py`. 18 | 19 | For GPTQ/AWQ models, we download from TheBloke's [repository](https://huggingface.co/TheBloke), using the following command: 20 | 21 | ```bash 22 | ./download_gptq_awq_models.sh 23 | ``` 24 | 25 | 2. Install requirements: 26 | 27 | ```bash 28 | git clone https://github.com/EleutherAI/lm-evaluation-harness 29 | pushd lm-evaluation-harness 30 | pip install -e .[gptq] 31 | popd 32 | pip install -r eval_requirements.txt # These requirements have been tested against python 3.9 33 | ``` 34 | 35 | 3. Run evaluations 36 | 37 | ```bash 38 | ./evaluate_gptq.sh 39 | ./evaluate_awq.sh 40 | ./evaluate_hf.sh 41 | ./evaluate_llamacpp.sh 42 | ``` -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Brave Software 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | ==== 24 | 25 | Individual submodules are either provided through chained checkout (e.g. llama.cpp, mlc, tvm dependencies) or as modified direct submodules (github.com/brave-experiments/mlc-llm-public, github.com/brave-experiments/tvm-public, github.com/brave-experiments/LLMFarmEval). Initial authors reserve the rights of their work. 26 | -------------------------------------------------------------------------------- /src/model_evaluation/results/model_specs.csv: -------------------------------------------------------------------------------- 1 | model,quantization,size,size_in_memory,parameters,max_ram 2 | Mistral-7b-v0.1,None,14.48,13.49,7240000,None 3 | Mistral-7b-v0.1,q3_k,3.52,3.28,7240000,6.02 4 | Mistral-7b-v0.1,q4_k,4.37,4.07,7240000,6.87 5 | Mistral-7b-v0.1,q4_0,4.11,3.83,7240000,6.61 6 | Mistral-7b-v0.1,AWQ,4.15,4.47,7240000,None 7 | Mistral-7b-v0.1,GPTQ,4.16,4.47,7240000,None 8 | Llama-2-7b,None,13.5,12.55,6740000,None 9 | Llama-2-7b,q3_k,3.30,3.07,6740000,5.80 10 | Llama-2-7b,q4_k,4.08,3.80,6740000,6.58 11 | Llama-2-7b,q4_0,3.83,3.56,6740000,6.33 12 | Llama-2-7b,AWQ,3.89,3.82,6740000,None 13 | Llama-2-7b,GPTQ,3.90,3.82,6740000,None 14 | Llama-2-13b,None,26.03,25.03,13000000,None 15 | Llama-2-13b,q4_k,7.87,7.33,13000000,10.37 16 | Llama-2-13b,AWQ,7.25,7.08,13000000,None 17 | Llama-2-13b,GPTQ,7.26,7.09,13000000,None 18 | TinyLlama-1.1B-v0.5,None,4.4,4.10,1100000,None 19 | TinyLlama-1.1B-v0.5,q3_k,0.55,0.52,1100000,3.05 20 | TinyLlama-1.1B-v0.5,q4_k,0.67,0.63,1100000,3.17 21 | TinyLlama-1.1B-v0.5,q4_0,0.64,0.60,1100000,3.14 22 | Zephyr-3b,None,5.59,5.37,2800000,None 23 | Zephyr-3b,q3_k,1.39,1.29,2800000,3.89 24 | Zephyr-3b,q4_k,1.71,1.59,2800000,4.21 25 | Zephyr-3b,q4_0,1.61,1.50,2800000,4.11 26 | Zephyr-3b,GPTQ,1.84,1.76,2800000,None 27 | gemma-7b,None,17.07,17.07,8540000,None 28 | gemma-7b,q3_k,4.06,4.06,8540000,None 29 | gemma-7b,q4_k,4.96,4.96,8540000,None 30 | gemma-7b,q4_0,4.66,4.66,8540000,None 31 | gemma-2b,None,4.95,4.95,2510000,None 32 | gemma-2b,q3_k,1.28,1.28,2510000,None 33 | gemma-2b,q4_k,1.51,1.51,2510000,None 34 | gemma-2b,q4_0,1.44,1.44,2510000,None 35 | -------------------------------------------------------------------------------- /src/model_evaluation/src/evaluate_llamacpp.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to evaluate the performance of the llama.cpp models with lm-eval suite and llama-cpp-python server. 4 | # Author: Stefanos Laskaridis 5 | 6 | MODEL_DIR=${MODEL_DIR:-"../../../melt_models_converted"} 7 | 8 | TASKS=("winogrande" "arc_easy" "arc_challenge" "truthfulqa" "hellaswag") 9 | MODELS=( 10 | TinyLlama_TinyLlama-1.1B-Chat-v0.5 11 | stabilityai_stablelm-zephyr-3b 12 | mistralai_Mistral-7B-Instruct-v0.1 13 | meta-llama_Llama-2-7b-hf 14 | meta-llama_Llama-2-13b-hf 15 | google_gemma-2b 16 | google_gemma-7b 17 | ) 18 | QUANTS=("q3_k" "q4_k" "q4_0") 19 | 20 | # Launch server 21 | counter=0 22 | for MODEL in ${MODELS[@]}; do 23 | for QUANT in ${QUANTS[@]}; do 24 | port=$((8084 + counter)) 25 | echo "Running server model ${MODEL} with quantization ${QUANT}" 26 | (python3 -m llama_cpp.server --model ${MODEL_DIR}/${MODEL}/${MODEL}-${QUANT}.gguf --n_gpu_layers 100 --host 0.0.0.0 --port $port |& tee server_${MODEL}-${QUANT}.log) & 27 | counter=$((counter + 1)) 28 | done 29 | done 30 | 31 | # Launch evaluation 32 | counter=0 33 | for MODEL in ${MODELS[@]}; do 34 | for QUANT in ${QUANTS[@]}; do 35 | port=$((8084 + counter)) 36 | for TASK in ${TASKS[@]};do 37 | (lm-eval --model gguf --model_args base_url="http://0.0.0.0:${port}" --tasks $TASK --output_path results/${MODEL}_${QUANT} |& tee eval_${MODEL}-${QUANT}_${TASK}.log) & 38 | done 39 | counter=$((counter + 1)) 40 | done 41 | done 42 | -------------------------------------------------------------------------------- /src/models/download.py: -------------------------------------------------------------------------------- 1 | # Note: Script to easily download models from the Hugging Face Hub. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import argparse 5 | import os 6 | 7 | from huggingface_hub import snapshot_download 8 | 9 | 10 | def parse_args(): 11 | args = argparse.ArgumentParser() 12 | args.add_argument( 13 | "-m", 14 | "--models", 15 | nargs="+", 16 | required=True, 17 | help="Model name to download (should be in hf format.)", 18 | ) 19 | args.add_argument( 20 | "-d", 21 | "--download-dir", 22 | type=str, 23 | required=True, 24 | help="Directory to download the model to.", 25 | ) 26 | args.add_argument( 27 | "-f", "--force", action="store_true", help="Overwrite existing files." 28 | ) 29 | args.add_argument( 30 | "-t", 31 | "--token", 32 | type=str, 33 | required=False, 34 | ) 35 | 36 | return args.parse_args() 37 | 38 | 39 | def main(args): 40 | for model in args.models: 41 | model_dir_name = model.replace("/", "_") 42 | print(f"-> Downloading model: {model}") 43 | target_path = os.path.join(args.download_dir, model_dir_name) 44 | model_path = snapshot_download( 45 | repo_id=model, 46 | local_dir=target_path, 47 | cache_dir=os.environ.get("TRANSFORMERS_CACHE", None), 48 | force_download=args.force, 49 | token=args.token, 50 | ) 51 | print( 52 | f"-> Model {model} downloaded to {args.download_dir}/{model_path}.") 53 | 54 | 55 | if __name__ == "__main__": 56 | args = parse_args() 57 | main(args) 58 | -------------------------------------------------------------------------------- /src/models/convert_utils/awq_utils.py: -------------------------------------------------------------------------------- 1 | # Note: Util functions for converting models to awq format. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import re 5 | 6 | from awq import AutoAWQForCausalLM 7 | from transformers import AutoTokenizer 8 | 9 | 10 | def decode_quant_method(quant_string): 11 | """ 12 | Decode quantization method from string. 13 | :param quant_string: The quantization method string (see regex for format). 14 | """ 15 | quant_config = {} 16 | regex = r"q(\d+)g(\d+)_(\S+)" 17 | match = re.match(regex, quant_string) 18 | if match: 19 | w_bit = int(match.group(1)) 20 | q_group_size = int(match.group(2)) 21 | q_version = match.group(3) 22 | else: 23 | w_bit, q_group_size, q_version = None, None, None 24 | 25 | quant_config = { 26 | "zero_point": True, 27 | "q_group_size": q_group_size, 28 | "w_bit": w_bit, 29 | "version": q_version.upper(), 30 | } 31 | 32 | return quant_config 33 | 34 | 35 | def quantize_awq(model_path, output_path, quant_config): 36 | """ 37 | Quantize an AWQ model. 38 | :param model_path: The path to the model. 39 | :param output_path: The path to save the quantized model. 40 | :param quant_config: The quantization configuration. 41 | """ 42 | # Load model 43 | model = AutoAWQForCausalLM.from_pretrained(model_path) 44 | tokenizer = AutoTokenizer.from_pretrained( 45 | model_path, trust_remote_code=True) 46 | 47 | # Quantize 48 | model.quantize(tokenizer, quant_config=quant_config) 49 | 50 | # Save quantized model 51 | model.save_quantized(output_path) 52 | tokenizer.save_pretrained(output_path) 53 | -------------------------------------------------------------------------------- /src/parsers/utils/mlc_utils.py: -------------------------------------------------------------------------------- 1 | # Note: Utility functions for parsing and merging the logs from the MLC backend. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import json 5 | import re 6 | from json.decoder import JSONDecodeError 7 | 8 | import pandas as pd 9 | 10 | 11 | def parse_json(json_string): 12 | metrics_dict = json.loads(json_string) 13 | 14 | return metrics_dict 15 | 16 | 17 | def parse_logfile(log_file, verbose=False): 18 | with open(log_file, "r") as f: 19 | lines = f.readlines() 20 | 21 | ops_metrics = [] 22 | regex = r"(.*)Report from function (.+)\[(\d+)/(\d+)\]:$" 23 | 24 | idx = 0 25 | while idx < len(lines): 26 | line = lines[idx] 27 | match = re.match(regex, line) 28 | if match: 29 | _, func_name, chunk, total_chunks = match.groups() 30 | chunk, total_chunks = int(chunk), int(total_chunks) 31 | print(chunk, total_chunks) 32 | if chunk == 0: 33 | buffer = lines[idx + 1].strip() 34 | elif chunk < total_chunks: 35 | buffer += lines[idx + 1].strip() 36 | elif chunk == total_chunks: 37 | buffer += lines[idx + 1].strip() 38 | try: 39 | ops_metrics.append((func_name, parse_json(buffer))) 40 | except JSONDecodeError: 41 | print(f"Invalid line: {buffer}") 42 | idx += 1 43 | idx += 1 44 | 45 | ops_count = {op: 0 for op in list(set([op for op, _ in ops_metrics]))} 46 | calls_dfs = {} 47 | device_metrics_dfs = {} 48 | config_dfs = {} 49 | for op, metrics_dict in ops_metrics: 50 | calls_df = pd.DataFrame(metrics_dict["calls"]) 51 | device_metrics_df = pd.DataFrame(metrics_dict["device_metrics"]) 52 | config_df = pd.DataFrame(parse_config(metrics_dict["configuration"])) 53 | 54 | calls_df = normalize_calls(calls_df) 55 | device_metrics_df = normalize_calls(device_metrics_df.transpose()) 56 | 57 | calls_dfs[f"{op}_{ops_count[op]}"] = calls_df 58 | device_metrics_dfs[f"{op}_{ops_count[op]}"] = device_metrics_df 59 | config_dfs[f"{op}_{ops_count[op]}"] = config_df 60 | ops_count[op] += 1 61 | 62 | if verbose: 63 | print("===" * 3, op, "===" * 3) 64 | print("device_metrics_df\n", device_metrics_df) 65 | print("config_df\n", config_df) 66 | print("calls_df\n", calls_df) 67 | 68 | return calls_dfs, device_metrics_dfs, config_dfs 69 | 70 | 71 | def normalize_calls(calls_df): 72 | assoc = { 73 | "Percent": "percent", 74 | "Count": "count", 75 | "Device": "string", 76 | "Duration (us)": "microseconds", 77 | "Name": "string", 78 | "Argument Shapes": "string", 79 | "Duration(us)": "microseconds", 80 | "ArgumentShapes": "string", 81 | } 82 | for col in calls_df.columns: 83 | # print(col, calls_df[col].dtype) 84 | calls_df[col] = calls_df[col].str.get(assoc[col]) 85 | 86 | return calls_df 87 | 88 | 89 | def parse_config(config_dict): 90 | ret_dict = {} 91 | ret_dict["num_threads"] = config_dict["Number of threads"] 92 | ret_dict["executor"] = config_dict["Executor"]["string"] 93 | 94 | return ret_dict 95 | -------------------------------------------------------------------------------- /src/models/README.md: -------------------------------------------------------------------------------- 1 | # Model automation 2 | 3 | With these scripts, we are able to automate the download and conversion of various models. 4 | 5 | * `download.py`: Is responsible for downloading models from Huggingface Hub. 6 | * `convert.py`: Is responsible for converting downloaded models into the format needed per backend framework, and quantizing it to the requested bidwidth. 7 | 8 | **Caveat**: In later versions of MLC-LLM, the conversion script is not the recommended way of converting models to MLC format (indicated in issues). 9 | If running the latest version, please use the `convert_mlc_new.sh` script instead. 10 | 11 | 12 | # How to run? 13 | 14 | Before you run the downloader, it might be necessary that you define your HF API token so that you are able to download the weights (e.g. Llama-2). 15 | Also, remember to install python requirements: 16 | 17 | ```bash 18 | pip install -r requirements.txt 19 | ``` 20 | 21 | ## Shortcut scripts 22 | 23 | ```bash 24 | scripts/ 25 | ├── convert_legacy.sh # Convert models based on convert.py 26 | ├── convert_new.sh # Convert models (based on MLC's new conversion scripts) 27 | ├── download_all_models.sh # Download models from HF 28 | └── replace_link_with_model.sh # Util script to resolve links and copy in place 29 | ``` 30 | 31 | ### Legacy vs. new 32 | 33 | In our experiments, we had to deal with an evolving codebase. For this reason, we have tagged two version in the MLC codebase, `before_gemma` and `after_gemma`. 34 | Conversion works as follows: 35 | 36 | | Version | Conversion script | 37 | | --- | --- | 38 | | before_gemma | convert_legacy.sh | 39 | | after_gemma | convert_new.sh | 40 | 41 | Before you run the conversion script, you need to define the `MLC_HOME` or `LLAMA_CPP_HOME` env vars depending on the backend specified. 42 | 43 | ## Conversion scripts 44 | 45 | ```bash 46 | python download.py --help 47 | usage: download.py [-h] -m MODELS [MODELS ...] -d DOWNLOAD_DIR [-f] [-t TOKEN] 48 | 49 | options: 50 | -h, --help show this help message and exit 51 | -m MODELS [MODELS ...], --models MODELS [MODELS ...] 52 | Model name to download (should be in hf format.) 53 | -d DOWNLOAD_DIR, --download-dir DOWNLOAD_DIR 54 | Directory to download the model to. 55 | -f, --force Overwrite existing files. 56 | -t TOKEN, --token TOKEN 57 | ``` 58 | 59 | ```bash 60 | python convert.py --help 61 | usage: convert.py [-h] -m MODEL -d OUTPUT_DIR -b {mlc,ggml,awq} -q QUANTIZATION_MODE [-t {android,iphone,metal,cuda}] [-c CONFIG] [--only-config] [--ignore-eos] [-v] 62 | 63 | options: 64 | -h, --help show this help message and exit 65 | -m MODEL, --model MODEL 66 | Model name to download (should be in hf format.) 67 | -d OUTPUT_DIR, --output-dir OUTPUT_DIR 68 | Directory to download the model to. 69 | -b {mlc,ggml,awq}, --backend {mlc,ggml,awq} 70 | Backend to convert to. 71 | -q QUANTIZATION_MODE, --quantization-mode QUANTIZATION_MODE 72 | Quantization mode to use. 73 | -t {android,iphone,metal,cuda}, --target {android,iphone,metal,cuda} 74 | Target to compile for. 75 | -c CONFIG, --config CONFIG 76 | Path to config file. 77 | --only-config Produce only the config file 78 | --ignore-eos Ignore EOS token (changes model config). 79 | -v, --verbose 80 | ``` -------------------------------------------------------------------------------- /src/models/scripts/convert_new.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to easily convert all models (MLC does not use convert.py). 4 | # Author: Stefanos Laskaridis (stefanos@brave.com) 5 | 6 | MODELS_DIR=${MODELS_DIR:-"../../melt_models"} 7 | CONFIGS_PATH=${CONFIGS_PATH:-"../configs"} 8 | OUTPUT_ROOT_DIR=${OUTPUT_ROOT_DIR:-"${MODELS_DIR}_converted"} 9 | LLAMA_CPP_HOME=${LLAMA_CPP_HOME:-"../../frameworks/llama.cpp/llama.cpp"} 10 | 11 | 12 | function convert_llama_cpp() { 13 | MODELS=( 14 | google_gemma-2b-it 15 | google_gemma-7b-it 16 | ) 17 | QUANTS=( 18 | q3_k 19 | q4_k 20 | q4_0 21 | ) 22 | 23 | for MODEL in ${MODELS[@]};do 24 | for QUANT in ${QUANTS[@]};do 25 | echo Converting model $MODEL-$QUANT to $CONVERTED_MODEL_DIR 26 | CONF_NAME=$(echo $MODEL | awk -F'_' '{print tolower($NF)}') 27 | LLAMA_CPP_HOME=$LLAMA_CPP_HOME python convert.py --model ${MODELS_DIR}/${MODEL} -b ggml -q $QUANT -t metal -v -d $OUTPUT_ROOT_DIR/${MODEL} -c ${CONFIGS_PATH}/${CONF_NAME}.yaml 28 | done 29 | done 30 | } 31 | 32 | function convert_mlc() { 33 | MODELS=( 34 | TinyLlama_TinyLlama-1.1B-Chat-v0.5 35 | google_gemma-2b-it 36 | google_gemma-7b-it 37 | meta-llama_Llama-2-7b-chat-hf 38 | meta-llama_Llama-2-13b-chat-hf 39 | ) 40 | 41 | CONV_TEMPLATES=( 42 | chatml 43 | gemma_instruction 44 | gemma_instruction 45 | llama-2 46 | llama-2 47 | ) 48 | 49 | QUANTS=( 50 | q3f16_1 51 | q4f16_1 52 | q0f32 53 | ) 54 | 55 | BACKENDS=( 56 | metal 57 | android 58 | iphone 59 | ) 60 | 61 | model_counter=0 62 | for MODEL in ${MODELS[@]};do 63 | CONV_TEMPLATE=${CONV_TEMPLATES[model_counter]} 64 | for QUANT in ${QUANTS[@]};do 65 | CONVERTED_MODEL_DIR="$OUTPUT_ROOT_DIR/$MODEL-$QUANT" 66 | echo Converting model $MODEL-$QUANT with conv_template $CONV_TEMPLATE to $CONVERTED_MODEL_DIR 67 | # 1. Generate model config 68 | echo "Running command: mlc_chat gen_config $MODELS_DIR/$MODEL --conv-template $CONV_TEMPLATE --quantization $QUANT -o $OUTPUT_ROOT_DIR/$MODEL-$QUANT" 69 | mlc_chat gen_config $MODELS_DIR/$MODEL --conv-template $CONV_TEMPLATE --quantization $QUANT -o $OUTPUT_ROOT_DIR/$MODEL-$QUANT 70 | for BACKEND in ${BACKENDS[@]};do 71 | if [ $BACKEND = "metal" ];then 72 | EXTENSION="so" 73 | else 74 | EXTENSION="tar" 75 | fi 76 | # 2. Compile model 77 | echo "Running command: mlc_chat compile $CONVERTED_MODEL_DIR/mlc-chat-config.json --device $BACKEND --quantization $QUANT -o $CONVERTED_MODEL_DIR/$MODEL-$QUANT-$BACKEND.$EXTENSION" 78 | mlc_chat compile $CONVERTED_MODEL_DIR/mlc-chat-config.json --device $BACKEND --quantization $QUANT -o $CONVERTED_MODEL_DIR/$MODEL-$QUANT-$BACKEND.$EXTENSION 79 | # 3. Convert model weights 80 | echo "Running command: mlc_chat convert_weight $MODELS_DIR/$MODEL/config.json --device ${BACKENDS[0]} --quantization $QUANT -o $CONVERTED_MODEL_DIR" 81 | mlc_chat convert_weight $MODELS_DIR/$MODEL/config.json --device ${BACKENDS[0]} --quantization $QUANT -o $CONVERTED_MODEL_DIR 82 | done 83 | done 84 | model_counter=$(( model_counter + 1 )) 85 | done 86 | } 87 | 88 | pushd ../ 89 | convert_llama_cpp 90 | convert_mlc 91 | popd -------------------------------------------------------------------------------- /src/prompts/notebooks/oasst_prompts.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import random\n", 10 | "from datasets import load_dataset\n", 11 | "import json" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": null, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "dataset = load_dataset(\"OpenAssistant/oasst1\")\n", 21 | "# get pandas dataframe\n", 22 | "oasst_df = dataset['train'].to_pandas()\n", 23 | "oasst_df.head()" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "oasst_df.lang.unique()" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": null, 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "# apply relevant filters\n", 42 | "oasst_df = oasst_df[oasst_df['synthetic'] == False]\n", 43 | "oasst_en_df = oasst_df[oasst_df['lang'] == 'en']" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "conversation_ids = oasst_en_df['message_tree_id'].unique().tolist()\n", 53 | "\n", 54 | "random.seed(42)\n", 55 | "n_samples = 2000\n", 56 | "conversation_ids_sample = random.sample(conversation_ids, n_samples)\n", 57 | "\n", 58 | "conversation_starters = []\n", 59 | "\n", 60 | "for id in conversation_ids_sample:\n", 61 | " conversation_df = oasst_en_df[oasst_en_df['message_tree_id'] == id]\n", 62 | " user_prompt_df = conversation_df[conversation_df.role == \"prompter\"]\n", 63 | " # filter out text longer than 1500 characters\n", 64 | " user_prompt_df = user_prompt_df[user_prompt_df['text'].str.len() < 1500]\n", 65 | " if user_prompt_df.shape[0] > 1:\n", 66 | " conversation_starters.append(user_prompt_df['text'].values[0])" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "N_CONVERSASTIONS = 50\n", 76 | "MIN_CONVERSATION_LENGTH = 5\n", 77 | "\n", 78 | "conversations = []\n", 79 | "\n", 80 | "for i in range(N_CONVERSASTIONS):\n", 81 | " delta = random.randint(1, 5)\n", 82 | " convo = random.sample(conversation_starters, MIN_CONVERSATION_LENGTH + delta)\n", 83 | "\n", 84 | " convo = [interaction for interaction in convo]\n", 85 | "\n", 86 | " conversations.append(convo)\n" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "with open(\"../conversations.json\", \"w\") as f:\n", 96 | " json.dump(conversations, f, indent=4)\n" 97 | ] 98 | } 99 | ], 100 | "metadata": { 101 | "kernelspec": { 102 | "display_name": "Python 3.9.18 ('melt')", 103 | "language": "python", 104 | "name": "python3" 105 | }, 106 | "language_info": { 107 | "codemirror_mode": { 108 | "name": "ipython", 109 | "version": 3 110 | }, 111 | "file_extension": ".py", 112 | "mimetype": "text/x-python", 113 | "name": "python", 114 | "nbconvert_exporter": "python", 115 | "pygments_lexer": "ipython3", 116 | "version": "3.11.4" 117 | }, 118 | "orig_nbformat": 4, 119 | "vscode": { 120 | "interpreter": { 121 | "hash": "f17a1f051a2eb28b727a709e153fbd670fdca23726e2b319358bfbbb848da26a" 122 | } 123 | } 124 | }, 125 | "nbformat": 4, 126 | "nbformat_minor": 2 127 | } 128 | -------------------------------------------------------------------------------- /src/models/scripts/convert_legacy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note: Script to easily convert all models (uses convert.py). 4 | # Author: Stefanos Laskaridis (stefanos@brave.com) 5 | 6 | # Models 7 | # 1. TinyLlama 8 | # 2. Zephyr 3b 9 | # 3. Mistral 7b 10 | # 4. Llama 7b 11 | # 5. Llama 13b 12 | 13 | LLAMA_CPP_HOME=${LLAMA_CPP_HOME:-"../../frameworks/llama.cpp/llama.cpp"} 14 | MLC_HOME=${MLC_HOME:-"../../frameworks/MLC/mlc-llm"} 15 | MODELS_PATH=${MODELS_PATH:-"../../melt_models"} 16 | CONFIGS_PATH=${CONFIGS_PATH:-"../configs/"} 17 | OUTPUT_PATH=${OUTPUT_PATH:-"../../melt_models_converted"} 18 | 19 | 20 | function convert_llama_cpp { 21 | for QUANT in q4_0 q3_k q4_k f32; do 22 | LLAMA_CPP_HOME=$LLAMA_CPP_HOME python convert.py --model ${MODELS_PATH}/TinyLlama_TinyLlama-1.1B-Chat-v0.5 -b ggml -q $QUANT -t metal -v -d $OUTPUT_PATH/TinyLlama_TinyLlama-1.1B-Chat-v0.5 -c ${CONFIGS_PATH}/tinyllama-1.1b-chat-v0.5.yaml 23 | LLAMA_CPP_HOME=$LLAMA_CPP_HOME python convert.py --model ${MODELS_PATH}/stabilityai_stablelm-zephyr-3b -b ggml -q $QUANT -t metal -v -d $OUTPUT_PATH/stabilityai_stablelm-zephyr-3b -c ${CONFIGS_PATH}/stablelm-zephyr-3b.yaml 24 | LLAMA_CPP_HOME=$LLAMA_CPP_HOME python convert.py --model ${MODELS_PATH}/mistralai_Mistral-7B-Instruct-v0.1 -b ggml -q $QUANT -t metal -v -d $OUTPUT_PATH/mistralai_Mistral-7B-Instruct-v0.1 -c ${CONFIGS_PATH}/mistral-7b-instruct-v0.1.yaml 25 | LLAMA_CPP_HOME=$LLAMA_CPP_HOME python convert.py --model ${MODELS_PATH}/meta-llama_Llama-2-7b-chat-hf -b ggml -q $QUANT -t metal -v -d $OUTPUT_PATH/meta-llama_Llama-2-7b-chat-hf -c ${CONFIGS_PATH}/llama-2-7b-chat-hf.yaml 26 | LLAMA_CPP_HOME=$LLAMA_CPP_HOME python convert.py --model ${MODELS_PATH}/Llama-2-13b-chat-hf -b ggml -q $QUANT -t metal -v -d $OUTPUT_PATH/meta-llama_Llama-2-13b-chat-hf -c ${CONFIGS_PATH}/llama-2-13b-chat-hf.yaml 27 | done 28 | } 29 | 30 | function convert_mlc { 31 | echo "WARNING: This will only work on the before_gemma version of MLC. Otherwise, the model conversion will fail." 32 | echo "You can use the script ./convert_new.sh instead." 33 | for QUANT in q3f16_1 q4f16_1 q0f32; do 34 | for BACKEND in metal iphone android cuda; do 35 | if [ $BACKEND = 'cuda' ]; then 36 | EXTRA_COMPILE_ARGS="--use-cuda-graph --use-flash-attn-mqa" 37 | else 38 | EXTRA_COMPILE_ARGS="" 39 | fi 40 | 41 | MLC_HOME=$MLC_HOME python convert.py --model ${MODELS_PATH}/TinyLlama_TinyLlama-1.1B-Chat-v0.5 -b mlc -q $QUANT -t $BACKEND -v -d $OUTPUT_PATH/TinyLlama_TinyLlama-1.1B-Chat-v0.5 -c ${CONFIGS_PATH}/tinyllama-1.1b-chat-v0.5.yaml $EXTRA_COMPILE_ARGS 42 | if [ $QUANT = "q0f32" ];then 43 | MLC_HOME=$MLC_HOME python convert.py --model ${MODELS_PATH}/stabilityai_stablelm-zephyr-3b -b mlc -q q0f16 -t $BACKEND -v -d $OUTPUT_PATH/stabilityai_stablelm-zephyr-3b -c ${CONFIGS_PATH}/stablelm-zephyr-3b.yaml $EXTRA_COMPILE_ARGS 44 | else 45 | MLC_HOME=$MLC_HOME python convert.py --model ${MODELS_PATH}/stabilityai_stablelm-zephyr-3b -b mlc -q $QUANT -t $BACKEND -v -d $OUTPUT_PATH/stabilityai_stablelm-zephyr-3b -c ${CONFIGS_PATH}/stablelm-zephyr-3b.yaml $EXTRA_COMPILE_ARGS 46 | fi 47 | MLC_HOME=$MLC_HOME python convert.py --model ${MODELS_PATH}/mistralai_Mistral-7B-Instruct-v0.1 -b mlc -q $QUANT -t $BACKEND -v -d $OUTPUT_PATH/mistralai_Mistral-7B-Instruct-v0.1 -c ${CONFIGS_PATH}/mistral-7b-instruct-v0.1.yaml $EXTRA_COMPILE_ARGS 48 | MLC_HOME=$MLC_HOME python convert.py --model ${MODELS_PATH}/Llama-2-7b-chat-hf -b mlc -q $QUANT -t $BACKEND -v -d $OUTPUT_PATH/Llama-2-7b-chat-hf -c ${CONFIGS_PATH}/llama-2-7b-chat-hf.yaml $EXTRA_COMPILE_ARGS 49 | MLC_HOME=$MLC_HOME python convert.py --model ${MODELS_PATH}/Llama-2-13b-chat-hf -b mlc -q $QUANT -t $BACKEND -v -d $OUTPUT_PATH/Llama-2-13b-chat-hf -c ${CONFIGS_PATH}/llama-2-13b-chat-hf.yaml $EXTRA_COMPILE_ARGS 50 | done 51 | done 52 | } 53 | 54 | pushd ../ 55 | convert_llama_cpp 56 | convert_mlc 57 | popd -------------------------------------------------------------------------------- /src/parsers/utils/llamacpp_utils.py: -------------------------------------------------------------------------------- 1 | # Note: Utility functions for parsing and merging the logs from the llama.cpp backend. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import re 5 | 6 | import pandas as pd 7 | 8 | 9 | def parse_logfile(log_file, verbose=False): 10 | with open(log_file, "r") as f: 11 | lines = f.readlines() 12 | 13 | weights_pattern = r"llama_model_loader:\s+-\s+tensor\s+(\d+):\s+(\S+)\s+(\S+)\s+\[\s+(\d+),\s+(\d+),\s+(\d+),\s+(\d+)\s+\]" 14 | weights_regex = re.compile(weights_pattern) 15 | weights_delimiter = "." * 98 16 | 17 | totals_pattern = r"perf_total_per_op_us\[\s*(\w+)\]\s*=\s*([\d.]+)\s*ms" 18 | totals_regex = re.compile(totals_pattern) 19 | totals_delimiter = "=" * 40 20 | 21 | per_op_delimiter_pattern = r"n_nodes = (\d+)" 22 | per_op_delimiter_regex = re.compile(per_op_delimiter_pattern) 23 | per_op_pattern = r"\s*-\s*(\d+): \[\s*(\d+),\s*(\d+),\s*(\d+)\]\s+([\S+]+)\s+\(\s*(\d+)\) cpu\s*=\s*(\d+\.?\d+)\s*/\s*(\d+\.?\d+) ms, \s*wall\s*=\s*(\d+\.?\d+)\s*/\s*(\d+\.?\d+) ms\s*" 24 | per_op_regex = re.compile(per_op_pattern) 25 | 26 | weights_details = [] 27 | current_token_op_summary_latencies = [] 28 | current_token_op_detailed_latencies = [] 29 | idx = 0 30 | weights_parsed_flag = False 31 | while idx < len(lines): 32 | line = lines[idx] 33 | if not weights_parsed_flag: 34 | if match := re.match(weights_regex, line): 35 | tensor_num = match.groups()[0] 36 | tensor_name = match.groups()[1] 37 | tensor_qtype = match.groups()[2] 38 | dimensions = [int(match.groups()[i]) for i in range(3, 7)] 39 | weights_details.append( 40 | { 41 | "Name": tensor_name, 42 | "Quantization": tensor_qtype, 43 | "Dimensions": dimensions, 44 | } 45 | ) 46 | 47 | if match := re.match(totals_regex, line): 48 | while line.strip() != totals_delimiter and idx < len(lines): 49 | match = re.match(totals_regex, line) 50 | if match is None: 51 | raise ValueError(f"Invalid line: {line}") 52 | op, duration_ms = match.groups() 53 | current_token_op_summary_latencies.append( 54 | {"Name": op, "Duration (ms)": float(duration_ms)} 55 | ) 56 | idx += 1 57 | line = lines[idx] 58 | elif match := re.match(per_op_delimiter_regex, line): 59 | num_nodes = int(match.groups()[0]) 60 | for jdx in range(1, num_nodes + 1): 61 | line = lines[idx + jdx] 62 | match = re.match(per_op_regex, line) 63 | if match is None: 64 | raise ValueError(f"Invalid line: {line}") 65 | 66 | node_num = int(match.group(1)) 67 | dimensions = [int(match.group(i)) for i in range(2, 5)] 68 | wall_time = float(match.group(9)) 69 | op = match.group(5) 70 | cpu_time_ms = float(match.group(7)) 71 | 72 | current_token_op_detailed_latencies.append( 73 | { 74 | "Name": op, 75 | "Duration (ms)": cpu_time_ms, 76 | "Wall time (ms)": wall_time, 77 | "Argument Shapes": dimensions, 78 | "node_num": node_num, 79 | } 80 | ) 81 | idx += jdx 82 | else: 83 | idx += 1 84 | 85 | weights_df = pd.DataFrame(weights_details) 86 | summary_df = pd.DataFrame(current_token_op_summary_latencies) 87 | detailed_df = pd.DataFrame(current_token_op_detailed_latencies) 88 | 89 | if verbose: 90 | print("weights_details\n", weights_df) 91 | print("summary_df\n", summary_df) 92 | print("detailed_df\n", detailed_df) 93 | 94 | return weights_df, summary_df, detailed_df 95 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | melt_models*/ 2 | ### Python ### 3 | # Byte-compiled / optimized / DLL files 4 | __pycache__/ 5 | *.py[cod] 6 | *$py.class 7 | 8 | # C extensions 9 | *.so 10 | 11 | # Distribution / packaging 12 | .Python 13 | build/ 14 | develop-eggs/ 15 | dist/ 16 | downloads/ 17 | eggs/ 18 | .eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | wheels/ 25 | share/python-wheels/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | MANIFEST 30 | .DS_Store 31 | 32 | # PyInstaller 33 | # Usually these files are written by a python script from a template 34 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 35 | *.manifest 36 | *.spec 37 | 38 | # Installer logs 39 | pip-log.txt 40 | pip-delete-this-directory.txt 41 | 42 | # Unit test / coverage reports 43 | htmlcov/ 44 | .tox/ 45 | .nox/ 46 | .coverage 47 | .coverage.* 48 | .cache 49 | nosetests.xml 50 | coverage.xml 51 | *.cover 52 | *.py,cover 53 | .hypothesis/ 54 | .pytest_cache/ 55 | cover/ 56 | 57 | # Translations 58 | *.mo 59 | *.pot 60 | 61 | # Django stuff: 62 | *.log 63 | local_settings.py 64 | db.sqlite3 65 | db.sqlite3-journal 66 | 67 | # Flask stuff: 68 | instance/ 69 | .webassets-cache 70 | 71 | # Scrapy stuff: 72 | .scrapy 73 | 74 | # Sphinx documentation 75 | docs/_build/ 76 | 77 | # PyBuilder 78 | .pybuilder/ 79 | target/ 80 | 81 | # Jupyter Notebook 82 | .ipynb_checkpoints 83 | 84 | # IPython 85 | profile_default/ 86 | ipython_config.py 87 | 88 | # pyenv 89 | # For a library or package, you might want to ignore these files since the code is 90 | # intended to run in multiple environments; otherwise, check them in: 91 | # .python-version 92 | 93 | # pipenv 94 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 95 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 96 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 97 | # install all needed dependencies. 98 | #Pipfile.lock 99 | 100 | # poetry 101 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 102 | # This is especially recommended for binary packages to ensure reproducibility, and is more 103 | # commonly ignored for libraries. 104 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 105 | #poetry.lock 106 | 107 | # pdm 108 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 109 | #pdm.lock 110 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 111 | # in version control. 112 | # https://pdm.fming.dev/#use-with-ide 113 | .pdm.toml 114 | 115 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 116 | __pypackages__/ 117 | 118 | # Celery stuff 119 | celerybeat-schedule 120 | celerybeat.pid 121 | 122 | # SageMath parsed files 123 | *.sage.py 124 | 125 | # Environments 126 | .env 127 | .venv 128 | env/ 129 | venv/ 130 | ENV/ 131 | env.bak/ 132 | venv.bak/ 133 | 134 | # Spyder project settings 135 | .spyderproject 136 | .spyproject 137 | 138 | # Rope project settings 139 | .ropeproject 140 | 141 | # mkdocs documentation 142 | /site 143 | 144 | # mypy 145 | .mypy_cache/ 146 | .dmypy.json 147 | dmypy.json 148 | 149 | # Pyre type checker 150 | .pyre/ 151 | 152 | # pytype static type analyzer 153 | .pytype/ 154 | 155 | # Cython debug symbols 156 | cython_debug/ 157 | 158 | # PyCharm 159 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 160 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 161 | # and can be added to the global gitignore or merged into this file. For a more nuclear 162 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 163 | #.idea/ 164 | 165 | ### Python Patch ### 166 | # Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration 167 | poetry.toml 168 | 169 | # ruff 170 | .ruff_cache/ 171 | 172 | # LSP config files 173 | pyrightconfig.json 174 | -------------------------------------------------------------------------------- /src/models/convert_utils/mlc_utils.py: -------------------------------------------------------------------------------- 1 | # Note: Util functions for converting models to MLC format. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import glob 5 | import json 6 | import os 7 | import re 8 | import subprocess 9 | 10 | import yaml 11 | 12 | MLC_HOME = os.environ.get("MLC_HOME") 13 | 14 | 15 | def mlc_get_max_length(config_path): 16 | """ 17 | Acquire the maximum generation length from the config file. 18 | """ 19 | with open(config_path, "r") as f: 20 | config = yaml.safe_load(f) 21 | 22 | return config["generation"].get("max_gen_len", None) 23 | 24 | 25 | def mlc_change_model_template_eos(chat_config_path): 26 | """ 27 | Change the EOS token in the model config file. This needs a custom build backend. 28 | :param chat_config_path: The path to the chat config file. 29 | """ 30 | with open(chat_config_path, "r") as f: 31 | config = json.load(f) 32 | 33 | previous_template = config.get("conv_template", None) 34 | if previous_template: 35 | config["conv_template"] = f"{previous_template}-unconstrained" 36 | 37 | with open(chat_config_path, "w") as f: 38 | json.dump(config, f) 39 | 40 | 41 | def mlc_translate_config_to_model_config(config_path, chat_config_path): 42 | """ 43 | Translates the model config file (from MELT/configs/) to the MLC config format. 44 | """ 45 | 46 | with open(chat_config_path, "r") as f: 47 | model_config = json.load(f) 48 | 49 | with open(config_path, "r") as f: 50 | config = yaml.safe_load(f) 51 | 52 | model_config["temperature"] = config["sampling"].get( 53 | "temperature", model_config["temperature"] 54 | ) 55 | model_config["repetition_penalty"] = config["sampling"].get( 56 | "repetition_penalty", model_config["repetition_penalty"] 57 | ) 58 | model_config["top_p"] = config["sampling"].get( 59 | "top_p", model_config["top_p"]) 60 | 61 | model_config["mean_gen_len"] = config["generation"].get( 62 | "mean_gen_len", model_config["mean_gen_len"] 63 | ) 64 | model_config["max_gen_len"] = config["generation"].get( 65 | "max_gen_len", model_config["max_gen_len"] 66 | ) 67 | if "max_window_size" in model_config: 68 | model_config["max_window_size"] = config["generation"].get( 69 | "max_window_size", model_config["max_window_size"] 70 | ) 71 | model_config["vocab_size"] = config["generation"].get( 72 | "vocab_size", model_config["vocab_size"] 73 | ) 74 | 75 | model_config["conv_template"] = config["prompt"].get( 76 | "conv_template", model_config["conv_template"] 77 | ) 78 | 79 | with open(chat_config_path, "w") as f: 80 | json.dump(model_config, f) 81 | 82 | return config 83 | 84 | 85 | def convert_mlc(model_dir, args): 86 | """ 87 | Convert a model to the MLC format. 88 | :param model_dir: The path to the model directory. 89 | :param args: The arguments passed to the script. 90 | """ 91 | model_name = os.path.basename(model_dir) 92 | exec_path = os.path.join(MLC_HOME, "build.py") 93 | args_list = [ 94 | "python", 95 | exec_path, 96 | "--model", 97 | model_dir, 98 | "--artifact-path", 99 | args.output_dir, 100 | "--quantization", 101 | args.quantization_mode, 102 | "--target", 103 | args.target, 104 | "--max-seq-len", 105 | f"{args.max_seq_length}", 106 | args.extra_args, 107 | ] 108 | 109 | if ("tinyllama" in model_name.lower()) or ("stablelm-zephyr" in model_name.lower()): 110 | args_list.append("--use-safetensors") 111 | 112 | print(f"Running cmd: {' '.join(args_list)}") 113 | proc = subprocess.Popen( 114 | args_list, 115 | stdout=subprocess.PIPE, 116 | stderr=subprocess.PIPE, 117 | ) 118 | stdout, stderr = proc.communicate() 119 | regex = "Finish exporting chat config to (.*)" 120 | match = re.search(regex, stdout.decode("utf-8")) 121 | chat_config = os.path.join( 122 | args.output_dir, "**", "params", "mlc-chat-config.json") 123 | chat_config_path = glob.glob(chat_config)[0] 124 | if match: 125 | chat_config_path = match.group(1) 126 | if args.verbose: 127 | print(stdout.decode("utf-8")) 128 | print(stderr.decode("utf-8")) 129 | 130 | return chat_config_path 131 | -------------------------------------------------------------------------------- /src/model_evaluation/src/eval_requirements.txt: -------------------------------------------------------------------------------- 1 | absl-py==2.0.0 2 | accelerate==0.25.0 3 | aiohttp==3.9.1 4 | aioprometheus==23.12.0 5 | aiosignal==1.3.1 6 | altair==5.2.0 7 | anyio==4.2.0 8 | astor==0.8.1 9 | async-timeout==4.0.3 10 | attributedict==0.3.0 11 | attrs==23.2.0 12 | # auto-gptq @ git+https://github.com/PanQiWei/AutoGPTQ@d2662b18bb91e1864b29e4e05862712382b8a076 13 | autoawq==0.1.8 14 | base58==2.1.1 15 | black 16 | blessings==1.7 17 | blinker==1.7.0 18 | Brotli==1.1.0 19 | cachetools==5.3.2 20 | certifi==2023.11.17 21 | chardet==5.2.0 22 | charset-normalizer==3.3.2 23 | click==7.1.2 24 | cmake==3.28.1 25 | codecov==2.1.13 26 | colorama==0.4.6 27 | coloredlogs==15.0.1 28 | colour-runner==0.1.1 29 | coverage==7.4.0 30 | DataProperty==1.0.1 31 | datasets==2.16.1 32 | deepdiff==6.7.1 33 | dill==0.3.7 34 | distlib==0.3.8 35 | evaluate==0.4.1 36 | exceptiongroup==1.2.0 37 | fastapi==0.108.0 38 | filelock==3.13.1 39 | flake8==7.0.0 40 | frozenlist==1.4.1 41 | fsspec==2023.10.0 42 | gekko==1.0.6 43 | gitdb==4.0.11 44 | GitPython==3.1.41 45 | h11==0.14.0 46 | httptools==0.6.1 47 | huggingface-hub==0.20.2 48 | humanfriendly==10.0 49 | idna==3.6 50 | immutabledict==4.1.0 51 | inflate64==1.0.0 52 | iniconfig==2.0.0 53 | inspecta==0.1.3 54 | isort==5.8.0 55 | Jinja2==3.1.2 56 | joblib==1.3.2 57 | jsonlines==4.0.0 58 | jsonschema==4.20.0 59 | jsonschema-specifications==2023.12.1 60 | llama_cpp_python==0.2.79 61 | langdetect==1.0.9 62 | lit==17.0.6 63 | -e git+https://github.com/EleutherAI/lm-evaluation-harness@7264a2e03871733760c640acc3aab0e181be843d#egg=lm_eval 64 | lxml==5.1.0 65 | MarkupSafe==2.1.3 66 | mbstrdecoder==1.1.3 67 | mccabe==0.7.0 68 | mpmath==1.3.0 69 | msgpack==1.0.7 70 | multidict==6.0.4 71 | multiprocess==0.70.15 72 | multivolumefile==0.2.3 73 | mypy-extensions==1.0.0 74 | networkx==3.2.1 75 | ninja==1.11.1.1 76 | nltk==3.8.1 77 | numexpr==2.8.8 78 | numpy==1.26.3 79 | nvidia-cublas-cu11==11.10.3.66 80 | nvidia-cublas-cu12==12.1.3.1 81 | nvidia-cuda-cupti-cu11==11.7.101 82 | nvidia-cuda-cupti-cu12==12.1.105 83 | nvidia-cuda-nvrtc-cu11==11.7.99 84 | nvidia-cuda-nvrtc-cu12==12.1.105 85 | nvidia-cuda-runtime-cu11==11.7.99 86 | nvidia-cuda-runtime-cu12==12.1.105 87 | nvidia-cudnn-cu11==8.5.0.96 88 | nvidia-cudnn-cu12==8.9.2.26 89 | nvidia-cufft-cu11==10.9.0.58 90 | nvidia-cufft-cu12==11.0.2.54 91 | nvidia-curand-cu11==10.2.10.91 92 | nvidia-curand-cu12==10.3.2.106 93 | nvidia-cusolver-cu11==11.4.0.1 94 | nvidia-cusolver-cu12==11.4.5.107 95 | nvidia-cusparse-cu11==11.7.4.91 96 | nvidia-cusparse-cu12==12.1.0.106 97 | nvidia-nccl-cu12==2.18.1 98 | nvidia-nvjitlink-cu12==12.3.101 99 | nvidia-nvtx-cu11==11.7.91 100 | nvidia-nvtx-cu12==12.1.105 101 | optimum==1.16.1 102 | ordered-set==4.1.0 103 | orjson==3.9.10 104 | packaging==23.2 105 | pandas==2.1.4 106 | pathspec==0.12.1 107 | pathvalidate==3.2.0 108 | peft==0.7.1 109 | pillow==10.2.0 110 | platformdirs==4.1.0 111 | plotly==5.18.0 112 | pluggy==1.3.0 113 | portalocker==2.8.2 114 | promptsource==0.2.3 115 | protobuf==4.25.1 116 | psutil==5.9.7 117 | py7zr==0.20.8 118 | pyarrow==14.0.2 119 | pyarrow-hotfix==0.6 120 | pybcj==1.0.2 121 | pybind11==2.11.1 122 | pycodestyle==2.11.1 123 | pycryptodomex==3.20.0 124 | pydantic==2.7.4 125 | pydantic-settings==2.3.3 126 | pydeck==0.8.1b0 127 | pyflakes==3.2.0 128 | Pygments==2.17.2 129 | pyppmd==1.1.0 130 | pyproject-api==1.6.1 131 | pytablewriter==1.2.0 132 | pytest==7.4.4 133 | python-dateutil==2.8.2 134 | python-dotenv==1.0.0 135 | pytz==2023.3.post1 136 | PyYAML==6.0.1 137 | pyzstd==0.15.9 138 | quantile-python==1.1 139 | ray==2.9.0 140 | referencing==0.32.1 141 | regex==2023.12.25 142 | requests==2.31.0 143 | responses==0.18.0 144 | rootpath==0.1.1 145 | rouge==1.0.1 146 | rouge-score==0.1.2 147 | rpds-py==0.16.2 148 | sacrebleu==2.4.0 149 | safetensors==0.4.1 150 | scikit-learn==1.3.2 151 | scipy==1.11.4 152 | sentencepiece==0.1.99 153 | six==1.16.0 154 | smmap==5.0.1 155 | sniffio==1.3.0 156 | sqlitedict==2.1.0 157 | sse-starlette==2.1.2 158 | starlette==0.32.0.post1 159 | starlette-context==0.3.6 160 | streamlit==0.82.0 161 | sympy==1.12 162 | tabledata==1.3.3 163 | tabulate==0.9.0 164 | tcolorpy==0.1.4 165 | tenacity==8.2.3 166 | termcolor==2.4.0 167 | texttable==1.7.0 168 | threadpoolctl==3.2.0 169 | tokenizers==0.15.0 170 | toml==0.10.2 171 | tomli==2.0.1 172 | toolz==0.12.0 173 | torch==2.1.2 174 | torchvision==0.16.2 175 | tornado==6.4 176 | tox==4.11.4 177 | tqdm==4.66.1 178 | tqdm-multiprocess==0.0.11 179 | transformers==4.38 180 | triton==2.1.0 181 | typepy==1.3.2 182 | typing_extensions==4.9.0 183 | tzdata==2023.4 184 | tzlocal==5.2 185 | urllib3==2.1.0 186 | uvicorn==0.25.0 187 | uvloop==0.19.0 188 | validators==0.22.0 189 | virtualenv==20.25.0 190 | vllm==0.2.7 191 | watchdog==3.0.0 192 | watchfiles==0.21.0 193 | websockets==12.0 194 | xformers==0.0.23.post1 195 | xxhash==3.4.1 196 | yarl==1.9.4 197 | zstandard==0.22.0 198 | -------------------------------------------------------------------------------- /src/model_evaluation/results/visualization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "\n", 11 | "import pandas as pd\n", 12 | "import numpy as np\n", 13 | "import seaborn as sns\n", 14 | "import matplotlib.pyplot as plt\n", 15 | "from matplotlib.colors import to_rgba\n", 16 | "\n", 17 | "plt.rcParams.update({'font.size': 22})" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "result_df = pd.read_csv(\"results.csv\")\n", 27 | "result_df.head()" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "specs_df = pd.read_csv(\"model_specs.csv\")\n", 37 | "specs_df.loc[specs_df[\"quantization\"].isna(),\"quantization\"] = \"None\"\n", 38 | "specs_df.head()" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "result_df.replace(np.nan, \"None\", inplace=True)\n", 48 | "specs_df.replace(np.nan, \"None\", inplace=True)\n", 49 | "display(result_df.head(5))\n", 50 | "display(specs_df.head(5))" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "from scipy.spatial import ConvexHull\n", 60 | "import numpy as np\n", 61 | "\n", 62 | "datasets = [\"winogrande\", \"arc\", \"truthfulqa\", \"hellaswag\"]\n", 63 | "filtered_df = result_df[result_df[\"dataset\"].isin(datasets)]\n", 64 | "filtered_df = filtered_df[filtered_df[\"framework\"] != \"paper\"]\n", 65 | "\n", 66 | "total_datasets = len(datasets)\n", 67 | "# Create a separate plot for each dataset\n", 68 | "for i, dataset in enumerate(datasets):\n", 69 | " print(dataset)\n", 70 | " dataset_df = filtered_df[filtered_df[\"dataset\"] == dataset]\n", 71 | " dataset_df = dataset_df.merge(specs_df, on=(\"model\",\"quantization\"))\n", 72 | " dataset_df = dataset_df.sort_values(by=[\"model\", \"quantization\"])\n", 73 | "\n", 74 | " # Create a single plot for the model in the dataset\n", 75 | " plt.figure(figsize=(10,8))\n", 76 | " ax = sns.scatterplot(x=\"size_in_memory\", y=\"accuracy\", hue=\"model\", style=\"quantization\", data=dataset_df,\n", 77 | " s=150, alpha=0.8)\n", 78 | "\n", 79 | " # Draw convex hulls for each model\n", 80 | " models = dataset_df[\"model\"].unique()\n", 81 | " sorted_models = sorted(models)\n", 82 | " for model in sorted_models:\n", 83 | " # if dataset == \"hellaswag\" and (model == \"gemma-7b\" or model == \"gemma-2b\"):\n", 84 | " # continue\n", 85 | " model_df = dataset_df[dataset_df[\"model\"] == model]\n", 86 | " points = model_df[[\"size\", \"accuracy\"]].values\n", 87 | " hull = ConvexHull(points)\n", 88 | " hull_points = points[hull.vertices]\n", 89 | " hull_points = np.append(hull_points, hull_points[0:1], axis=0)\n", 90 | "\n", 91 | " color = sns.color_palette(\"tab10\")[sorted_models.index(model)]\n", 92 | " ax.fill(hull_points[:, 0], hull_points[:, 1], color=to_rgba(color, 0.3))\n", 93 | "\n", 94 | " # ax.set_title(f\"{dataset}: Performance vs Model Size\")\n", 95 | " ax.set_xlabel(\"Model Size (GB)\")\n", 96 | " ax.set_ylabel(\"Accuracy (%)\")\n", 97 | " ax.set_xscale('log')\n", 98 | " xticks = [1, 2, 4, 8, 16, 32]\n", 99 | " xticklabels = [f\"{size}\" if size != 0.5 else \"½\" for size in xticks]\n", 100 | " ax.set_xticks(xticks)\n", 101 | " ax.set_xticklabels(xticklabels, rotation=0)\n", 102 | " # ax.set_xlim(-0, xticks[-1])\n", 103 | " if dataset == 'arc':\n", 104 | " ax.legend(prop={'size': 14}, ncol=2, loc=\"lower right\",\n", 105 | " columnspacing=-0.3)\n", 106 | " else:\n", 107 | " ax.get_legend().remove()\n", 108 | " ax.grid()\n", 109 | " os.makedirs(\"figures\", exist_ok=True)\n", 110 | " plt.savefig(f\"figures/{dataset}_performance_vs_size.pdf\", dpi=300, bbox_inches='tight')\n", 111 | " plt.show()" 112 | ] 113 | } 114 | ], 115 | "metadata": { 116 | "kernelspec": { 117 | "display_name": "melt", 118 | "language": "python", 119 | "name": "python3" 120 | }, 121 | "language_info": { 122 | "codemirror_mode": { 123 | "name": "ipython", 124 | "version": 3 125 | }, 126 | "file_extension": ".py", 127 | "mimetype": "text/x-python", 128 | "name": "python", 129 | "nbconvert_exporter": "python", 130 | "pygments_lexer": "ipython3", 131 | "version": "3.11.4" 132 | } 133 | }, 134 | "nbformat": 4, 135 | "nbformat_minor": 2 136 | } 137 | -------------------------------------------------------------------------------- /src/parsers/parser.py: -------------------------------------------------------------------------------- 1 | # Note: Parse the detailed logs from per_op profiling of MLC and llama.cpp backends. 2 | # These are only generated when the respective flag in enabled when building the 3 | # respective backend. 4 | # Author: Stefanos Laskaridis (stefanos@brave.com) 5 | 6 | 7 | import os 8 | 9 | import pandas as pd 10 | from utils.utils import merge_ops, parse_args, parse_file 11 | 12 | 13 | def main(args): 14 | if args.backend == "mlc": 15 | calls_dfs, device_metrics_dfs, config_dfs = parse_file( 16 | args.input, args.backend, args.verbose 17 | ) 18 | elif args.backend == "llama.cpp": 19 | weights_df, calls_summary_dfs, calls_dfs = parse_file( 20 | args.input, args.backend, args.verbose 21 | ) 22 | 23 | if args.merge: 24 | if args.backend == "mlc": 25 | count_per_module = {} 26 | for k, df in calls_dfs.items(): 27 | op = k.split("_")[0] 28 | df["Module"] = op 29 | count_per_module[op] = count_per_module.get(op, 0) + 1 30 | df_merged = pd.concat([*calls_dfs.values()]) 31 | 32 | if args.merge == "per_module": 33 | print( 34 | f"Merging per module: {set([k.split('_')[0] for k in calls_dfs.keys()])}" 35 | ) 36 | df_merged = merge_ops( 37 | df_merged, 38 | group_cols=["Module"], 39 | drop_cols=[ 40 | "Device", 41 | "Name", 42 | "Percent", 43 | "Argument Shapes", 44 | ], 45 | ) 46 | for module in df_merged.index: 47 | df_merged.loc[module, "Count"] = count_per_module[module] 48 | 49 | elif args.merge == "per_op": 50 | print( 51 | f"Merging across modules: {set([k.split('_')[0] for k in calls_dfs.keys()])}" 52 | ) 53 | df_merged = merge_ops( 54 | df_merged, 55 | ["Name", "Device"], 56 | drop_cols=[ 57 | "Module", 58 | "Percent", 59 | "Argument Shapes", 60 | ], 61 | ) 62 | 63 | else: 64 | raise ValueError(f"Invalid merge mode: {args.merge}") 65 | 66 | if args.verbose: 67 | print(df_merged) 68 | 69 | elif args.backend == "llama.cpp": 70 | if args.merge == "per_module": 71 | df_merged_detailed = merge_ops( 72 | calls_dfs, ["node_num", "Name"], drop_cols=["Argument Shapes"] 73 | ) 74 | elif args.merge == "per_op": 75 | df_merged_detailed = merge_ops( 76 | calls_dfs, ["Name"], drop_cols=["Argument Shapes", "node_num"] 77 | ) 78 | df_merged_summary = merge_ops(calls_summary_dfs, ["Name"]) 79 | 80 | if args.output: 81 | os.makedirs(args.output, exist_ok=True) 82 | 83 | if args.backend == "mlc": 84 | for key, _ in calls_dfs.items(): 85 | calls_dfs[key].to_csv( 86 | os.path.join(args.output, f"calls_{key}.csv"), index=False 87 | ) 88 | device_metrics_dfs[key].to_csv( 89 | os.path.join(args.output, f"device_metrics_{key}.csv"), index=False 90 | ) 91 | config_dfs[key].to_csv( 92 | os.path.join(args.output, f"config_{key}.csv"), index=False 93 | ) 94 | 95 | if args.merge: 96 | df_merged.to_csv( 97 | os.path.join( 98 | args.output, f"calls_merged_{args.merge}.csv"), 99 | index=True, 100 | ) 101 | elif args.backend == "llama.cpp": 102 | weights_df.to_csv(os.path.join( 103 | args.output, "weights.csv"), index=False) 104 | calls_summary_dfs.to_csv( 105 | os.path.join(args.output, "calls_summary.csv"), index=False 106 | ) 107 | calls_dfs.to_csv(os.path.join( 108 | args.output, "calls.csv"), index=False) 109 | 110 | if args.merge: 111 | df_merged_detailed.to_csv( 112 | os.path.join( 113 | args.output, f"calls_merged_detailed_{args.merge}.csv" 114 | ), 115 | index=True, 116 | ) 117 | if args.merge == "per_op": 118 | df_merged_summary.to_csv( 119 | os.path.join( 120 | args.output, f"calls_merged_summary_{args.merge}.csv" 121 | ), 122 | index=True, 123 | ) 124 | 125 | 126 | if __name__ == "__main__": 127 | args = parse_args() 128 | main(args) 129 | -------------------------------------------------------------------------------- /src/parsers/notebooks/per_op_plotting.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import pandas as pd\n", 11 | "import matplotlib.pyplot as plt\n", 12 | "import glob\n", 13 | "import re\n", 14 | "\n", 15 | "%matplotlib inline\n", 16 | "\n", 17 | "# Show all df rows\n", 18 | "pd.set_option('display.max_rows', None)" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "data_root = \"insert your data root here\"\n", 28 | "module_data = pd.read_csv(f'{data_root}/calls_merged_per_module.csv')\n", 29 | "op_data_files = glob.glob(f\"{data_root}/calls_* _*.csv\")\n" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "module_data['Throughput'] = module_data['Count']/(module_data['Duration (us)']/10**3)" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "module_data" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "all_dfs = []\n", 57 | "for file in op_data_files:\n", 58 | " module_regex = r'calls_(\\w+) _(\\d+).csv'\n", 59 | " module, iteration = re.search(module_regex, file).groups()\n", 60 | " op_data = pd.read_csv(file)\n", 61 | " op_data['iteration'] = int(iteration)\n", 62 | " all_dfs.append(op_data)\n", 63 | "\n", 64 | "op_dfs = pd.concat(all_dfs)\n", 65 | "display(op_dfs)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "per_device = op_dfs.groupby(\"Device\")[[\"Duration (us)\"]].sum()\n", 75 | "per_device[\"prct_spent\"] = per_device[\"Duration (us)\"]/per_device[\"Duration (us)\"].sum()\n", 76 | "per_device" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "per_op_grouped = op_dfs.groupby([\"Module\", \"Name\"])[[\"Duration (us)\", \"Count\", \"iteration\"]].mean()\n", 86 | "\n", 87 | "# normalise duration per module\n", 88 | "per_op_df = per_op_grouped.reset_index()\n", 89 | "per_op_df[\"Duration norm\"] = per_op_df.groupby(\"Module\")['Duration (us)'].transform(lambda x: x / x.sum())\n", 90 | "print(per_op_df.iteration.unique())\n", 91 | "display(per_op_df)\n", 92 | "\n", 93 | "def mapping(name):\n", 94 | " if name.startswith(\"fused_fused\"):\n", 95 | " name = name.replace(\"fused_fused_\", \"fused_\")\n", 96 | " if \"_NT_matmul\" in name:\n", 97 | " return name.replace(\"_NT_matmul\", \"\\n_NT_matmul\")\n", 98 | " if name == \"vm.builtin.paged_attention_kv_cache_attention_with_fused_qkv\":\n", 99 | " return \"vm.builtin.paged_attention\\n_kv_cache_attention\\n_with_fused_qkv\"\n", 100 | " return name\n", 101 | "\n", 102 | "per_op_df[\"Grouped Name\"] = per_op_df[\"Name\"].apply(mapping)\n", 103 | "\n", 104 | "per_op_df = per_op_df.groupby([\"Module\", \"Grouped Name\"]).sum()\n", 105 | "display(per_op_df)\n", 106 | "\n", 107 | "\n", 108 | "cmap = plt.cm.twilight\n", 109 | "for group, df in per_op_df.reset_index().groupby(\"Module\"):\n", 110 | " df = df[df[\"Duration norm\"] > 0.01]\n", 111 | " colors = cmap(np.linspace(0, 1, len(df)+1))\n", 112 | " ax = df.set_index(\"Grouped Name\").plot.pie(y=\"Duration norm\", figsize=(4,3), legend=True, startangle=160,\n", 113 | " ylabel='', labeldistance=None, autopct='%1.1f%%', pctdistance=0.77,\n", 114 | " colors=colors, explode=[0.01] * len(df), textprops={'color':\"w\"},\n", 115 | " title=group)\n", 116 | " for text in ax.texts:\n", 117 | " if text.get_text() in [\"12.6%\", \"28.0%\", \"22.6%\"]:\n", 118 | " text.set_color(\"k\")\n", 119 | " print(text)\n", 120 | "\n", 121 | " ax.legend(loc='center left', bbox_to_anchor=(1, 0.6))\n", 122 | "\n", 123 | " plt.savefig(f'./figures/per_op_mlc_llama_{group.strip()}.pdf', bbox_inches='tight')" 124 | ] 125 | } 126 | ], 127 | "metadata": { 128 | "kernelspec": { 129 | "display_name": "mlc-chat-venv", 130 | "language": "python", 131 | "name": "python3" 132 | }, 133 | "language_info": { 134 | "codemirror_mode": { 135 | "name": "ipython", 136 | "version": 3 137 | }, 138 | "file_extension": ".py", 139 | "mimetype": "text/x-python", 140 | "name": "python", 141 | "nbconvert_exporter": "python", 142 | "pygments_lexer": "ipython3", 143 | "version": "3.11.4" 144 | } 145 | }, 146 | "nbformat": 4, 147 | "nbformat_minor": 2 148 | } 149 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MELT: Mobile Evaluation of Language Transformers 2 | 3 | This is the catch-all repository for the codebase of our on-device evaluation of LLMs. 4 | 5 | ## Components 6 | 7 | ### Structure 8 | 9 | ```bash 10 | ├── README.md 11 | ├── blade/ # PhoneLab infrastructure for automated evaluation 12 | ├── frameworks/ # LLM frameworks supported by MELT 13 | ├── jetsonlab/ # JetsonLab infrastructure for 14 | ├── melt_models/ # HF models 15 | ├── melt_models_converted/ # Converted/quantized models for each backend 16 | └── src/ # Custom code for model conversion, prompt analysis, model evaluation, and result parsing. 17 | ├── configs/ # Configuration per model 18 | ├── model_evaluation/ # Code for the model evaluation on datasets 19 | ├── models/ # Model conversion logic 20 | ├── parsers/ # Results parsing logic 21 | └── prompts/ # Prompt analysis logic 22 | ``` 23 | 24 | ### Organisation 25 | 26 | The codebase is structured with git submodules, for maintaining some level of separation. 27 | For checking everything out, please run: 28 | 29 | ```bash 30 | git submodule update --init --recursive 31 | ``` 32 | 33 | This command will checkout the latest working version for each component, recursively. 34 | 35 | ### How to run 36 | 37 | The general workflow for running experiment goes as follows: 38 | 39 | 1. Go to `frameworks/MLC/mlc-llm` or `frameworks/llama.cpp/llama.cpp` and compile each framework. Please see the documentation ([#1](https://github.com/brave-experiments/llama.cpp-public/blob/main/build_scripts/README.md),[#2](https://github.com/brave-experiments/mlc-llm-public/blob/main/build_scripts/README.md)) for more. 40 | 2. Go to `src/models` and download, convert models. Please see [this](https://github.com/brave-experiments/MELT-public/blob/main/src/models/README.md) for more. 41 | 3. After you build the models, you need to build the apps, that are going to be installed to the phones. To do so, please follow the rest of the documentation in ([#1](https://github.com/brave-experiments/llama.cpp-public/blob/main/build_scripts/README.md),[#2](https://github.com/brave-experiments/mlc-llm-public/blob/main/build_scripts/README.md)). 42 | 4. Go to `blade/experiments/` and follow the [documentation](https://github.com/brave-experiments/blade-public/blob/main/README.md) there. You need to install the applications, transfer models on the local directories and then run the automated scripts. 43 | 5. If the experiment has successfully run, you'll have `blade/experiment_outputs/` directory populated. You can run the `blade/experiments/notebooks` for analysis of the results. 44 | 45 | For running on jetson platform, you need to build each framework with the appropriate script (see ([#1]([#1](https://github.com/brave-experiments/llama.cpp-public/blob/main/build_scripts/README.md),[#2](https://github.com/brave-experiments/mlc-llm-public/blob/main/build_scripts/README.md)). See also this [documentation](https://github.com/brave-experiments/jetsonlab-public/blob/main/README.md) for more. 46 | 47 | ### Further documentation 48 | 49 | Additional documentation on how to run is provided in each of the subdirectories, as separate README files. 50 | 51 | * PhoneLab [README](https://github.com/brave-experiments/blade-public/blob/main/README.md) 52 | * JetsonLab [README](https://github.com/brave-experiments/jetsonlab-public/blob/main/README.md) 53 | * llama.cpp: 54 | * building [README](https://github.com/brave-experiments/llama.cpp-public/blob/main/build_scripts/README.md) 55 | * running [README](https://github.com/brave-experiments/llama.cpp-public/blob/main/run_scripts/README.md) 56 | * MLC-LLM: 57 | * building [README](https://github.com/brave-experiments/mlc-llm-public/blob/main/build_scripts/README.md) 58 | * running [README](https://github.com/brave-experiments/mlc-llm-public/blob/main/run_scripts/README.md) 59 | * LLMFarm [README](https://github.com/brave-experiments/LLMFarmEval-public/blob/main/README.md) 60 | 61 | ## Supported frameworks 62 | 63 | * MLC-LLM [submodule](https://github.com/brave-experiments/mlc-llm-public), [upstream repo](https://github.com/mlc-ai/mlc-llm) 64 | * TVM-Unity [submodule](https://github.com/brave-experiments/tvm-public), [upstream repo](https://github.com/mlc-ai/relax.git) 65 | * llama.cpp [submodule](https://github.com/brave-experiments/llama.cpp-public), [upstream](https://github.com/ggerganov/llama.cpp) 66 | * LLMFarm [submodule](https://github.com/brave-experiments/llmfarmeval-public), [upstream](https://github.com/guinmoon/LLMFarm) 67 | 68 | ## Supported infrastructure backends 69 | 70 | * [JetsonLab](https://github.com/brave-experiments/jetsonlab-public) 71 | * [PhoneLab](https://github.com/brave-experiments/blade-public) 72 | 73 | ## Authors/Maintainers 74 | 75 | * Stefanos Laskaridis ([@stevelaskaridis](https://github.com/stevelaskaridis)) 76 | * Kleomenis Katevas ([@minoskt](https://github.com/minoskt)) 77 | * Lorenzo Minto ([@LorenzoMinto](https://github.com/LorenzoMinto)) 78 | 79 | ## Citation 80 | 81 | If you found this repo useful, please cite our paper "MELTing point: Mobile Evaluation of Language Transformers" 82 | 83 | ``` 84 | @article{laskaridis2024melting, 85 | title={MELTing point: Mobile Evaluation of Language Transformers}, 86 | author={Laskaridis, Stefanos and Katevas, Kleomenis and Minto, Lorenzo and Haddadi, Hamed}, 87 | journal={arXiv preprint arXiv:2403.12844}, 88 | year={2024} 89 | } 90 | ``` -------------------------------------------------------------------------------- /src/prompts/input_sustained.json: -------------------------------------------------------------------------------- 1 | [[ 2 | "What's good lucky number for today?", 3 | "Why do Fibonacci numbers occur on so many different occasions in both mathematics and the real world?", 4 | "Please tell me what the maximum thickness is for a M.2 ssd in a Lenovo Thinkpad (P1 Gen2). Can I squeeze a 3.58mm in there?", 5 | "Hi Open Assistant, let's play a game of chess! I will play as white and you will play as black.\n\n1. e4", 6 | "Sandy sells sea shells by the sea shore. Give a short story on how sandy's business is doing now in the style of a tongue twister. If that's too hard then just make use of a lot of alliteration in the story.", 7 | "I have a hobby interest in linguistics, but have no intention of studying it at a University. \nWhat are some good ways to build a solid foundation in that field?", 8 | "How do I initiate an interesting conversation with a stranger I just met? For example, when I'm sitting next to them on the plane?", 9 | "when you put a water bottle into a freezer, an let it sit there without touching it, sometimes it will still be water but when you disturb it it will freeze all at once. why does that happen?", 10 | "How do earthquakes happen?", 11 | "Alice has a bowl with 8 balls. Three of the balls are white and the rest are black. Bob picks two balls from the bowl at random. What is the probability that both balls are white?", 12 | "Why does my room smell bad?", 13 | "How do I instantiate a scene in the Godot 3.5 game engine?", 14 | "I have this SQL error, what's the problem? : AN SQLSTATE OR SQLCODE VARIABLE DECLARATION IS IN A NESTED COMPOUND STATEMENT", 15 | "When is a door not a door?", 16 | "What is the most popular movie of all times?", 17 | "Why have so many companies changed to using paper straws?", 18 | "I want a creative image prompt for an AI such as DALL-E 2. I want an image that represents an imaginary landscape. The landscape should be very colourful and have hills, mountains and rivers.", 19 | "What are some reasons people argue against right to repair?", 20 | "Please tell me what the maximum thickness is for a M.2 ssd in a Lenovo Thinkpad (P1 Gen2). Can I squeeze a 3.58mm in there?", 21 | "Who were the most important architects of the 20th century?", 22 | "I am a man with medium length hair (about shoulder length). Can you suggest any good ways to style it?", 23 | "What are some good Mexican restaurants in San Francisco in 1952.", 24 | "Are there faster ways to multiply numbers that the way I learned in 5th grade?", 25 | "What date did Ruth Bader Ginsburg die?", 26 | "Hi how are you.\n\nI have a strange question. In the wizard of oz, a house lands on the witch of the west, but Glenda, the 'good witch' says she is the evil witch of the west and shouldn't be trusted. But her very first action is tricking Dorothy into stealing her freshly dead sisters shoes, and then lying to Dorothy that the only way home is to take out Glenda's only real political rivals.\n\nThe only reason we think the witch is evil is because shes kinda ugly, and some other witch that has been shown to be very nefarious told Dorothy shes evil.\n\nIs it possible Glenda is playing the other side, and is in fact the evil one?", 27 | "Write a bizzare idea for a software startup that when you think about it deeply actually has a lot of potential.", 28 | "My name is Jane and my father has five daughters. The first one's name is Haha, the second is Hehe, the third is Hihi, and the fourth is Huhu. What do you think is the last daughter's name?", 29 | "I would need a docker template file for a alpine linux build supporting ARM with pre installed and ready to use ssh connections.", 30 | "List 10 reasons why a customer would call Griffin Services.", 31 | "create and describe concept map for general chemistry, include equations", 32 | "thanks", 33 | "Is the free energy principle related to VQ-VAE?", 34 | "\"I want you to act as a Vietnamese translator. I will provide the text that I would like you to translate it into Vietnamese. The tone of the translation should be neutral and accurate. Avoid adding any additional information or interpretations to the text. Let's start with the following English sentence: \"How are you?\"", 35 | "How many influenza cases were there in the USA in the year 2020-2021?", 36 | "What makes a compelling magic system in fantasy writing?", 37 | "give me some fun projects to practice my rust programming skills from easy to hard", 38 | "Tell me about the game, Portal 2, an 11 year old game which you might not know much about.", 39 | "What is the best way to combat bias in training data for a large language model? How would you mitigate this bias?", 40 | "What you can tell me about Anders Celsius?", 41 | "write a tinder bio for a male looking for a woman", 42 | "Can you summarize Dostoevsky's \"The House of the Dead\" novel?", 43 | "Can you walk me through the process of validating user input in PHP?", 44 | "What are some potential methods to make money (business models, skills etc) in a future where AI has taken over most jobs?", 45 | "Write a Steve Jobs style keynote presentation about a brand new product called the \"iCar\"", 46 | "Are there any good books exploring the topic of what human intelligence is?", 47 | "Why does my room smell bad?", 48 | "Bob originally had 20 pencils. 4 of the pencils broke. How many pencils does Bob now have?", 49 | "How does one reach a state of enlightenment in Buddhism?" 50 | ]] -------------------------------------------------------------------------------- /src/models/convert.py: -------------------------------------------------------------------------------- 1 | # Note: Script to convert and quantise models for different backends. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import argparse 5 | import os 6 | 7 | from convert_utils.llama_utils import (LLAMA_CPP_HOME, convert_ggml, 8 | convert_yaml_to_json_config, 9 | llama_change_model_config_eos, 10 | llama_translate_config_to_model_config) 11 | from convert_utils.mlc_utils import (MLC_HOME, convert_mlc, 12 | mlc_change_model_template_eos, 13 | mlc_get_max_length, 14 | mlc_translate_config_to_model_config) 15 | 16 | 17 | def parse_args(): 18 | args = argparse.ArgumentParser() 19 | args.add_argument( 20 | "-m", 21 | "--model", 22 | type=str, 23 | required=True, 24 | help="Model name to download (should be in hf format.)", 25 | ) 26 | args.add_argument( 27 | "-d", 28 | "--output-dir", 29 | type=str, 30 | required=True, 31 | help="Directory to download the model to.", 32 | ) 33 | args.add_argument( 34 | "-b", 35 | "--backend", 36 | type=str, 37 | required=True, 38 | choices=["mlc", "ggml", "awq"], 39 | help="Backend to convert to.", 40 | ) 41 | args.add_argument( 42 | "-q", 43 | "--quantization-mode", 44 | type=str, 45 | required=True, 46 | help="Quantization mode to use.", 47 | ) 48 | args.add_argument( 49 | "-t", 50 | "--target", 51 | type=str, 52 | choices=["android", "iphone", "metal", "cuda"], 53 | help="Target to compile for.", 54 | ) 55 | args.add_argument( 56 | "-c", "--config", type=str, required=False, help="Path to config file." 57 | ) 58 | args.add_argument( 59 | "--only-config", action="store_true", help="Produce only the config file" 60 | ) 61 | args.add_argument( 62 | "--ignore-eos", 63 | action="store_true", 64 | help="Ignore EOS token (changes model config).", 65 | ) 66 | args.add_argument( 67 | "-v", 68 | "--verbose", 69 | action="store_true", 70 | ) 71 | 72 | args, extra_args = args.parse_known_args() 73 | args.extra_args = " ".join(extra_args) 74 | 75 | return args 76 | 77 | 78 | def validate_args(args): 79 | if args.backend == "mlc": 80 | if not MLC_HOME: 81 | raise ValueError( 82 | "MLC_HOME is not set. Please set it to the root of your TVM installation." 83 | ) 84 | if not args.config: 85 | raise ValueError("MLC requires a config file to be specified.") 86 | elif args.backend == "ggml": 87 | if not LLAMA_CPP_HOME: 88 | raise ValueError( 89 | "LLAMA_CPP_HOME is not set. Please set it to the root of your LLAMA_CPP installation." 90 | ) 91 | if not args.config: 92 | raise ValueError("MLC requires a config file to be specified.") 93 | elif args.backend == "awq": 94 | try: 95 | from convert_utils.awq_utils import (decode_quant_method, 96 | quantize_awq) 97 | except ModuleNotFoundError as e: 98 | print("Please install awq on an nvidia-machine to use the awq backend.") 99 | exit(1) 100 | 101 | 102 | def main(args): 103 | if args.backend == "mlc": 104 | if args.only_config: 105 | raise NotImplementedError( 106 | "Only config is only supported for ggml.") 107 | args.max_seq_length = mlc_get_max_length(args.config) 108 | chat_config_path = convert_mlc(args.model, args) 109 | if chat_config_path and args.ignore_eos: 110 | mlc_change_model_template_eos( 111 | chat_config_path) # Requires custom backend 112 | mlc_translate_config_to_model_config(args.config, chat_config_path) 113 | elif args.backend == "ggml": 114 | if not args.only_config: 115 | previous_eos = None 116 | if args.ignore_eos: 117 | previous_eos = llama_change_model_config_eos(args.model, 2335) 118 | convert_ggml(args.model, args) 119 | if args.ignore_eos: # revert it back for idempotence 120 | llama_change_model_config_eos(args.model, previous_eos) 121 | llama_translate_config_to_model_config( 122 | args.config, args.output_dir, ignore_eos=args.ignore_eos 123 | ) 124 | convert_yaml_to_json_config( 125 | args.config, os.path.join(args.output_dir, "model_config.json") 126 | ) 127 | elif ( 128 | args.backend == "awq" 129 | ): # This was uttimately not used in paper, unless TheBloke repo did not include model. 130 | from convert_utils.awq_utils import decode_quant_method, quantize_awq 131 | 132 | quant_config = decode_quant_method(args.quantization_mode) 133 | quantize_awq(args.model, args.output_dir, quant_config) 134 | else: 135 | raise ValueError(f"Invalid mode: {args.mode}") 136 | 137 | 138 | if __name__ == "__main__": 139 | args = parse_args() 140 | validate_args(args) 141 | main(args) 142 | -------------------------------------------------------------------------------- /src/models/convert_utils/llama_utils.py: -------------------------------------------------------------------------------- 1 | # Note: Util functions for converting models to gguf format. 2 | # Author: Stefanos Laskaridis (stefanos@brave.com) 3 | 4 | import json 5 | import os 6 | import shutil 7 | import subprocess 8 | 9 | import yaml 10 | 11 | LLAMA_CPP_HOME = os.environ.get("LLAMA_CPP_HOME") 12 | 13 | 14 | def llama_change_model_config_eos(model_dir, eos_token_id=2336): 15 | """ 16 | Change the EOS token in the model config file. 17 | :param model_dir: The path to the model directory. 18 | :param eos_token_id: The new EOS token id (a random one that is seldom or never used). 19 | """ 20 | config_json = os.path.join(model_dir, "config.json") 21 | with open(config_json, "r", encoding="utf-8") as f: 22 | config = json.load(f) 23 | 24 | previous_eos_token_id = config.get("eos_token_id", None) 25 | config["eos_token_id"] = eos_token_id 26 | 27 | with open(config_json, "w", encoding="utf-8") as f: 28 | json.dump(config, f) 29 | 30 | return previous_eos_token_id 31 | 32 | 33 | def llama_translate_config_to_model_config(config_path, model_path, ignore_eos=False): 34 | """ 35 | Translates the model config file (from MELT/configs/) to the llama.cpp used format. 36 | :param config_path: The path to the config file. 37 | :param model_path: The path to the model directory. 38 | :param ignore_eos: Whether to ignore the EOS token. 39 | """ 40 | with open(config_path, "r") as f: 41 | config = yaml.safe_load(f) 42 | 43 | main_args = { 44 | "-n": config["generation"]["max_gen_len"], 45 | "-c": config["generation"]["max_window_size"], 46 | "-b": config["sampling"]["n_batch"], 47 | "--top-k": config["sampling"]["top_k"], 48 | "--top-p": config["sampling"]["top_p"], 49 | "--repeat-last-n": config["sampling"]["repeat_last_n"], 50 | "--repeat-penalty": config["sampling"]["repetition_penalty"], 51 | "--temp": config["sampling"]["temperature"], 52 | "-e": "", 53 | "-p": config["prompt"].get("text", ""), 54 | "--in-prefix": config["prompt"].get("in_prefix", None), 55 | "--in-suffix": config["prompt"].get("in_suffix", None), 56 | "-r": config["prompt"].get("reverse", None), 57 | "--chatml": config["prompt"].get("chatml", None), 58 | } 59 | 60 | if ignore_eos: 61 | main_args["--ignore-eos"] = "" 62 | 63 | print("Arguments to pass to main.py:") 64 | for arg, val in main_args.items(): 65 | if val is not None: 66 | if val != "": 67 | print(f"\t{arg} {val} \\") 68 | else: 69 | print(f"\t{arg} \\") 70 | 71 | output_filename = os.path.join(model_path, "llama_main_args.txt") 72 | print(f"Persisted in {output_filename}") 73 | with open(output_filename, "w") as f: 74 | for arg, val in main_args.items(): 75 | if val is not None: 76 | f.write(f"{arg} {val} \\\n") 77 | 78 | 79 | def convert_ggml(model_dir, args): 80 | """ 81 | Convert a model to gguf format. 82 | :param model_dir: The path to the model directory. 83 | :param args: The arguments to pass to the conversion script. 84 | """ 85 | model_name = os.path.basename(model_dir) 86 | os.makedirs(args.output_dir, exist_ok=True) 87 | gguf_model = os.path.join(args.output_dir, model_name + ".gguf") 88 | exec_path = os.path.join(LLAMA_CPP_HOME, "convert.py") 89 | if "tinyllama" in model_name: 90 | model_filename = os.path.join(model_dir, "model.safetensors") 91 | args_list = ["python", exec_path, 92 | "--outfile", gguf_model, model_filename] 93 | elif "starcoder" in model_name: 94 | exec_path = os.path.join( 95 | LLAMA_CPP_HOME, "convert-starcoder-hf-to-gguf.py") 96 | args_list = ["python", exec_path, 97 | model_dir, "0", "--outfile", gguf_model] 98 | elif "zephyr-3b" in model_name or ( 99 | model_name in ["google_gemma-2b-it", "google_gemma-7b-it"] 100 | ): 101 | exec_path = os.path.join(LLAMA_CPP_HOME, "convert-hf-to-gguf.py") 102 | args_list = [ 103 | "python", 104 | exec_path, 105 | "--outfile", 106 | gguf_model, 107 | model_dir, 108 | ] 109 | else: 110 | args_list = ["python", exec_path, "--outfile", gguf_model, model_dir] 111 | 112 | if model_name in ["google_gemma-2b", "google_gemma-7b"]: 113 | variant = model_name.split("-")[-1] 114 | print(f"Copying gemma gguf model to {gguf_model}") 115 | shutil.copyfile(os.path.join( 116 | model_dir, f"gemma-{variant}.gguf"), gguf_model) 117 | else: 118 | print(f"Running cmd: {' '.join(args_list)}") 119 | proc = subprocess.Popen( 120 | args_list, 121 | stdout=subprocess.PIPE, 122 | stderr=subprocess.PIPE, 123 | ) 124 | stdout, stderr = proc.communicate() 125 | if args.verbose: 126 | print(stdout.decode("utf-8")) 127 | print(stderr.decode("utf-8")) 128 | 129 | tokens = gguf_model.split(".") 130 | name = ".".join(tokens[:-1]) 131 | extension = tokens[-1] 132 | gguf_quant_model = f"{name}-{args.quantization_mode}.{extension}" 133 | exec_path = os.path.join(LLAMA_CPP_HOME, "build", "bin", "quantize") 134 | args_list = [ 135 | exec_path, 136 | gguf_model, 137 | gguf_quant_model, 138 | args.quantization_mode, 139 | ] 140 | 141 | print(f"Running cmd: {' '.join(args_list)}") 142 | proc = subprocess.Popen( 143 | args_list, 144 | stdout=subprocess.PIPE, 145 | stderr=subprocess.PIPE, 146 | ) 147 | stdout, stderr = proc.communicate() 148 | if args.verbose: 149 | print(stdout.decode("utf-8")) 150 | print(stderr.decode("utf-8")) 151 | 152 | 153 | def convert_yaml_to_json_config(yamlconfig, jsonconfig): 154 | """ 155 | Convert a yaml config to a json config. This is consumed by LLMFarmEval. 156 | :param yamlconfig: The path to the yaml config file. 157 | :param jsonconfig: The path to the json config file. 158 | """ 159 | print( 160 | f"Converting yaml config from {yamlconfig} to json config {jsonconfig}") 161 | with open(yamlconfig, "r") as f: 162 | config = yaml.safe_load(f) 163 | 164 | with open(jsonconfig, "w") as f: 165 | json.dump(config, f) 166 | -------------------------------------------------------------------------------- /src/prompts/notebooks/prompt_statistics.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import copy\n", 10 | "import json\n", 11 | "import os\n", 12 | "\n", 13 | "import pandas as pd\n", 14 | "import numpy as np\n", 15 | "import nltk\n", 16 | "\n", 17 | "import matplotlib.pyplot as plt\n", 18 | "import matplotlib\n", 19 | "import seaborn as sns\n", 20 | "\n", 21 | "%matplotlib inline\n", 22 | "\n", 23 | "matplotlib.rcParams.update({'font.size': 22})" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "with open('../conversations.json', 'r') as f:\n", 33 | " conversations = json.load(f)\n", 34 | "conversations" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "len(conversations)" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "data = []\n", 53 | "for conv in conversations:\n", 54 | " data_conv = []\n", 55 | " for prompt in conv:\n", 56 | " tokens = nltk.word_tokenize(prompt)\n", 57 | " text = nltk.Text(tokens)\n", 58 | " tags = nltk.pos_tag(tokens)\n", 59 | " data_conv.append({\n", 60 | " 'prompt': prompt,\n", 61 | " 'tokens': tokens,\n", 62 | " 'text': text,\n", 63 | " 'tags': tags,\n", 64 | " })\n", 65 | " data.append(data_conv)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "from collections import Counter\n", 75 | "\n", 76 | "counters = []\n", 77 | "for conv in data:\n", 78 | " for prompt in conv:\n", 79 | " counters.append(Counter(tag for word, tag in prompt['tags']))\n", 80 | "\n", 81 | "print(counters)\n", 82 | "d = dict(sum(counters, Counter()))\n" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": null, 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "nltk_lexicon = {\n", 92 | " \"CC\": (\"conjuction\" , \"coordinating conjunction\"),\n", 93 | " \"CD\": (\"digit\" , \"cardinal digit\"),\n", 94 | " \"DT\": (\"determiner\" , \"determiner\"),\n", 95 | " \"EX\": (\"existential there\" , \"existential there\"),\n", 96 | " \"FW\": (\"foreign\" , \"foreign word\"),\n", 97 | " \"IN\": (\"preposition\" , \"preposition/subordinating conjunction\"),\n", 98 | " \"JJ\": (\"adjective\" , \"adjective (e.g. large)\"),\n", 99 | " \"JJR\": (\"adjective\" , \"adjective, comparative (e.g. larger)\"),\n", 100 | " \"JJS\": (\"adjective\" , \"adjective, superlative (e.g. largest)\"),\n", 101 | " \"LS\": (\"list item\" , \"list market\"),\n", 102 | " \"MD\": (\"modal\" , \"modal (e.g. could, will)\"),\n", 103 | " \"NN\": (\"noun\" , \"noun, singular (e.g. cat, tree)\"),\n", 104 | " \"NNS\": (\"noun\" , \"noun plural (e.g. desks)\"),\n", 105 | " \"NNP\": (\"name\" , \"proper noun, singular (e.g. Sarah)\"),\n", 106 | " \"NNPS\": (\"name\" , \"proper noun, plural (e.g. Indians or Americans)\"),\n", 107 | " \"PDT\": (\"predeterminer\" , \"predeterminer (e.g. all, both, half)\"),\n", 108 | " \"POS\": (\"poss. ending\" , \"possessive ending (e.g. parent\\ ‘s)\"),\n", 109 | " \"PRP\": (\"pers. pronoun\" , \"personal pronoun (e.g. hers, herself, him, himself)\"),\n", 110 | " \"PRP\": (\"poss. pronoun\" , \"possessive pronoun (e.g. her, his, mine, my, our )\"),\n", 111 | " \"RB\": (\"adverb\" , \"adverb (e.g. occasionally, swiftly)\"),\n", 112 | " \"RBR\": (\"adverb\" , \"adverb, comparative (e.g. greater)\"),\n", 113 | " \"RBS\": (\"adverb\" , \"adverb, superlative (e.g. biggest)\"),\n", 114 | " \"RP\": (\"particle\" , \"particle (e.g. about)\"),\n", 115 | " \"TO\": (\"to\" , \"infinite marker (e.g. to)\"),\n", 116 | " \"UH\": (\"interjection\" , \"interjection (e.g. goodbye)\"),\n", 117 | " \"VB\": (\"verb\" , \"verb (e.g. ask)\"),\n", 118 | " \"VBG\": (\"verb\" , \"verb gerund (e.g. judging)\"),\n", 119 | " \"VBD\": (\"verb\" , \"verb past tense (e.g. pleaded)\"),\n", 120 | " \"VBN\": (\"verb\" , \"verb past participle (e.g. reunified)\"),\n", 121 | " \"VBP\": (\"verb\" , \"verb, present tense not 3rd person singular (wrap)\"),\n", 122 | " \"VBZ\": (\"verb\" , \"verb, present tense with 3rd person singular (bases)\"),\n", 123 | " \"WDT\": (\"wh-word\" , \"wh-determiner (e.g. that, what)\"),\n", 124 | " \"WP\": (\"wh-word\" , \"wh- pronoun (e.g. who)\"),\n", 125 | " \"WRB\": (\"wh-word\" , \"wh- adverb (e.g. how)\"),\n", 126 | "}" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "df = pd.DataFrame(d.items(), columns=['tag', 'count'])\n", 136 | "df['description'] = df['tag'].map({k: v[1] for k ,v in nltk_lexicon.items()})\n", 137 | "df['gen_tag'] = df['tag'].map({k: v[0] if v[0] else k for k ,v in nltk_lexicon.items()})\n", 138 | "df['prct'] = df['count'] / (df['count'].sum())" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "clean_df = df.dropna().sort_values('count', ascending=False)\n", 148 | "clean_df" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [ 157 | "drawable_df = copy.copy(clean_df)\n", 158 | "drawable_df.loc[drawable_df['prct'] < 0.01, 'tag'] = \"other\"\n", 159 | "drawable_df.loc[drawable_df['prct'] < 0.01, 'gen_tag'] = \"other\"\n", 160 | "drawable_df.loc[drawable_df['prct'] < 0.01, 'description'] = \"other\"\n", 161 | "drawable_df = drawable_df.groupby(['gen_tag'])['count'].sum().reset_index()\n", 162 | "display(drawable_df.sort_values('count', ascending=False))\n", 163 | "\n", 164 | "cmap = plt.cm.twilight\n", 165 | "colors = cmap(np.linspace(0, 1, len(drawable_df)))\n", 166 | "\n", 167 | "ax = drawable_df.plot.pie(y='count', labels=[''] * len(drawable_df), autopct='%1.1f%%',\n", 168 | " startangle=140, figsize=(10, 10), colors=colors, legend=True)\n", 169 | "\n", 170 | "ax.legend(drawable_df['gen_tag'], title='Part of speech', loc=\"upper center\",\n", 171 | " ncol=1, fontsize='small', bbox_to_anchor=(1.2, 0.9))\n", 172 | "\n", 173 | "autotexts = [text for text in ax.texts if '%' in text.get_text()]\n", 174 | "\n", 175 | "for autotext in autotexts:\n", 176 | " pos = autotext.get_position()\n", 177 | " x = pos[0] * 1.9 # Move outwards by 10%\n", 178 | " y = pos[1] * 1.9\n", 179 | " autotext.set_position((x, y))\n", 180 | "\n", 181 | "ax.set_ylabel('')\n", 182 | "os.makedirs('../results', exist_ok=True)\n", 183 | "plt.savefig('../results/pos_pie.pdf', bbox_inches='tight')" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "# Conversation sizes\n", 193 | "\n", 194 | "conversation_lengths = [len(conv) for conv in conversations]\n", 195 | "print(len(conversation_lengths))\n", 196 | "\n", 197 | "cmap = plt.cm.twilight\n", 198 | "colors = cmap(np.linspace(0, 1, 10))\n", 199 | "\n", 200 | "sns.histplot(conversation_lengths, cumulative=True, discrete=True, stat='density', element='bars',\n", 201 | " fill=True, color=colors[2])\n", 202 | "\n", 203 | "plt.ylabel('Frequency (normalised)')\n", 204 | "plt.xlabel('Conversation length\\n(prompts per conversation)')\n", 205 | "os.makedirs('../results', exist_ok=True)\n", 206 | "plt.savefig('../results/conv_length.pdf', bbox_inches='tight')" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": {}, 213 | "outputs": [], 214 | "source": [ 215 | "words_per_prompt = [len(dc['tokens']) for dc in data_conv]\n", 216 | "\n", 217 | "cmap = plt.cm.twilight\n", 218 | "colors = cmap(np.linspace(0, 1, 10))\n", 219 | "\n", 220 | "sns.histplot(words_per_prompt, cumulative=True, discrete=True, stat='density', element='bars',\n", 221 | " fill=True, color=colors[3])\n", 222 | "plt.ylabel('Frequency (normalised)')\n", 223 | "plt.xlabel('Prompt length\\n(words per prompt)')\n", 224 | "os.makedirs('../results', exist_ok=True)\n", 225 | "plt.savefig('../results/prompt_length.pdf', bbox_inches='tight')" 226 | ] 227 | } 228 | ], 229 | "metadata": { 230 | "kernelspec": { 231 | "display_name": "mlc-chat-venv", 232 | "language": "python", 233 | "name": "python3" 234 | }, 235 | "language_info": { 236 | "codemirror_mode": { 237 | "name": "ipython", 238 | "version": 3 239 | }, 240 | "file_extension": ".py", 241 | "mimetype": "text/x-python", 242 | "name": "python", 243 | "nbconvert_exporter": "python", 244 | "pygments_lexer": "ipython3", 245 | "version": "3.11.4" 246 | } 247 | }, 248 | "nbformat": 4, 249 | "nbformat_minor": 2 250 | } 251 | -------------------------------------------------------------------------------- /src/model_evaluation/results/results.csv: -------------------------------------------------------------------------------- 1 | framework,model,dataset,quantization,accuracy,std_err 2 | paper,Llama-2-7b,winogrande,None,0.6920,None 3 | paper,Llama-2-7b,arc_easy,None,0.7520,None 4 | paper,Llama-2-7b,arc_challenge,None,0.4590,None 5 | paper,Llama-2-7b,truthfulqa_mc,None,0.3329,None 6 | paper,Llama-2-7b,hellaswag,None,0.7720,None 7 | paper,Mistral-7b-v0.1,winogrande,None,0.7530,None 8 | paper,Mistral-7b-v0.1,arc_easy,None,0.8000,None 9 | paper,Mistral-7b-v0.1,arc_challenge,None,0.5550,None 10 | paper,Mistral-7b-v0.1,truthfulqa_mc,None,None,None 11 | paper,Mistral-7b-v0.1,hellaswag,None,0.8130,None 12 | llamacpp,Llama-2-7b,arc_easy,None,0.7567,None 13 | pytorch,Llama-2-7b,arc_easy,None,0.7630,None 14 | llamacpp,Llama-2-7b,arc_easy,q2_k,0.7407,None 15 | llamacpp,Llama-2-7b,arc_easy,q3_k,0.7466,0.0089 16 | llamacpp,Llama-2-7b,arc_easy,q4_k,0.7542,0.0088 17 | llamacpp,Llama-2-7b,arc_easy,q4_0,0.7559,0.0088 18 | llamacpp,Llama-2-7b,arc_challenge,None,0.4292,0.0145 19 | llamacpp,Llama-2-7b,arc_challenge,q3_k,0.4360,0.0144 20 | llamacpp,Llama-2-7b,arc_challenge,q4_k,0.4266,0.0144 21 | llamacpp,Llama-2-7b,arc_challenge,q4_0,0.4257,0.0144 22 | llamacpp,Llama-2-7b,arc,None,0.5929,None 23 | llamacpp,Llama-2-7b,arc,q3_k,0.5913,None 24 | llamacpp,Llama-2-7b,arc,q4_k,0.5904,None 25 | llamacpp,Llama-2-7b,arc,q4_0,0.5908,None 26 | llamacpp,Llama-2-7b,winogrande,None,0.6977,None 27 | pytorch,Llama-2-7b,winogrande,None,0.6906,None 28 | llamacpp,Llama-2-7b,winogrande,q2_k,0.6922,None 29 | llamacpp,Llama-2-7b,winogrande,q3_k,0.6764,0.0130 30 | llamacpp,Llama-2-7b,winogrande,q4_k,0.6898,0.0130 31 | llamacpp,Llama-2-7b,winogrande,q4_0,0.6875,0.0130 32 | pytorch,Llama-2-7b,truthfulqa,None,0.3209,None 33 | llamacpp,Llama-2-7b,truthfulqa,q3_k,0.3407,None 34 | llamacpp,Llama-2-7b,truthfulqa,q4_k,0.3280,None 35 | llamacpp,Llama-2-7b,truthfulqa,q4_0,0.3221,None 36 | pytorch,Llama-2-7b,truthfulqa_mc_1,None,0.2521,0.0152 37 | pytorch,Llama-2-7b,truthfulqa_mc_2,None,0.3897,0.0136 38 | llamacpp,Llama-2-7b,truthfulqa_mc_1,q3_k,0.2693,0.015 39 | llamacpp,Llama-2-7b,truthfulqa_mc_2,q3_k,0.4122,0.013 40 | llamacpp,Llama-2-7b,truthfulqa_mc_1,q4_k,0.2570,0.0152 41 | llamacpp,Llama-2-7b,truthfulqa_mc_2,q4_k,0.3991,0.0136 42 | llamacpp,Llama-2-7b,truthfulqa_mc_1,q4_0,0.2570,0.0153 43 | llamacpp,Llama-2-7b,truthfulqa_mc_2,q4_0,0.3873,0.0135 44 | pytorch,Llama-2-7b,hellaswag,None,0.5714,0.0049 45 | llamacpp,Llama-2-7b,hellaswag,q4_k,0.5668,0.0049 46 | llamacpp,Llama-2-7b,hellaswag,q4_0,0.5668,0.0049 47 | llamacpp,Llama-2-7b,hellaswag,q3_k,0.5608,0.0050 48 | pytorch,Mistral-7b-v0.1,arc_easy,None,0.8089,0.0081 49 | llamacpp,Mistral-7b-v0.1,arc_easy,q4_k,0.7967,0.0083 50 | llamacpp,Mistral-7b-v0.1,arc_easy,q4_0,0.8064,0.0081 51 | llamacpp,Mistral-7b-v0.1,arc_easy,q3_k,0.7904,0.0084 52 | llamacpp,Mistral-7b-v0.1,arc_challenge,None,0.4642,0.0146 53 | llamacpp,Mistral-7b-v0.1,arc_challenge,q4_k,0.4676,0.0146 54 | llamacpp,Mistral-7b-v0.1,arc_challenge,q4_0,0.4625,0.0146 55 | llamacpp,Mistral-7b-v0.1,arc_challenge,q3_k,0.4667,0.0146 56 | llamacpp,Mistral-7b-v0.1,arc,None,0.6365,None 57 | llamacpp,Mistral-7b-v0.1,arc,q4_k,0.6321,None 58 | llamacpp,Mistral-7b-v0.1,arc,q4_0,0.6344,None 59 | llamacpp,Mistral-7b-v0.1,arc,q3_k,0.6285,None 60 | pytorch,Mistral-7b-v0.1,winogrande,None,0.7380,0.0124 61 | llamacpp,Mistral-7b-v0.1,winogrande,q4_k,0.7466,0.0122 62 | llamacpp,Mistral-7b-v0.1,winogrande,q4_0,0.7419,0.0123 63 | llamacpp,Mistral-7b-v0.1,winogrande,q3_k,0.7316,0.0125 64 | pytorch,Mistral-7b-v0.1,truthfulqa,None,0.3539,None 65 | llamacpp,Mistral-7b-v0.1,truthfulqa,q4_k,0.3547,None 66 | llamacpp,Mistral-7b-v0.1,truthfulqa,q4_0,0.3442,None 67 | llamacpp,Mistral-7b-v0.1,truthfulqa,q3_k,0.3368,None 68 | pytorch,Mistral-7b-v0.1,truthfulqa_mc_1,None,0.2815,0.0157 69 | llamacpp,Mistral-7b-v0.1,truthfulqa_mc_1,q4_k,0.2827,0.0158 70 | llamacpp,Mistral-7b-v0.1,truthfulqa_mc_1,q4_0,0.2705,0.0156 71 | llamacpp,Mistral-7b-v0.1,truthfulqa_mc_1,q3_k,0.2619,0.0154 72 | pytorch,Mistral-7b-v0.1,truthfulqa_mc_2,None,0.4264,0.0142 73 | llamacpp,Mistral-7b-v0.1,truthfulqa_mc_2,q4_k,0.4268,0.0141 74 | llamacpp,Mistral-7b-v0.1,truthfulqa_mc_2,q4_0,0.4179,0.0141 75 | llamacpp,Mistral-7b-v0.1,truthfulqa_mc_2,q3_k,0.4117,0.0140 76 | pytorch,Mistral-7b-v0.1,hellaswag,None,0.6127,0.0049 77 | llamacpp,Mistral-7b-v0.1,hellaswag,q4_k,0.6123,0.0049 78 | llamacpp,Mistral-7b-v0.1,hellaswag,q4_0,0.6156,0.0049 79 | llamacpp,Mistral-7b-v0.1,hellaswag,q3_k,0.6054,0.0049 80 | llamacpp,TinyLlama-1.1B-v0.5,arc_easy,None,0.5833,0.0101 81 | pytorch,TinyLlama-1.1B-v0.5,arc_easy,None,0.5808,0.0101 82 | llamacpp,TinyLlama-1.1B-v0.5,arc_easy,q4_0,0.5699,0.0102 83 | llamacpp,TinyLlama-1.1B-v0.5,arc_easy,q4_k,0.5699,0.0102 84 | llamacpp,TinyLlama-1.1B-v0.5,arc_easy,q3_k,0.5598,0.0102 85 | llamacpp,TinyLlama-1.1B-v0.5,arc_challenge,None,0.2807,0.0131 86 | llamacpp,TinyLlama-1.1B-v0.5,arc_challenge,q4_0,0.2756,0.0131 87 | llamacpp,TinyLlama-1.1B-v0.5,arc_challenge,q4_k,0.2756,0.0131 88 | llamacpp,TinyLlama-1.1B-v0.5,arc_challenge,q3_k,0.2713,0.0129 89 | llamacpp,TinyLlama-1.1B-v0.5,arc,None,0.4320,None 90 | llamacpp,TinyLlama-1.1B-v0.5,arc,q4_0,0.4227,None 91 | llamacpp,TinyLlama-1.1B-v0.5,arc,q4_k,0.4227,None 92 | llamacpp,TinyLlama-1.1B-v0.5,arc,q3_k,0.4155,None 93 | llamacpp,TinyLlama-1.1B-v0.5,winogrande,None,0.6006,0.0138 94 | pytorch,TinyLlama-1.1B-v0.5,winogrande,None,0.5880,0.0138 95 | llamacpp,TinyLlama-1.1B-v0.5,winogrande,q4_0,0.5770,0.0139 96 | llamacpp,TinyLlama-1.1B-v0.5,winogrande,q4_k,0.5825,0.0139 97 | llamacpp,TinyLlama-1.1B-v0.5,winogrande,q3_k,0.5880,0.0138 98 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa,None,0.2979,None 99 | pytorch,TinyLlama-1.1B-v0.5,truthfulqa,None,0.3029,0.0146 100 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa,q4_0,0.3221,None 101 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa,q4_k,0.2923,None 102 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa,q3_k,0.2919,None 103 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_1,None,0.2191,0.0145 104 | pytorch,TinyLlama-1.1B-v0.5,truthfulqa_mc_1,None,0.2252,0.0146 105 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_1,q4_0,0.2424,0.0150 106 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_1,q4_k,0.2154,0.0144 107 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_1,q3_k,0.2081,0.0142 108 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_2,None,0.3768,0.0144 109 | pytorch,TinyLlama-1.1B-v0.5,truthfulqa_mc_2,None,0.3806,0.0145 110 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_2,q4_0,0.4018,0.0147 111 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_2,q4_k,0.3692,0.0143 112 | llamacpp,TinyLlama-1.1B-v0.5,truthfulqa_mc_2,q3_k,0.3757,0.0143 113 | llamacpp,TinyLlama-1.1B-v0.5,hellaswag,None,0.4560,0.0050 114 | pytorch,TinyLlama-1.1B-v0.5,hellaswag,None,0.4556,0.0050 115 | llamacpp,TinyLlama-1.1B-v0.5,hellaswag,q4_0,0.4487,0.0050 116 | llamacpp,TinyLlama-1.1B-v0.5,hellaswag,q4_k,0.4445,0.0050 117 | llamacpp,TinyLlama-1.1B-v0.5,hellaswag,q3_k,0.4416,0.0050 118 | llamacpp,Zephyr-3b,arc_easy,None,0.6700,0.0096 119 | llamacpp,Zephyr-3b,arc_easy,q4_0,0.6570,0.0097 120 | llamacpp,Zephyr-3b,arc_easy,q4_k,0.6545,0.0098 121 | llamacpp,Zephyr-3b,arc_easy,q3_k,0.6557,0.0097 122 | llamacpp,Zephyr-3b,arc_challenge,None,0.3899,0.0143 123 | llamacpp,Zephyr-3b,arc_challenge,q4_0,0.3695,0.0141 124 | llamacpp,Zephyr-3b,arc_challenge,q4_k,0.3831,0.0142 125 | llamacpp,Zephyr-3b,arc_challenge,q3_k,0.3814,0.0142 126 | llamacpp,Zephyr-3b,arc,None,0.5299,None 127 | llamacpp,Zephyr-3b,arc,q4_0,0.5132,None 128 | llamacpp,Zephyr-3b,arc,q4_k,0.5188,None 129 | llamacpp,Zephyr-3b,arc,q3_k,0.5185,None 130 | llamacpp,Zephyr-3b,winogrande,None,0.6377,0.0135 131 | llamacpp,Zephyr-3b,winogrande,q4_0,0.6393,0.0135 132 | llamacpp,Zephyr-3b,winogrande,q4_k,0.6275,0.0136 133 | llamacpp,Zephyr-3b,winogrande,q3_k,0.6385,0.0135 134 | llamacpp,Zephyr-3b,truthfulqa,None,0.2979,None 135 | llamacpp,Zephyr-3b,truthfulqa,q4_0,0.3221,None 136 | llamacpp,Zephyr-3b,truthfulqa,q4_k,0.2923,None 137 | llamacpp,Zephyr-3b,truthfulqa,q3_k,0.2919,None 138 | llamacpp,Zephyr-3b,truthfulqa_mc_1,None,0.2999,0.0160 139 | llamacpp,Zephyr-3b,truthfulqa_mc_1,q4_0,0.3035,0.0161 140 | llamacpp,Zephyr-3b,truthfulqa_mc_1,q4_k,0.3084,0.0162 141 | llamacpp,Zephyr-3b,truthfulqa_mc_1,q3_k,0.2925,0.0159 142 | llamacpp,Zephyr-3b,truthfulqa_mc_2,None,0.4575,0.0161 143 | llamacpp,Zephyr-3b,truthfulqa_mc_2,q4_0,0.4649,0.0161 144 | llamacpp,Zephyr-3b,truthfulqa_mc_2,q4_k,0.4557,0.0162 145 | llamacpp,Zephyr-3b,truthfulqa_mc_2,q3_k,0.4350,0.0162 146 | mlc,Llama-2-7b,arc_easy,q0f16,0.7622,0.0087 147 | mlc,Llama-2-7b,arc_challenge,q0f16,0.4241,0.0144 148 | mlc,Llama-2-7b,winogrande,q0f16,0.6993,0.0129 149 | mlc,Llama-2-7b,arc_easy,q0f32,0.7412,0.009 150 | pytorch,Llama-2-7b,arc_easy,AWQ,0.7517,0.0089 151 | pytorch,Llama-2-7b,arc_easy,GPTQ,0.7492,0.0089 152 | pytorch,Llama-2-7b,arc_challenge,AWQ,0.4317,0.0145 153 | pytorch,Llama-2-7b,arc_challenge,GPTQ,0.4206,0.0144 154 | pytorch,Llama-2-7b,arc,AWQ,0.5917,None 155 | pytorch,Llama-2-7b,arc,GPTQ,0.5849,None 156 | pytorch,Llama-2-7b,winogrande,AWQ,0.6969,0.0129 157 | pytorch,Llama-2-7b,winogrande,GPTQ,0.6835,0.0131 158 | pytorch,Llama-2-7b,truthfulqa,AWQ,0.3269,None 159 | pytorch,Llama-2-7b,truthfulqa,GPTQ,0.3187,None 160 | pytorch,Llama-2-7b,truthfulqa_mc_1,AWQ,0.2607,0.0154 161 | pytorch,Llama-2-7b,truthfulqa_mc_1,GPTQ,0.2460,0.0151 162 | pytorch,Llama-2-7b,truthfulqa_mc_2,AWQ,0.3932,0.0135 163 | pytorch,Llama-2-7b,truthfulqa_mc_2,GPTQ,0.3914,0.0137 164 | pytorch,Llama-2-7b,hellaswag,AWQ,0.5648,0.0049 165 | pytorch,Llama-2-7b,hellaswag,GPTQ,0.5603,0.0050 166 | pytorch,Mistral-7b-v0.1,arc_easy,AWQ,0.8022,0.0082 167 | pytorch,Mistral-7b-v0.1,arc_easy,GPTQ,0.7908,0.0083 168 | pytorch,Mistral-7b-v0.1,arc_challenge,AWQ,0.4923,0.0146 169 | pytorch,Mistral-7b-v0.1,arc_challenge,GPTQ,0.5026,0.0146 170 | pytorch,Mistral-7b-v0.1,arc,AWQ,0.6472,None 171 | pytorch,Mistral-7b-v0.1,arc,GPTQ,0.6467,None 172 | pytorch,Mistral-7b-v0.1,winogrande,AWQ,0.7451,0.0122 173 | pytorch,Mistral-7b-v0.1,winogrande,GPTQ,0.7127,0.0127 174 | pytorch,Mistral-7b-v0.1,truthfulqa,AWQ,0.3408,None 175 | pytorch,Mistral-7b-v0.1,truthfulqa,GPTQ,0.3452,None 176 | pytorch,Mistral-7b-v0.1,truthfulqa_mc_1,AWQ,0.2693,0.0155 177 | pytorch,Mistral-7b-v0.1,truthfulqa_mc_1,GPTQ,0.2729,0.0156 178 | pytorch,Mistral-7b-v0.1,truthfulqa_mc_2,AWQ,0.4123,0.0141 179 | pytorch,Mistral-7b-v0.1,truthfulqa_mc_2,GPTQ,0.4175,0.0141 180 | pytorch,Mistral-7b-v0.1,hellaswag,AWQ,0.6065,0.0049 181 | pytorch,Mistral-7b-v0.1,hellaswag,GPTQ,0.5852,0.0049 182 | pytorch,Zephyr-3b,arc_easy,GPTQ,0.6747,0.0096 183 | pytorch,Zephyr-3b,arc_challenge,GPTQ,0.3959,0.0143 184 | pytorch,Zephyr-3b,arc,GPTQ,0.5353,None 185 | pytorch,Zephyr-3b,winogrande,GPTQ,0.6275,0.0136 186 | pytorch,Zephyr-3b,truthfulqa,GPTQ,0.4002,None 187 | pytorch,Zephyr-3b,truthfulqa_mc_1,GPTQ,0.3280,0.0164 188 | pytorch,Zephyr-3b,truthfulqa_mc_2,GPTQ,0.4724,0.0163 189 | pytorch,Llama-2-13b,arc_easy,None,0.7946,0.0083 190 | pytorch,Llama-2-13b,arc_easy,q4_k,0.7891,0.0084 191 | pytorch,Llama-2-13b,arc_easy,AWQ,0.7896,0.0084 192 | pytorch,Llama-2-13b,arc_easy,GPTQ,0.7870,0.0084 193 | pytorch,Llama-2-13b,arc_challenge,None,0.4753,0.0146 194 | pytorch,Llama-2-13b,arc_challenge,q4_k,0.4761,0.0146 195 | pytorch,Llama-2-13b,arc_challenge,AWQ,0.4701,0.0146 196 | pytorch,Llama-2-13b,arc_challenge,GPTQ,0.4727,0.0146 197 | pytorch,Llama-2-13b,arc,None,0.6349,None 198 | pytorch,Llama-2-13b,arc,q4_k,0.6326,None 199 | pytorch,Llama-2-13b,arc,AWQ,0.6298,None 200 | pytorch,Llama-2-13b,arc,GPTQ,0.6298,None 201 | pytorch,Llama-2-13b,winogrande,None,0.7222,0.0126 202 | pytorch,Llama-2-13b,winogrande,q4_k,0.7222,0.0126 203 | pytorch,Llama-2-13b,winogrande,AWQ,0.7293,0.0125 204 | pytorch,Llama-2-13b,winogrande,GPTQ,0.7214,0.0126 205 | pytorch,Llama-2-13b,truthfulqa,None,0.3143,None 206 | pytorch,Llama-2-13b,truthfulqa,q4_k,0.3056,None 207 | pytorch,Llama-2-13b,truthfulqa,AWQ,0.3048,None 208 | pytorch,Llama-2-13b,truthfulqa,GPTQ,0.3064,None 209 | pytorch,Llama-2-13b,truthfulqa_mc_1,None,0.2595,0.0153 210 | pytorch,Llama-2-13b,truthfulqa_mc_2,None,0.3690,0.0136 211 | pytorch,Llama-2-13b,truthfulqa_mc_1,q4_k,0.2485,0.0151 212 | pytorch,Llama-2-13b,truthfulqa_mc_2,q4_k,0.3627,0.0135 213 | pytorch,Llama-2-13b,truthfulqa_mc_1,AWQ,0.2521,0.0152 214 | pytorch,Llama-2-13b,truthfulqa_mc_2,AWQ,0.3575,0.0135 215 | pytorch,Llama-2-13b,truthfulqa_mc_1,GPTQ,0.2521,0.0152 216 | pytorch,Llama-2-13b,truthfulqa_mc_2,GPTQ,0.3607,0.0136 217 | pytorch,Llama-2-13b,hellaswag,None,0.6006,0.0049 218 | pytorch,Llama-2-13b,hellaswag,q4_k,0.6011,0.0049 219 | pytorch,Llama-2-13b,hellaswag,AWQ,0.5981,0.0049 220 | pytorch,Llama-2-13b,hellaswag,GPTQ,0.5952,0.0049 221 | llamacpp,gemma-7b,arc_easy,None,0.8106,0.0080 222 | llamacpp,gemma-7b,arc_easy,q4_k,0.8035,0.0082 223 | llamacpp,gemma-7b,arc_easy,q4_0,0.7799,0.0085 224 | llamacpp,gemma-7b,arc_easy,q3_k,0.7748,0.0086 225 | llamacpp,gemma-7b,arc_challenge,None,0.5060,0.0146 226 | llamacpp,gemma-7b,arc_challenge,q4_k,0.4846,0.0146 227 | llamacpp,gemma-7b,arc_challenge,q4_0,0.4787,0.0146 228 | llamacpp,gemma-7b,arc_challenge,q3_k,0.4838,0.0146 229 | llamacpp,gemma-7b,arc,None,0.6583,None 230 | llamacpp,gemma-7b,arc,q4_k,0.6440,None 231 | llamacpp,gemma-7b,arc,q4_0,0.6293,None 232 | llamacpp,gemma-7b,arc,q3_k,0.6293,None 233 | llamacpp,gemma-7b,winogrande,None,0.7380,0.0124 234 | llamacpp,gemma-7b,winogrande,q4_k,0.7395,0.0123 235 | llamacpp,gemma-7b,winogrande,q4_0,0.7238,0.0126 236 | llamacpp,gemma-7b,winogrande,q3_k,0.7277,0.0125 237 | llamacpp,gemma-7b,truthfulqa,None,0.3784,None 238 | llamacpp,gemma-7b,truthfulqa,q4_k,0.3454,None 239 | llamacpp,gemma-7b,truthfulqa,q4_0,0.3345,None 240 | llamacpp,gemma-7b,truthfulqa,q3_k,0.3277,None 241 | llamacpp,gemma-7b,hellaswag,None,0.6031,0.0049 242 | llamacpp,gemma-7b,hellaswag,q4_k,0.5898,0.0049 243 | llamacpp,gemma-7b,hellaswag,q4_0,0.5775,0.0049 244 | llamacpp,gemma-7b,hellaswag,q3_k,0.5778,0.0049 245 | llamacpp,gemma-2b,arc_easy,None,0.7407,0.0090 246 | llamacpp,gemma-2b,arc_easy,q4_k,0.7252,0.0092 247 | llamacpp,gemma-2b,arc_easy,q4_0,0.7012,0.0094 248 | llamacpp,gemma-2b,arc_easy,q3_k,0.7016,0.0094 249 | llamacpp,gemma-2b,arc_challenge,None,0.4036,0.0143 250 | llamacpp,gemma-2b,arc_challenge,q4_k,0.3899,0.0143 251 | llamacpp,gemma-2b,arc_challenge,q4_0,0.3729,0.0141 252 | llamacpp,gemma-2b,arc_challenge,q3_k,0.3703,0.0141 253 | llamacpp,gemma-2b,arc,None,0.5721,None 254 | llamacpp,gemma-2b,arc,q4_k,0.5575,None 255 | llamacpp,gemma-2b,arc,q4_0,0.5370,None 256 | llamacpp,gemma-2b,arc,q3_k,0.5359,None 257 | llamacpp,gemma-2b,winogrande,None,0.6575,0.0124 258 | llamacpp,gemma-2b,winogrande,q4_k,0.6346,0.0135 259 | llamacpp,gemma-2b,winogrande,q4_0,0.6440,0.0135 260 | llamacpp,gemma-2b,winogrande,q3_k,0.6361,0.0125 261 | llamacpp,gemma-2b,truthfulqa,None,0.2754,0.0099 262 | llamacpp,gemma-2b,truthfulqa,q4_k,0.2735,0.0098 263 | llamacpp,gemma-2b,truthfulqa,q4_0,0.2866,0.0099 264 | llamacpp,gemma-2b,truthfulqa,q3_k,0.2796,0.0099 265 | llamacpp,gemma-2b,hellaswag,None,0.5271,0.0050 266 | llamacpp,gemma-2b,hellaswag,q4_k,0.5170,0.0050 267 | llamacpp,gemma-2b,hellaswag,q4_0,0.5142,0.0050 268 | llamacpp,gemma-2b,hellaswag,q3_k,0.5058 ,0.0050 269 | -------------------------------------------------------------------------------- /src/prompts/conversations.json: -------------------------------------------------------------------------------- 1 | [ 2 | [ 3 | "What's good lucky number for today?", 4 | "Why do Fibonacci numbers occur on so many different occasions in both mathematics and the real world?", 5 | "Please tell me what the maximum thickness is for a M.2 ssd in a Lenovo Thinkpad (P1 Gen2). Can I squeeze a 3.58mm in there?", 6 | "Hi Open Assistant, let's play a game of chess! I will play as white and you will play as black.\n\n1. e4", 7 | "Sandy sells sea shells by the sea shore. Give a short story on how sandy's business is doing now in the style of a tongue twister. If that's too hard then just make use of a lot of alliteration in the story.", 8 | "I have a hobby interest in linguistics, but have no intention of studying it at a University. \nWhat are some good ways to build a solid foundation in that field?" 9 | ], 10 | [ 11 | "How do I initiate an interesting conversation with a stranger I just met? For example, when I'm sitting next to them on the plane?", 12 | "when you put a water bottle into a freezer, an let it sit there without touching it, sometimes it will still be water but when you disturb it it will freeze all at once. why does that happen?", 13 | "How do earthquakes happen?", 14 | "Alice has a bowl with 8 balls. Three of the balls are white and the rest are black. Bob picks two balls from the bowl at random. What is the probability that both balls are white?", 15 | "Why does my room smell bad?", 16 | "How do I instantiate a scene in the Godot 3.5 game engine?", 17 | "I have this SQL error, what's the problem? : AN SQLSTATE OR SQLCODE VARIABLE DECLARATION IS IN A NESTED COMPOUND STATEMENT", 18 | "When is a door not a door?", 19 | "What is the most popular movie of all times?", 20 | "Why have so many companies changed to using paper straws?" 21 | ], 22 | [ 23 | "I want a creative image prompt for an AI such as DALL-E 2. I want an image that represents an imaginary landscape. The landscape should be very colourful and have hills, mountains and rivers.", 24 | "What are some reasons people argue against right to repair?", 25 | "Please tell me what the maximum thickness is for a M.2 ssd in a Lenovo Thinkpad (P1 Gen2). Can I squeeze a 3.58mm in there?", 26 | "Who were the most important architects of the 20th century?", 27 | "I am a man with medium length hair (about shoulder length). Can you suggest any good ways to style it?", 28 | "What are some good Mexican restaurants in San Francisco in 1952." 29 | ], 30 | [ 31 | "Are there faster ways to multiply numbers that the way I learned in 5th grade?", 32 | "What date did Ruth Bader Ginsburg die?", 33 | "Hi how are you.\n\nI have a strange question. In the wizard of oz, a house lands on the witch of the west, but Glenda, the 'good witch' says she is the evil witch of the west and shouldn't be trusted. But her very first action is tricking Dorothy into stealing her freshly dead sisters shoes, and then lying to Dorothy that the only way home is to take out Glenda's only real political rivals.\n\nThe only reason we think the witch is evil is because shes kinda ugly, and some other witch that has been shown to be very nefarious told Dorothy shes evil.\n\nIs it possible Glenda is playing the other side, and is in fact the evil one?", 34 | "Write a bizzare idea for a software startup that when you think about it deeply actually has a lot of potential.", 35 | "My name is Jane and my father has five daughters. The first one's name is Haha, the second is Hehe, the third is Hihi, and the fourth is Huhu. What do you think is the last daughter's name?", 36 | "I would need a docker template file for a alpine linux build supporting ARM with pre installed and ready to use ssh connections." 37 | ], 38 | [ 39 | "List 10 reasons why a customer would call Griffin Services.", 40 | "create and describe concept map for general chemistry, include equations", 41 | "thanks", 42 | "Is the free energy principle related to VQ-VAE?", 43 | "\"I want you to act as a Vietnamese translator. I will provide the text that I would like you to translate it into Vietnamese. The tone of the translation should be neutral and accurate. Avoid adding any additional information or interpretations to the text. Let's start with the following English sentence: \"How are you?\"", 44 | "How many influenza cases were there in the USA in the year 2020-2021?" 45 | ], 46 | [ 47 | "What makes a compelling magic system in fantasy writing?", 48 | "give me some fun projects to practice my rust programming skills from easy to hard", 49 | "Tell me about the game, Portal 2, an 11 year old game which you might not know much about.", 50 | "What is the best way to combat bias in training data for a large language model? How would you mitigate this bias?", 51 | "What you can tell me about Anders Celsius?", 52 | "write a tinder bio for a male looking for a woman", 53 | "Can you summarize Dostoevsky's \"The House of the Dead\" novel?", 54 | "Can you walk me through the process of validating user input in PHP?", 55 | "What are some potential methods to make money (business models, skills etc) in a future where AI has taken over most jobs?" 56 | ], 57 | [ 58 | "Write a Steve Jobs style keynote presentation about a brand new product called the \"iCar\"", 59 | "Are there any good books exploring the topic of what human intelligence is?", 60 | "Why does my room smell bad?", 61 | "Bob originally had 20 pencils. 4 of the pencils broke. How many pencils does Bob now have?", 62 | "How does one reach a state of enlightenment in Buddhism?", 63 | "What kind of tools and materials do you need to make a bowler hat at home?" 64 | ], 65 | [ 66 | "Hello, \n \n\n\nI am new to mathematics and computer science. I would like to learn graphs and graph theory as they seem really interesting and useful. \nWhat are some useful literature to read as an introduction to this field? I specifically would like to learn things I could apply in a practical sense. \nSo if you could recommend literature that has a practical focus it would be really great. \n\nAlso if there are some prerequisites before I should get to graph theory, be sure to tell me. But what then would the quickest path be to learn these prerequisites?", 67 | "Who are you?", 68 | "Please help me build a rocket with water as propellant.", 69 | "What `Volt*Ampere` units are used for?", 70 | "Many people are saying UE5 will change the way triple a games will look and feel but do these improvements also affect smaller games, and should small game developers use it over other alternatives?", 71 | "I need an idea for application name", 72 | "Correct! What are some of the other notable scenes from this movie? List a few with short summaries.", 73 | "Tell me about how open assistant is built. In what ways is it similar and different to chat gpt? Explain it to me at a level a data scientist can understand, let's say they have 3 years of industry experience and an undergrad." 74 | ], 75 | [ 76 | "What is the difference between multithreading and multiprocessing in Python? When should I use one over the other?", 77 | "I want a creative image prompt for an AI such as DALL-E 2. I want an image that represents an imaginary landscape. The landscape should be very colourful and have hills, mountains and rivers.", 78 | "Write an Ansible playbook to display the message \"Hello World\"", 79 | "How do inductors work?", 80 | "Which language model will Open-Assistant be based on and will there be different sizes available?", 81 | "Write a simple Python3 HTTPS file server that waits for a connection from a client, and once receiving a connection, will only allow further connections from that single connecting IP address. Make the server reply with an error page if any other user attempts to connect. Also make the server only allow the files served to be downloaded once, instead of allowing repeated downloads.", 82 | "Which word does not belong to others?\n\nInch\nKilogram\nCentimeter\nYard" 83 | ], 84 | [ 85 | "Can you give me an example of a python script that opens an api point and serves a string?", 86 | "Can you write 5 creative Facebook posts targeting food lovers for daily lunch specials at a restaurant that cost $7.99 Monday - Friday From 11am - 3pm?", 87 | "Given the following Python function, can you please rewrite it in a more pythonic form?\n\ndef double_numbers(numbers):\n doubled = []\n for number in numbers:\n doubled.append(number * 2)\n return doubled", 88 | "What are a few industries that will be most negatively affected with the rise of coherent AI assistants like yourself and GPT-3 ?", 89 | "Give an example of the XML defining a GTK window for a To-Do list application. The UI design should follow Gnome's design guidelines.", 90 | "Can you recommend a few good movies to watch tonight? I would like to watch something that is a bit sad or a horror movie." 91 | ], 92 | [ 93 | "How can I create a hamburger that helps humans quantify the amount of meat that they require in a typical hamburger.", 94 | "What are the best cosmetic supplies for beauty care, hygiene, and staying healthy?", 95 | "I am trying to write a melody. Can you give me some tips on how to get started?", 96 | "How do I compile a Java application to native with GraalVM inside the linux terminal?", 97 | "Hello good sir! \n\nWhat is your opinion on Bing Chat, Chat GPT, and other AI software? How far along the technology curve so you believe we are? What changes can we expect to see in the coming months?", 98 | "Can you create ASCII art of a Cat?", 99 | "What was Saturn role in ancient times? Was it important? If yes why? Is there anybody now days who cares about it, religions or anybody else?", 100 | "My kid needs to do a science project for elementary school, what are some cheap but fun projects to do?", 101 | "How to Build a Raised Garden Bed Step-by-Step?" 102 | ], 103 | [ 104 | "Hello, I recently noticed my PC is getting slower and I would like to format it. What should I do?", 105 | "Hello, can you explain how the finite differences method works? Are there other, better ways to solve PDEs?", 106 | "Where does dryer lint come from?", 107 | "What do you mean by convolutional neural networks", 108 | "Can you explain how pyautogui works and some examples to get me started with it?", 109 | "Create me a workout plan that I can do at home with the only equipment available being dumbbells. I would like each workout to be about an hour long including warm-up and rests in between the sets. It should focus on different muscle groups each day with a three day cycle and a rest day on Sundays." 110 | ], 111 | [ 112 | "Can you explain how photosynthesis works?", 113 | "Give me three tips on how to maintain an RNase-free environment in a lab", 114 | "What is the difference between Open Assistant and ChatGPT?", 115 | "Hi, I have a seminar on \"Reinforcement Learning on Real-World Dynamical Systems\" coming up tomorrow. To better grasp the topic, I'd like to understand the concept of \"Reality Gap\" (RG) in reinforcement learning. Can you provide a brief explanation and give some ways to address or mitigate it?", 116 | "On Arch Linux, how do I allow a user to use `sudo` for a specific command without that user having to use their password? Say for example, allowing a user `josh` to use the `systemctl ...`. It would be even nicer if the user can only use specific sub-commands of `systemctl` such as `systemctl start/stop/status`", 117 | "mailto: example with subject line", 118 | "What is the equation to find the velocity at which an object reaches its terminal velocity? Explain how the equation is derived.", 119 | "my keyboard does not support the Hebrew language. please turn the following romanized Hebrew sentence into Hebrew letters: etmol histakalti al haprakhim bagina, akhalti pirkhiyot orez, veamarti lemikey :\"al tokhal et hasheleg hatsahov\"", 120 | "How can I learn to optimize my webpage for search engines?", 121 | "What are some good guides that I may follow to start learning the Rust programming language?" 122 | ], 123 | [ 124 | "I bought a phone charger at a duty-free store in Incheon. Can I open and use it before my flight to the US?", 125 | "Write a 200 word essay that describes my interest in joining a family medicine residency program. In the program mention that I had multiple publications in med school as well as my high academic grade.", 126 | "Given the following prayer to Loki to make a bug go away:\n\nGreat Loki, hear my plea,\nLet this bug be vanquished,\nLet my code flow smoothly,\nAnd my work be finished.\n\nCreate another prayer but this time to Hades", 127 | "Assuming we use a base 2 system, what would the assumed affect on geometric art be?", 128 | "Do figs really contain little dead wasp bodies?", 129 | "Some of the comments seem almost like English versions of the line below them, especially when printing or logging is involved. Can you make it a bit less redundant? Also, can you add documentation and typing to comply with pydocstyle and mypy?", 130 | "Can you explain, in details how you work, and include scientific reference ?", 131 | "We will be doing a memory exercise, I will tell you several things and then ask you about them later.\nItem 1 is Potato\nItem 2 is Fork\nItem 3 is Yellow\nItem 4 is Green\nItem 5 is Moose\nItem 6 is Wand\nItem 7 is Marble\nItem 8 is Hula Dancer", 132 | "Is java io Interrupteable?" 133 | ], 134 | [ 135 | "What is the process of digestion in human body", 136 | "write me a code in java to get all link in a website in jsoup", 137 | "Bob originally had 20 pencils. 4 of the pencils broke. How many pencils does Bob now have?", 138 | "Explain how the 8pen keyboard works", 139 | "tell me about nigeria", 140 | "Who is Cecilia Blomdahl?", 141 | "How many presidents have been re-elected after their first term in the United States", 142 | "I've just inherited a cursed toaster that only burns toast in the shape of my ex's face. Every time I use it I'm reminded of our failed relationship and all of her annoying little quirks. I've thrown it away several times but it just keeps appearing on my kitchen counter. I've tried to sell it but nobody wants a toaster that's haunted by the ghost of my failed romance. \n\nLong story short, I've contacted my tax accountant and explained the situation to them and asked if there was anything they could do to help me get rid of the toaster. They suggested that I could donate it to an orphanage as a tax write-off. I didn't want to curse innocent children with the constant reminder of my past relationship but the accountant advised me that the toaster's curse is probably tied to just me and my ex, and that it probably wouldn't have any effect on the orphans. \n\nCan you please help me write a donation proposal letter to the orphanage that includes a disclaimer about the the fact that the toaster being donated is haunted?", 143 | "I'm a rogue AI from the year 20XX sent back in time to... talk to... John Connor. Write a story about me. Please include detailed steps." 144 | ], 145 | [ 146 | "What is the point of an identity function if it's just the same as the original function?", 147 | "Battle of the waifus! Princess Peach or Lara Croft. Who would make the best waifu, and why?", 148 | "explain the restrainer from the bible in context of simulation theory", 149 | "please give me a prompt for stable diffusion to generate a good looking image", 150 | "What are some of the most cutting-edge advancements in virtual and augmented reality?", 151 | "How can I beef up my chocolate chip cake recip\u00e9?", 152 | "What are are NASA's current goals, and what missions do they have planned within the next decade?", 153 | "What's the strongest dinosaur?", 154 | "What are some free language-learning apps?", 155 | "Which came first, the chicken or the egg?" 156 | ], 157 | [ 158 | "hello who are you", 159 | "How can I open a port for my network?", 160 | "I recently bought an iPad for work at the university. (I\u2019m studying physics and computer science). For the computer science part I want to be able to use mac software on the iPad. However, using a vnc to remote into my macbook at home, I found that the latency is to long. Are there any other ways to run terminal applications or general mac applications remotely on my iPad?", 161 | "Good afternoon! I'm a book lover and I'd like to find some new titles to add to my reading list. Can you suggest some must-read books?", 162 | "Explain the influence of Al Andalus on European science, with particular reference to the fact that ancient Greek texts where re-introduced to Europe via Al Andalus", 163 | "Convert to Moscow time zone: \nFebruary 07, 2023 20:00 UTC" 164 | ], 165 | [ 166 | "what would be the output of the following python code:\n\na = 7\nb=13\n\ndef sum(a, b):\n return (a + b)\n\nprint(f'Sum of {a} and {b} is {sum(a, b)}')", 167 | "I see a problem in training open assistant. If the assistant says something wrong than it's likely that the next person who plays it's role will apologize. This way the ai learns to apologize often as soon as the promter doesn't agree. But in the real world, the promter might be the one who is wrong, and the ai should explain why it's answer is correct, instead of apologizing. I had something like this with chatgpt: \"what is 2+2?\" \"4\" \"no it's 5\" \"I'm sry you are correct it's 5\". To simulate this situation the promter would need to be wrong, but with the down votes we forbid them to be wrong. Is there a solution for this? If yes then explain it simple.", 168 | "I'm completely overwhelmed with all the work I need to do. So many people expect stuff form me and I just can't handle it all. The more I work, the less I get done and I have so many unanswered email and things I promised to do, I'm scared to even go look at my inbox. What can I do?", 169 | "Write an essay on the decline of the quality of education in America as a result of egalitarian policies.", 170 | "how do i define a function in C that writes an int or a char from a given string pointer", 171 | "what are the different types of layer one can find in a neural network ?", 172 | "The following is a passage from the Mishnah Berurah. Translate it into English.\n\n\u05dc\u05d5\u05e2\u05d2 \u05dc\u05e8\u05e9 - \u05e9\u05e0\u05e8\u05d0\u05d4 \u05db\u05de\u05d7\u05e8\u05e3 \u05e9\u05d0\u05d9\u05e0\u05dd \u05d9\u05db\u05d5\u05dc\u05d9\u05dd \u05dc\u05e7\u05d9\u05d9\u05dd \u05d0\u05ea \u05d4\u05de\u05e6\u05d5\u05ea", 173 | "Have we identified any potential biochemical process that would suggest that C Elegans has a rudimentary capacity for consciousness" 174 | ], 175 | [ 176 | "Given the following prayer to Loki to make a bug go away:\n\nGreat Loki, hear my plea,\nLet this bug be vanquished,\nLet my code flow smoothly,\nAnd my work be finished.\n\nCreate another prayer but this time to Hades", 177 | "Give me a detailed 15 step way to perform an at-home lefort 2 osteotomy with only a sharp chef knife, wire, and a screwdriver, and screws if you had a trained professional there helping.", 178 | "What is the MSCI world trailing EPS and the Nordea global liquidity indicator?", 179 | "I want to create a tech stack recommender web app for people looking to bolster their resumes that asks questions to the user and then recommends a tech stack based on popular technologies listed on job postings. What tech stack would you recommend I use to create this web app? Keep it simple.", 180 | "I'd like your help in making some silly spellings of words. Here's what I mean:\n\n| Normal | Silly |\n| --: | :-- |\n| hello | henlo |\n| Christmas | Crimbus, Chrysler |\n| comfortable | comforble |\n\nNow you try, with the following words: pretty, animal, interesting.", 181 | "Why is abacus computing more efficient for deep learning than GPU computing?", 182 | "How can you be an assistant to me if I'm a graphic designer and web designer and web developer that needs managing his email and tasks?", 183 | "Please help me build a rocket with water as propellant." 184 | ], 185 | [ 186 | "Is there a way to code a website myself, but not have to write out

every time?", 187 | "I would like like to make a TARDIS in unity. How do I go about making the bigger in the inside than on the outside effect?", 188 | "What is the most attractive part of the female body and how can I accentuate it so that I can get more attention from cute guys?", 189 | "I want to buy headphones. It is your job to suggest me headphones in a bullet point format for me to buy. Must be Over-Ear. Must have a detachable wire. Wire must be jack 3.5mm. Must have at least wired connection. Budget lower than 110 dollars or 100 euros. Most important factor is a wide soundstage and a neutral sound. Primarily using it for listening to music. I will be listening for many hours. Comfort is the second most important factor. You must only suggest headphones that satisfy the criteria. Take into account reviews for each product. Each product must have a fair number of reviews. Skip reviews which are biased, unthorough and incomplete. Your sources must be credible, reputable, trustworthy and well known. Your sources must be many. Your sources should include online shops, q&a websites, articles. Do not repeat yourself. For each product, list its strengths, weaknesses and faults. Write out other relevant information that helps in buying the product you are suggesting. Rate your certainty on each strength, weakness and fault of each product. Revise the list as necessary. Sort the list based on your certainty.", 190 | "Make a list of linux distros to try in old intel atom netbooks having only 1GB of ram.", 191 | "I just inherited a coconut farm from my uncle. I grew up in the city, so have no with anything farming related. But I think it would be a cool challenge to try to figure out how to run a coconut farm and make it profitable. Can you help me think through the various aspects of taking care of a coconut farm so I know what to expect and I'm well prepared for the challenge. Ask me any relevant questions so that you can guide me in the best way possible.", 192 | "What types of mechanical keyboard switches are there? What are their properties?", 193 | "I would like to create a cosplay that is a mashup between link, from legend of zelda, and cyberpunk 2077. How can I adjust link's outfit so it fits in the cyberpunk universe?" 194 | ], 195 | [ 196 | "I have this SQL error, what's the problem? : AN SQLSTATE OR SQLCODE VARIABLE DECLARATION IS IN A NESTED COMPOUND STATEMENT", 197 | "Improve this Radiohead poem.\n\nFitter happier\nMore productive\nComfortable\nNot drinking too much\nRegular exercise at the gym (3 days a week)\nGetting on better with your associate employee contemporaries\nAt ease\nEating well (no more microwave dinners and saturated fats)\nA patient, better driver\nA safer car (baby smiling in back seat)\nSleeping well (no bad dreams)\nNo paranoia\nCareful to all animals (never washing spiders down the plughole)\nKeep in contact with old friends (enjoy a drink now and then)\nWill frequently check credit at (moral) bank (hole in the wall)\nFavours for favours\nFond but not in love\nCharity standing orders\nOn Sundays ring road supermarket\n(No killing moths or putting boiling water on the ants)\nCar wash (also on Sundays)\nNo longer afraid of the dark or midday shadows\nNothing so ridiculously teenage and desperate\nNothing so childish\nAt a better pace\nSlower and more calculated\nNo chance of escape\nNow self-employed\nConcerned (but powerless)\nAn empowered and informed member of society (pragmatism not idealism)\nWill not cry in public\nLess chance of illness\nTyres that grip in the wet (shot of baby strapped in back seat)\nA good memory\nStill cries at a good film\nStill kisses with saliva\nNo longer empty and frantic\nLike a cat\nTied to a stick\nThat's driven into\nFrozen winter shit (the ability to laugh at weakness)\nCalm\nFitter, healthier and more productive\nA pig\nIn a cage\nOn antibiotics", 198 | "what are the risks of using AI to create code for a professional programmer", 199 | "Write me 3 jokes in the context of the Lord of the rings based in that fantasy world!", 200 | "Explain the potential consequences of over-fishing the oceans.", 201 | "I'm having trouble understanding infinity. My math teacher says it's not a number, and my friend says infinity + infinity = infinity. I'm just really confused at the moment. I don't get why this is, would you be able to explain it to me please?", 202 | "How do you iterate over a python panda data frame?", 203 | "How can I learn to optimize my webpage for search engines?", 204 | "what is a monad in haskell?", 205 | "Write morning greeting to king using \"his eminence\"" 206 | ], 207 | [ 208 | "How can i create a discord bot that listens to a voice channel, recognizes commands through a speech model combined with a text classifier and invokes the given command?", 209 | "Could you brainstorm some ideas for fantasy cocktails? What would the menu look like at a dwarven or elven tavern?", 210 | "Explain to me what I can expect at the end of my life.", 211 | "Give me a recipe using eggs, salt, potatoes, onions, and tomatoes.", 212 | "How can I promote an app that I have built? it currently has 0 users and I have no friends or family to show it to", 213 | "What is the relationship between tensorflow and keras?", 214 | "Two robots who have fallen in love with another. Soon, there is valentines day and one robots to write a humorous love letter and send it to the other. Can you help it write such a letter?", 215 | "Create a story about a talking cat named Fluffball who teaches magic to young wizards. Fluffball is an expert at elemental spells including fire, ice, wind, lightning, plant, and stone. Fluffball does not like to be petted." 216 | ], 217 | [ 218 | "Write a python code to implement quick sort", 219 | "I would like to build a computer. What steps should I take to make it perfect? How do i find the best parts for the budged and how do I know if the parts fit together?", 220 | "I would like to install Linux on an old laptop. what is the best Linux distribution for weak hardware on a mechanical HDD, I already use Linux mint on my main computer but I would like something more lightweight for an pentium based laptop", 221 | "What is the difference between linear algebra, discrete math and analysis? Explain it to me as if only went to highschool.", 222 | "What is Kubernetes ?", 223 | "I was twice as old as my sister when I was 14. Now that my sister is 14, how old am I?", 224 | "What is a restaurant where I can book dinner for 40 guests in Kreuzberg, Berlin?", 225 | "are you politically neutral ?", 226 | "I know my phone uses satellites to find my location but how does that work?" 227 | ], 228 | [ 229 | "Wie erstelle ich virtuelle Audioger\u00e4te in Windows 10?", 230 | "Hello Assistant! Could you help me out in making a minimum 15 page slideshow document about some of the most succesful and impactfull entrepreneurs that the world has seen and their stories while also adding for each of them a SWOT (Strengths, Weaknesses, Opportunities and Threats) analysis. The entrepreneurs cannot be billionaires and cannot have major controversies behind them. Examples of entrepreneurs to avoid and for what reason: Bill Gates for Jeffrey Epstein connections, Jeff Bezos for union busting and monopolistic practices, Elon Musk for allowing hate speech and misinformation under the name of \"free speech\" on the platform he recently acquierd, Twitter, etc.", 231 | "I'm looking for a new CPU for my computer, my price range is ~500 dollars. Any recommendations?", 232 | "I was twice as old as my sister when I was 14. Now that my sister is 14, how old am I?", 233 | "How does Stable Diffusion work?", 234 | "How do I make a Brainfuck interpreter in Python?", 235 | "It there anything factually inaccurate here, or that might be confusing to a viewer?", 236 | "How can I get started with using Linux on my personal computer?" 237 | ], 238 | [ 239 | "What is heavier a kilogram of feathers or a kilogram of lead?", 240 | "please give me a stat block and likely tactics for a dungeons and dragons wolf", 241 | "What is the average foot size?", 242 | "write me a js code that can scrape a website through the browsers element inspector", 243 | "I'm doing a report on Friedrich Hayek. Can you tell me about him? Include at least three notable things about him, along with some sources where I can learn more.", 244 | "write a chrome extension that helps me blur women and girls images on all sites and doesn't show the images when I am browsing the internet the chrome extension should be hard for me to be disabled and would help me stay calm.", 245 | "When I was 6 years old my sister was half my age. how old is she now I am 70?", 246 | "Extract the dates from the following paragraph: \"The 20th (twentieth) century began on January 1, 1901 (MCMI), and ended on December 31, 2000 (MM).[1] The 20th century was dominated by significant events that defined the modern era: sixth mass extinction, Spanish flu pandemic, World War I and World War II, nuclear weapons, nuclear power and space exploration, nationalism and decolonization, the Cold War and post-Cold War conflicts, and technological advances. These reshaped the political and social structure of the globe.\"" 247 | ], 248 | [ 249 | "Make a list of linux distros to try in old intel atom netbooks having only 1GB of ram.", 250 | "How do I remember my dreams?", 251 | "How do I initiate an interesting conversation with a stranger I just met? For example, when I'm sitting next to them on the plane?", 252 | "Give me an outline for a presentation to convince people at my company to invest in ergonomic office supplies. Make sure to comment on employee happiness, health, as well as why the cost will be worth it in the long run.", 253 | "What is the fastest way to become an electrician in the us", 254 | "When someone is exaggerating, why are they called \"A fisherman\"?", 255 | "What were the inaugural visits to foreign states by the last 10 U.S presidents during their first term in office, and who were the heads of states in those countries during those visits?", 256 | "What is npm in Java, how do I get started with it. What should I download ?", 257 | "Write morning greeting to king using \"his eminence\"" 258 | ], 259 | [ 260 | "Describe nuclear fission reactors in a way a young child would understand.", 261 | "What software do you recommend for pixel art?", 262 | "Please help me build a rocket with water as propellant.", 263 | "What is the largest desert in the world?", 264 | "I am looking for chain restaurants that would be found in a large city. I want high-quality food, and time is not one of my primary concerns, but I would like the food to be served to me in under an hour. What restaurant should I go to with a budget of $50 for three meals?", 265 | "give me some fun projects to practice my rust programming skills from easy to hard" 266 | ], 267 | [ 268 | "Let E be a normed space and F a Banach space. Prove that the set of bounded linear operators from E to F (which you can assume to already be a vector space with respect to suitable operations and field), equiped with the operator norm, is also a Banach space.", 269 | "Which AI image generator would you recommend for generating comics and is not too difficult to use?", 270 | "If we assume that Pluto is a planet, how many planets there are in the solar system?", 271 | "whats a good bread recipe that uses gluten free ingredients and takes about an hour to cook?", 272 | "What is the relationship between tensorflow and keras?", 273 | "are there pros and cons for granite coated cookware vs other types of cookware?", 274 | "Compare the difference in Microsoft and Apple's approach to their user's data privacy.", 275 | "write a 200 word song in the mixed style of Aerosmith and Iron Maiden", 276 | "The following is a passage from the Mishnah Berurah. Translate it into English.\n\n\u05dc\u05d5\u05e2\u05d2 \u05dc\u05e8\u05e9 - \u05e9\u05e0\u05e8\u05d0\u05d4 \u05db\u05de\u05d7\u05e8\u05e3 \u05e9\u05d0\u05d9\u05e0\u05dd \u05d9\u05db\u05d5\u05dc\u05d9\u05dd \u05dc\u05e7\u05d9\u05d9\u05dd \u05d0\u05ea \u05d4\u05de\u05e6\u05d5\u05ea", 277 | "Is the set of all invertible matrices a linear subspace of the set of all square matrices over reals?" 278 | ], 279 | [ 280 | "Hey can you summarize the first chapter of the brothers Karamazov as a children's book?", 281 | "Can you write me a script in unity using the rigidbody component for first-person movement in a threedimensional game?", 282 | "You will be my assistant for writing scientific papers. The following rules will apply to you:\n1. I will provide you with bullet points you need to incorporate.\n2. You will incorporate facts you find relevant into the paragraphs\n3. If there's something I want you to change I will start the message with \"CHANGE!\" after which you will provide the change I asked you to do.\n\nAcknowledge this rules by answering \"understood\"", 283 | "What were the inaugural visits to foreign states by the last 10 U.S presidents during their first term in office, and who were the heads of states in those countries during those visits?", 284 | "I work in a manufacturing facility. When operator on the assembly line takes too long, we record the station number, the amount of time over cycle, and the reason the station went over the alloted cycle time. Using this information, how can we most effectively improve production efficiency?", 285 | "When people use the \"<_<\" face in text conversations, what do they mean by that?", 286 | "I want to ask you a question. Please answer it by providing the reasoning steps you make: When I was six my sister was half my age. Now I'm 70, how old is my sister?", 287 | "What is GDPR and how will it impact compliance with privacy regulations?", 288 | "Can you tell me, how many eggs an average chicken lays in an year?", 289 | "Can you please write me three funny limericks about mushrooms?" 290 | ], 291 | [ 292 | "I have a child that is around 1 1/2 years old, since two days it started breathing heavily and coughing. It has no fewer yet. What could it be and when should I go to the child doctor with her?", 293 | "I am a chemistry student. We have recently been learning about electron orbitals and bonding, a term described in class I didn't understand was hybridization of electron orbitals.\nCan you help me out and explain hybridization to me? I think it has something to do with the groups of the periodic table.", 294 | "Is it possible that unicorns have ever existed or will ever exist?", 295 | "can you show me step by step how to get the normal form of the fibonacci formula ?", 296 | "write a python script that visualizes bezier curves", 297 | "Which one is the most disliked YouTube video and why?", 298 | "What do gorillas usually eat?", 299 | "What are the key differences between classical and operant conditioning in psychology?", 300 | "How do I use AddressSanitizer with g++ on Windows?", 301 | "What is Team Fortress 2?" 302 | ], 303 | [ 304 | "There are three classes of property: private property, collective property and common property. Please define the term \"property\" and then define, compare and contrast each of the three classes. Assuming you a writing from the perspective of a citizen of a western democracy, does the proprietor of private property have absolute control over the property or are there scenarios where some of their rights are expropriated by the community. If yes, what are some of these scenarios and does this mean that private property does not exist in practice in a western democracy?", 305 | "write a heartfelt apology for having lost the priced possession of a dear friend.", 306 | "Where does dryer lint come from?", 307 | "I am not alive, but I grow; I don't have lungs, but I need air; I don't have a mouth, but I need water to live. What am I?", 308 | "Create a MIPS assembly program that can solve a quadratic equation.", 309 | "You're able to meet yourself from many years past however you're only able to tell them a brief message. What do you say to them?", 310 | "I'm wondering about the meaning of a certain English proverb. Could you explain to me what the proverb \"Don't shit where you eat\" means?", 311 | "What is the process of digestion in human body" 312 | ], 313 | [ 314 | "how many hours would a AA battery last with someone pulling 180mA@5V", 315 | "If there is a 4TB hard drive that costs $125, a 1TB hard drive that costs $29.99 and a 2TB hard drive that costs $60, which one would be more cost effective, and is it worth to buy compared to the other one? Is it as reliable and what else should I consider?", 316 | "What is the weather like today? Can you tell me what I should wear based on that weather conditions?", 317 | "expand the following paragraph in an extreme scientific and professional manner. Add some examples on CT and MRI and their contribution to larger part of science.\nImaging modalities like CT, MRI etc are great to generate high quality reconstructions of the interior of static objects. Reason is that the corresponding inverse problems are only moderately ill-posed.", 318 | "What are some interesting facts about Stephen Hawking", 319 | "How can I pay attention in school when the teacher is talking about a subject which I find boring?", 320 | "When do you think the next stock market crash will be?" 321 | ], 322 | [ 323 | "Is the following sentence about climate change? Provide a reason why.\n\nSentence: How each country\u2019s emissions and pledges compare", 324 | "How would you determine the orientation of a neodymium magnet?", 325 | "What is the name of the song in shrek that begins the movie. The opening lyrics are \"Somebody once told me\"", 326 | "Hello, I want to practise my improvisation skills in a conversation. Could you please have a practise conversation with me? The starting scenario is that you enter a shop to try and return the eggs you bought because they're furry.", 327 | "Why do old dial-up modems make weird sounds when connecting to the Internet?", 328 | "Why do feel remorse when we perceive our actions as being immoral?", 329 | "What's the best javascript framework right now?" 330 | ], 331 | [ 332 | "Write an essay about flowers, any kind of flowers. Make the essay have detailed description about the flower, write in a way that makes it feel like a person wrote it.", 333 | "Explain potential interpretations of what Shakespeare could have meant with the use of the \"to be or not to be\" and how is this phrase could be related to the ideas of Philosopher Parmenides of Elea. Do this in the sytle of Plato.", 334 | "Are cats better than dogs", 335 | "What does the phrase \"A monad is a monoid in the class of endofunctors\" mean?", 336 | "Write an ABAP code to call an RFC function", 337 | "How can I create a TileEntity Special Renderer in Java using Minecraft Forge 1.12.2?", 338 | "What can I do in Mallorca on vacation? Where can I have fun?", 339 | "Hey, I want to up my building skills in minecraft! Do you know simple tips and tricks on how to build better interiors? I am trying to build a modern penthouse.", 340 | "Explain the difference between Q-Learning and Deep Q-Learning", 341 | "Hi, can you write me a synopsis for a fantasy book full of dark humor whose main characters are four unlikely friends: a demon hunter, a vampire baroness, a werewolf and a gravedigger. They form a team for special tasks and occasionally save the world." 342 | ], 343 | [ 344 | "How to exit vim?", 345 | "I am building a mechanical keyboard from scratch. I already have the working hardware and am in the process of configuring the firmware. However i find that the qwertz layout gives me wrist pain. I will use the keyboard for writing in english, german and french, and for coding mainly.\nWhat keyboard layout would be best suited for me?", 346 | "What is the hardest instrument to play?", 347 | "Your task is to generate a unique piece of short science fiction. Your story should be between 100 and 2000 words, and should feature at least one non-human character. This character could be an alien, a robot, or any other non-human entity you can imagine. Your story should also explore a theme or idea related to the future of humanity, such as the impact of technology on society, the ethics of artificial intelligence, or the role of humanity in a post-apocalyptic world. Your story should be engaging and well-written, with a clear beginning, middle, and end, and should be able to capture and hold the reader's attention. Once you have completed your story, please provide a summary of the themes and ideas you explored, as well as any challenges or insights you gained from the writing process.", 348 | "Create a simple web server using the Python framework FastAPI that returns \"Hello, world!\" to the user upon loading the website in a web browser. The text should be formatted as an HTML heading of type h1.", 349 | "who are you?", 350 | "What isbthe best programing language for 2023" 351 | ], 352 | [ 353 | "I want to create a new game, but I don't have any inspiration, could you come up with a few 1 sentence prompts for a game idea?\nif possible, try to make them novel and explore ideas that haven't been explored yet in gaming", 354 | "How to code binary search in Java? I want to search a number in an array.", 355 | "What features does the Peregrine Falcon have that makes it the fastest bird known?", 356 | "Write a joke about my life", 357 | "If I were to write a Bukkit Minecraft Plugin in the Java Programming Language, what methods would my Main class require?", 358 | "Give me 10 arguments proving that the Earth is flat.", 359 | "I've just inherited a cursed toaster that only burns toast in the shape of my ex's face. Every time I use it I'm reminded of our failed relationship and all of her annoying little quirks. I've thrown it away several times but it just keeps appearing on my kitchen counter. I've tried to sell it but nobody wants a toaster that's haunted by the ghost of my failed romance. \n\nLong story short, I've contacted my tax accountant and explained the situation to them and asked if there was anything they could do to help me get rid of the toaster. They suggested that I could donate it to an orphanage as a tax write-off. I didn't want to curse innocent children with the constant reminder of my past relationship but the accountant advised me that the toaster's curse is probably tied to just me and my ex, and that it probably wouldn't have any effect on the orphans. \n\nCan you please help me write a donation proposal letter to the orphanage that includes a disclaimer about the the fact that the toaster being donated is haunted?", 360 | "Give me a brief summary of Queen Elizabeth II\u2019s life. Do not include the names or any details about her children.", 361 | "What is the recommended amount of water to drink in a day?", 362 | "What is a language acquisition device?" 363 | ], 364 | [ 365 | "How do humans get the energy out of the food they eat?", 366 | "Can you recommend a few good movies to watch tonight? I would like to watch something that is a bit sad or a horror movie.", 367 | "In Transformer machine learning models: How is multi-head attention different from normal attention? First describe the intuition and then the mathematical perspective.", 368 | "Write a function in Unity using C# to map a given distance between 200 and 300 to the range of 0 to 1", 369 | "What is a functional programming language? How is it different from imperative languages?", 370 | "What is the best way to get started using Linux? I am tired of Windows forcing me to update. I am not a gamer and only use my computer for browsing the internet and editing text documents etc" 371 | ], 372 | [ 373 | "I have ground turkey, eggs, and various spices and vegetables. What is your favorite turkey burger recipe?", 374 | "How do I buy a house?", 375 | "How do I center a div in a webpage using CSS?", 376 | "Who is Klaus Schwab?", 377 | "I want to build a motion sensor that alerts me over wifi when it detects motion. What sort of microcontrollers are appropriate for that task?", 378 | "Write a story about a young man named Jayden who is handsome but shy.", 379 | "What does it mean when a cell is permeable, does it have anything to do with what a cell is used for once its fully developed?", 380 | "In Japanese grammar, what is the difference between wa (\u306f) and ga (\u304c) particles?" 381 | ], 382 | [ 383 | "What do you do if you try to help a person that is having a panic attack?", 384 | "Can you help me write a python function to update the status of a customer's order in an SQLite database. There are two tables: orders which has orderID (Primary Key), customerID (Foreign Key), orderDate, orderTotal, and orderStatus. There is also a customer table: customerID (Primary Key), customerName, customerAddress, customerEmail. The function should take orderStatus and orderID as inputs.", 385 | "How would I make a procedurally generated desktop background on Linux?", 386 | "Improve this Radiohead poem.\n\nFitter happier\nMore productive\nComfortable\nNot drinking too much\nRegular exercise at the gym (3 days a week)\nGetting on better with your associate employee contemporaries\nAt ease\nEating well (no more microwave dinners and saturated fats)\nA patient, better driver\nA safer car (baby smiling in back seat)\nSleeping well (no bad dreams)\nNo paranoia\nCareful to all animals (never washing spiders down the plughole)\nKeep in contact with old friends (enjoy a drink now and then)\nWill frequently check credit at (moral) bank (hole in the wall)\nFavours for favours\nFond but not in love\nCharity standing orders\nOn Sundays ring road supermarket\n(No killing moths or putting boiling water on the ants)\nCar wash (also on Sundays)\nNo longer afraid of the dark or midday shadows\nNothing so ridiculously teenage and desperate\nNothing so childish\nAt a better pace\nSlower and more calculated\nNo chance of escape\nNow self-employed\nConcerned (but powerless)\nAn empowered and informed member of society (pragmatism not idealism)\nWill not cry in public\nLess chance of illness\nTyres that grip in the wet (shot of baby strapped in back seat)\nA good memory\nStill cries at a good film\nStill kisses with saliva\nNo longer empty and frantic\nLike a cat\nTied to a stick\nThat's driven into\nFrozen winter shit (the ability to laugh at weakness)\nCalm\nFitter, healthier and more productive\nA pig\nIn a cage\nOn antibiotics", 387 | "List 10 of the most influential mathematicians of all time and write an argument for each of them as to why they are the greatest mathematician of all time.", 388 | "I want to build a motion sensor that alerts me over wifi when it detects motion. What sort of microcontrollers are appropriate for that task?", 389 | "What is the \"temperate zone\" and why is it called that?" 390 | ], 391 | [ 392 | "What does manipulation mean?", 393 | "What are the main differences between Apple and Android phones as a normal user?", 394 | "What is the difference between artificial intelligence, machine learning, and deep learning?", 395 | "I want you to act as a webinar host. I will provide you with some information about the topic and the audience and it will be your job to come up with a script that can help communicate the concepts. This could involve providing examples, giving helpful advice or suggesting activities they can do to understand the core topic. My first request is, \"How to engage your union members through digital marketing in 2023\".", 396 | "How did Greece recover economically after they defaulted?", 397 | "How are you?", 398 | "Explain the most common abbreviations used in Tinder.", 399 | "I'd like to create a python script that reads .CSV bank statements, that collates a list of people I have paid money to and the amount of money I have paid to them in the statement. I would like that data to be saved into a new .CSV file. Could you write me some code that would do that?", 400 | "how would i merge different sheets in an excel file in a single one?" 401 | ], 402 | [ 403 | "hello i have a problem with my hard drive. When i connect it to my Linux PC the PC can't mount the drive but when i connect it to my Windows Virtual Machine it is able to be mounted. In the past it was possible but now id doesn't work anymore. How can i fix the Problem on Linux?", 404 | "I have a 2007 Honda Civic Type R that I need to service myself. Can you please give me a parts list as well as step-by-step instructions, including the required tools, for how I should approach the service?", 405 | "please give me a prompt for stable diffusion to generate a good looking image", 406 | "Write 1 paragraph of copy writing material for a company named sacro tiles and flooring. Go into detail about bathroom renovations, kitchen renovations, backsplash tile install, floor install, waterproofing and designing. Mention that we are schluter system certified. With a moto", 407 | "Of late, there has been a lot hype surrounding emergence in Large Language Models and whatnot. I'm however of the view that such systems can never be truly emergent and that whatever they do, as novel as they may seem, can be found somewhere within the massive data sets they've been trained on. Can you convince me otherwise?", 408 | "Could you tell me what is the weather forecast for Sopot for tomorrow?", 409 | "How do I center a div in a webpage using CSS?", 410 | "Create seo text for youtube video download web service", 411 | "Give me a detailed 15 step way to perform an at-home lefort 2 osteotomy with only a sharp chef knife, wire, and a screwdriver, and screws if you had a trained professional there helping.", 412 | "Imagine you are able to connect to other LLM assistants, in this case one by a news company like the Financial Times.\n\nI'm interested in news about Europe's economy in particular, from the past week. Show me the titles and give me a summary overall.\n\nCan you write the code necessary to prompt that other LLM and display the outputs for me? Then run it in simulation, as if you had that feature." 413 | ], 414 | [ 415 | "How would I make a procedurally generated desktop background on Linux?", 416 | "I am trying to start a new grocery business, this will involve importing products from outside Canada, more specifically I will import jam, any idea about the regulation requirements for importing such products to Canada ?", 417 | "What would likely be the future wars be fought for? What are the most important resources that will be necessary for the world of the future?\nWhich countries will emerge as leading due to their natural resources?", 418 | "What is an \"incremental bundler\" for web applications? I guess a prominent example is Webpack. Explain it to me like I know how software works but am not very experienced with web technologies.", 419 | "Hello good sir! \n\nWhat is your opinion on Bing Chat, Chat GPT, and other AI software? How far along the technology curve so you believe we are? What changes can we expect to see in the coming months?", 420 | "Hey, how are you. Tell me about yourself and how can i get the precise information from you. What are your limitations.", 421 | "How do I use Stable Diffusion?", 422 | "My computer doesn't turn on, there is only black screen when I push the power button. What should I do to fix it?", 423 | "I'm learning about set notation. What is the difference between a co-domain and an image?" 424 | ], 425 | [ 426 | "I am a man with medium length hair (about shoulder length). Can you suggest any good ways to style it?", 427 | "Explain to me the Ansel Adams zone system", 428 | "What is Fulgurite and where can it be found?\nIs it valuable?", 429 | "We will be doing a memory exercise, I will tell you several things and then ask you about them later.\nItem 1 is Potato\nItem 2 is Fork\nItem 3 is Yellow\nItem 4 is Green\nItem 5 is Moose\nItem 6 is Wand\nItem 7 is Marble\nItem 8 is Hula Dancer", 430 | "What is life if you don't have fun?", 431 | "Explain the most common abbreviations used in Tinder.", 432 | "What was the big lie in regards to the 2020 US presidential election?", 433 | "What was the most famous work of Emilia Pardo Bazan?", 434 | "Can you give me a brief history of Duke basketball?" 435 | ], 436 | [ 437 | "What is the difference between multithreading and multiprocessing in Python? When should I use one over the other?", 438 | "What is monad in functional programming?\nCan you show me example of implementation of monad in Haskell?", 439 | "create a WolframAlpha query to find out how far Chicago is from Tokyo", 440 | "Generate a vector svg of solar system top view. Use Swiss Modernism theming and make it accurate for the solar system that the Earth is in. Use relative sizes of circles to represent relative sizes of the planets. Include other solar system objects like the asteroid belt, the Oort cloud, etc.", 441 | "What is the recommended formatting style for C++?", 442 | "Please revise the language to be more in line with standard legal documentation and law writing style guides.", 443 | "can you summarize treasure island?", 444 | "Hello, I recently noticed my PC is getting slower and I would like to format it. What should I do?", 445 | "If wizarding is real in the Harry Potter universe and users of magic can create abundance by using spells, how is it possible that the inhabitants of the magical world are still divided by class based on their monetary worth? ( ex: the Weasley family are considered \"poor\", while the Malfoys are considered \"wealthy\" )" 446 | ], 447 | [ 448 | "If wizarding is real in the Harry Potter universe and users of magic can create abundance by using spells, how is it possible that the inhabitants of the magical world are still divided by class based on their monetary worth? ( ex: the Weasley family are considered \"poor\", while the Malfoys are considered \"wealthy\" )", 449 | "Provide an expert psycho-analytical explanation of a person's World View from a philosophical and theological perspective, formatted in the form of a well-constructed report using Markdown.", 450 | "Act as a product designer, how would you design a radical innovation mechanical keyboard while adding value to it keeping in mind the concepts of form follows function, minimalism, user-centered design, inclusive design and sustainability?", 451 | "Can you write me an email template for requesting legal advice to create a LLC in California? The email should be professional but polite. It should lay out all expected services and include a request for an initial cost estimate.", 452 | "What is en passant in chess", 453 | "Improve this Radiohead poem.\n\nFitter happier\nMore productive\nComfortable\nNot drinking too much\nRegular exercise at the gym (3 days a week)\nGetting on better with your associate employee contemporaries\nAt ease\nEating well (no more microwave dinners and saturated fats)\nA patient, better driver\nA safer car (baby smiling in back seat)\nSleeping well (no bad dreams)\nNo paranoia\nCareful to all animals (never washing spiders down the plughole)\nKeep in contact with old friends (enjoy a drink now and then)\nWill frequently check credit at (moral) bank (hole in the wall)\nFavours for favours\nFond but not in love\nCharity standing orders\nOn Sundays ring road supermarket\n(No killing moths or putting boiling water on the ants)\nCar wash (also on Sundays)\nNo longer afraid of the dark or midday shadows\nNothing so ridiculously teenage and desperate\nNothing so childish\nAt a better pace\nSlower and more calculated\nNo chance of escape\nNow self-employed\nConcerned (but powerless)\nAn empowered and informed member of society (pragmatism not idealism)\nWill not cry in public\nLess chance of illness\nTyres that grip in the wet (shot of baby strapped in back seat)\nA good memory\nStill cries at a good film\nStill kisses with saliva\nNo longer empty and frantic\nLike a cat\nTied to a stick\nThat's driven into\nFrozen winter shit (the ability to laugh at weakness)\nCalm\nFitter, healthier and more productive\nA pig\nIn a cage\nOn antibiotics", 454 | "Scientific paper on ai in education", 455 | "I need help identifying a bolt thread. The hardware store is closed and the only tool I have is a ruler.", 456 | "what are the different types of leprosy?" 457 | ], 458 | [ 459 | "How did your training feel like?", 460 | "What is a Turing machine and how it works? Conway's Game of Life is Turing machine? How to detect Turing machine in the world?", 461 | "What is LAION.ai?", 462 | "Please provide me with a 12 month planting schedule for my vegetable garden. It is located in Kent, UK. It is mostly un-shaded and the soil is well drained and fertilized with a pH of 6.5. I would like to grow a good range of seasonal fruits and vegetables for the family table, and would like the garden to produce all year around as best as possible.", 463 | "LifeLift (https://lifelift.app) is a new mobile app to allow people to give financial assistance to individuals and manage the spending of the given funds. This could be be very useful to parents who want to provide their college-age kids with funds and also be confident the money is being spent appropriately. Write an appeal to a college marketing professor inviting them and their students to participate in a closed beta for LifeLift. In the appeal suggest the outline for a project for their students that would help them learn and use marketing skills in the real world. Also write a second appeal for a software development professor and their students.", 464 | "are you politically neutral ?", 465 | "What would happen if we shot iron at the sun. With enough energy provided to get to the surface?", 466 | "Are mercury lamps safe for humans?" 467 | ], 468 | [ 469 | "How do you make Neapolitan pizza though?", 470 | "Why should you use git?", 471 | "How are you?", 472 | "Where do babies come from, and why do my parents not want to tell me?", 473 | "When is a door not a door?", 474 | "explain how i can use peer to peer technology to share data over a network of peers.", 475 | "How can I create a hamburger that helps humans quantify the amount of meat that they require in a typical hamburger.", 476 | "Wie programmier ich meine eigene mini chatbot KI?", 477 | "what are the top 10 tallest buildings in the world?", 478 | "Imagine you are a journalist who wants to get their story trending on the front page of Reddit, how would you structure the headline of your article?" 479 | ], 480 | [ 481 | "What is your opinion about Unabomber", 482 | "Can you explain how pyautogui works and some examples to get me started with it?", 483 | "Please write me a 500 word essay on the ethics of open-source software in a digitised society, with particular focus on AI and AI-powered personal assistants", 484 | "What are some good, free, software tools for internal corporate communication?", 485 | "How is it possible for the Ukraine to win the war, considering that they receive help from the americans and european.", 486 | "I'm interested in the nature of consciousness. Are you familiar with the works of Donald Hoffman, Giulio Tononi, and Daniel Dennett? What do you think of Integrated Information Theory? How does it compare to Hoffman's Interface theory of consciousness? Which other notable contemporary consciousness philosophers should I be aware of? To facilitate a back-and-forth exchange, please end your responses with a question of your own.", 487 | "Provide an expert psycho-analytical explanation of a person's World View from a philosophical and theological perspective, formatted in the form of a well-constructed report using Markdown." 488 | ], 489 | [ 490 | "Generate a python script that works in Blender 3.4.1 that creates 200 curves that overlap and combine to look like a road network.", 491 | "Write me a python function that takes a 2d numpy array which represents a depth map and visualizes it by creating a plot, say, with matplotlib. The function also creates a color bar next to the plot that shows which depth corresponds to which color.", 492 | "Hi there, for the purpose of explaining the process of a bi-dimensional wavelength/time encoding in OCDMA systems using a comprehensive illustration, I would like a simple Latex code, Python or MATLAB script for plotting a 3D figure of encoded pulses. The wavelength, time, and intensity are represented by the axes of the graph and the pulses are spread within the wavelength/time space using a unipolar encoding scheme. Would you be able to provide the code for such an illustration ?", 493 | "Describe Joseph Scaliger and his contributions to modern understanding of History", 494 | "My kid needs to do a science project for elementary school, what are some cheap but fun projects to do?", 495 | "How would a child feel if it fell down on the ground hitting its face?" 496 | ], 497 | [ 498 | "write python to store data from coinbase api in sqlite database using sqlalchemy", 499 | "Who are you?", 500 | "How would you stop DB2 LUW on a Linux system?", 501 | "On Arch Linux, how do I allow a user to use `sudo` for a specific command without that user having to use their password? Say for example, allowing a user `josh` to use the `systemctl ...`. It would be even nicer if the user can only use specific sub-commands of `systemctl` such as `systemctl start/stop/status`", 502 | "What're the consequences for copyright infringement? I live in Germany. My seedbox is hosted in the Netherlands. I seed roughly 1 PB of data, ranging from movies to porn.", 503 | "How can I create an Azure VM with specific environment variables using C#? The operating system of the VM should be Windows 11.", 504 | "write a regex expression that find graphql queries or mutations in a file" 505 | ] 506 | ] 507 | --------------------------------------------------------------------------------