├── LICENSE ├── MITRE_EDR_eval1.csv ├── MITRE_EDR_eval_round1.sh ├── README.md └── output.png /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Ziyad Alshehri 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /MITRE_EDR_eval1.csv: -------------------------------------------------------------------------------- 1 | Vendor,None,Telemetry,IOC,GeneralBehavior,SpecificBehavior,Delayed,Tainted,ConfigChange,Enrichment 2 | CarbonBlack,35,213,0,5,29,0,0,0,113 3 | CounterTack,40,152,0,9,11,0,89,0,64 4 | CrowdStrike,31,207,0,144,60,87,79,0,14 5 | Endgame,35,180,0,35,49,49,150,0,59 6 | Microsoft,29,248,1,51,39,33,70,0,10 7 | RSA,62,160,0,0,0,0,0,0,1 8 | SentinelOne,37,207,0,5,0,0,91,0,10 9 | -------------------------------------------------------------------------------- /MITRE_EDR_eval_round1.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Requires jq, and an unbiased mind! 3 | 4 | # 🎨 5 | YELLOW="\033[0;33m" 6 | RED="\033[1;31m" 7 | GREEN="\033[1;32m" 8 | NOCOLOR="\033[0m" 9 | 10 | # Listing our contestants 🏎 11 | declare -a Vendors=("CarbonBlack" "CounterTack" "CrowdStrike" "Endgame" "Microsoft" "RSA" "SentinelOne") 12 | 13 | # Generating a CSV file, save as MITRE_EDR_eval1.csv 14 | echo "Vendor,None,Telemetry,IOC,GeneralBehavior,SpecificBehavior,Delayed,Tainted,ConfigChange,Enrichment" >> MITRE_EDR_eval1.csv 15 | 16 | echo 17 | for Vendor in "${Vendors[@]}" 18 | do 19 | echo -e "${YELLOW}$Vendor${NOCOLOR}" 20 | 21 | # Setting up variables for our metrics 22 | none=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "None" | wc -l | awk '{printf("%s",$1)}' | awk '{printf("%s",$1)}') 23 | tele=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "Telemetry" | wc -l | awk '{printf("%s",$1)}') 24 | ioc=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "Indicator of Compromise" | wc -l | awk '{printf("%s",$1)}') 25 | g_behav=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "General Behavior" | wc -l | awk '{printf("%s",$1)}') 26 | s_behav=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "Specific Behavior" | wc -l | awk '{printf("%s",$1)}') 27 | delayed=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "Delayed" | wc -l | awk '{printf("%s",$1)}') 28 | tainted=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "Tainted" | wc -l | awk '{printf("%s",$1)}') 29 | conf_change=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "Config Change" | wc -l | awk '{printf("%s",$1)}') 30 | enrichment=$(curl -s https://attackevals.mitre.org/$Vendor.1.APT3.1_Results.json | jq . | grep "Enrichment" | wc -l | awk '{printf("%s",$1)}') 31 | 32 | # Adding to the CSV file 33 | echo "$Vendor,`echo "$none"`,`echo "$tele"`,`echo "$ioc"`,`echo "$g_behav"`,`echo "$s_behav"`,`echo "$delayed"`,`echo "$tainted"`,`echo "$conf_change"`,`echo "$enrichment"`" >> MITRE_EDR_eval1.csv 34 | 35 | # Printing the main results, uncomment all lines for the additional ones 36 | # Review (https://attackevals.mitre.org/methodology/detection-categorization.html) 37 | echo -e "\t${RED}None${NOCOLOR} :\t\t\t `echo "$none"`" 38 | echo -e "\t${GREEN}Telemetry${NOCOLOR} :\t\t `echo "$tele"`" 39 | echo -e "\t${GREEN}IOC${NOCOLOR} :\t\t\t `echo "$ioc"`" 40 | echo -e "\t${GREEN}General Behavior${NOCOLOR} :\t `echo "$g_behav"`" 41 | echo -e "\t${GREEN}Specific Behavior${NOCOLOR} :\t `echo "$s_behav"`" 42 | # echo -e "\t${RED}Delayed${NOCOLOR} :\t\t `echo "$delayed"`" 43 | # echo -e "\t${RED}Tainted${NOCOLOR} :\t\t `echo "$tainted"`" 44 | # echo -e "\t${RED}Config Change${NOCOLOR} :\t\t `echo "$conf_change"`" 45 | echo -e "\t${GREEN}Enrichment${NOCOLOR} :\t\t `echo "$enrichment"`\n" 46 | echo "==========================================" 47 | echo 48 | 49 | done 50 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## MITRE EDR Evaluation Reporter 2 | 3 | MITRE recently published the results of their EDR evaluation (the very first round), which is purely based on the Adversary Emulation Approach, and the [ATT&CK](https://attack.mitre.org/) matrix. 4 | 5 | > MITRE ATT&CK™ is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community. 6 | 7 | ## Objectives 8 | 9 | MITRE posted the results in great detail [here](https://attackevals.mitre.org), however, parsing the different JSON report files is no fun. This script pulls the detection rate of the attack techniques on each product, and genearte a text & csv outputs. 10 | 11 |

12 | 13 |

14 | 15 | ## MITRE Testing Methodology 16 | 17 | MITRE Testing Methodology was described in [this document](https://attackevals.mitre.org/methodology/) with all the deatials necessary to understand how the test lab was setup. 18 | 19 | ### Detection Categorization 20 | 21 | **None**: The vendor is unable to detect red activity due to capability limitations or other reasons. If data is available that is not directly relevant to the procedure tested, this will be categorized as “None.” In these cases, a vendor may receive a categorization of “None” with additional notes and screenshots about that data. 22 | 23 | **Telemetry**: The capability produces some minimally processed data that is accessible to an end user and directly indicates that the red team activity occurred after the user performs human analysis. There is no evidence of complex logic or an advanced rule leading to the data output, and no labeling occurs other than simple field labeling. The detection needs to be demonstrably and logically related to the actual procedure performed. Proof of detection could include the view, query, or API search used to access the data and/or the detection output (e.g., table view or process tree). 24 | 25 | **Indicator Of Compromise (IOC)**: The vendor identifies the red team activity based on known hashes, IP addresses, C2 domain, tool names, tool strings, or module names. Proof of detection could include the rule name, API/query used to access the data, and/or detection output 26 | 27 | **Enrichment**: The capability captures data (usually data as described above in the “Telemetry Available” category) and then enriches it with additional information such as a rule name, labels, tags, or ATT&CK tactics or techniques that would assist in a user’s analysis of the data beyond what would have been originally presented. 28 | 29 | **General Behavior**: The capability produces an alert detection for suspicious or potentially malicious behavior based on some type of reported complex logic or rule (beyond a simple “if X, display Y Rule Name,” which would be categorized as Enrichment). 30 | 31 | **Specific Behavior**: The capability detects suspicious behavior based on some complex rule or logic and provides an ATT&CK “technique”-level description of the activity (beyond a simple “if X, display Y Rule Name,” which would be categorized as Enrichment). 32 | 33 | 34 | ### Modifier Detection Types 35 | 36 | **Delayed**: The capability does not detect the activity in real-time or near-real-time when the red team executes the action, but subsequent alerts, data, enrichment, or additional processing produce a detection for the activity. 37 | 38 | **Tainted**: The capability detects the activity based on previously identified suspicious/malicious behavior that is related to or “tainted by” the detection. 39 | 40 | **Configuration Change**: A detection is made possible by a special configuration change or additional API access that allows data not normally accessible to the end user to become available. 41 | 42 | ## Prerequisites 43 | 44 | [JQ](https://stedolan.github.io/jq/), and an unbiased mind! 45 | 46 | ## Running the code 47 | 48 | ``` 49 | bash MITRE_EDR_eval_round1.sh 50 | ``` 51 | 52 | ## Important Notes 53 | 54 | 1- This is the first EDR evaluation against MITRE ATT&CK, specifically against [APT3](https://attack.mitre.org/groups/G0022/) threat group attack techniques. More tests will come later, and probably against more vendors as well. This repo will be updated once these reports become avaliable. 55 | 56 | 2- You have to read the entire page here [MITRE Evaluation](https://attackevals.mitre.org/evaluations.html), the high level analysis of the final metrics can significantly impact your analysis. The evaluation page has screenshots for all the products, and the exact findings to determine if that's something you'd be concerned about in your environment (YMMV). 57 | 58 | 3- I'm not affiliated with MITRE or any EDR vendor, and take no responsibility or liability of the results/reports. 59 | -------------------------------------------------------------------------------- /output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zshehri/MITRE_EDR_Eval/42029543ea006e6d4c8e70ebe05e20031d7f6f63/output.png --------------------------------------------------------------------------------