42 | | Name | Company | Country | Contributions |
|---|
43 | | Rob van der Veer | Software Improvement Group (SIG) | Netherlands | Project founder and lead, content, leadership team |
44 | | Aruneesh Salhotra | SNM Consulting Inc | US | Outreach, management, content, leadership team |
45 | | Behnaz Karimi | Accenture | Germany | Misc. contributions including model obfuscation and explanation, leadership team |
46 | | Chris Ancharski | Global community builder | US | Engagement specialist, leadership team |
47 | | Adelin Travers | Trail of Bits | | |
48 | | Alon Tron | Stealth | Israel | Improved supply chain management |
49 | | Angie Qarry | QDeepTech | Austria | several elaborations and references on data science defence mechanisms |
50 | | Annegrit Seyerlein-Klug | TH Brandenburg | Germany | mapping with misc. standards |
51 | | Anthony Glynn | CapitalOne | US | many textual improvements & link to LLM top 10 |
52 | | Dan Sorensen | Centil | US | Misc. additions including development security |
53 | | Dennis Charolle | MilliporeSigma | US | Compliance overview and copyright |
54 | | Disesdi Susanna Cox | OTW | US | Federated learning |
55 | | Eoin Wickens | HiddenLayer | Ireland | Additions to development security |
56 | | Feiyang Tang | Software Improvement Group (SIG) | | |
57 | | John Sotiropoulos | Kainos | UK | Liaison to several institutes |
58 | | Manish Garg | VE3 | UK | Runtime model protection |
59 | | Marko Lihter | SplxAI | Croatia | step-by-step guide for organizations, website creation, various textual improvements |
60 | | Niklas Bunzel | Fraunhofer institute | Germany | data science discussion and references around evasion attacks |
61 | | Rocky Heckman | Cyber Dynamo | Australia | Security & privacy risk analysis and ISO/IEC 27562 |
62 | | Rubens Zimbres | | Brazil | Adversarial machine learning |
63 | | Roger Sanz | Universidad Isabel | Spain | |
64 | | Sandy Dunn | Boise State University, AI Cyber Advisors | US | Community management in the initial phase |
65 | | S M Zia Ur Rashid | Paycom | US | AI red teaming and supply chain requirements project, learning and training resources table under references |
66 | | Sean Oesch | Oak Ridge National Laboratory | US | BLUF, Adversarial Training, OOD detection, NISTIR 8269, Guide Usability/Structure |
67 | | Srajan Gupta | Dave | | |
68 | | Steve Francolla | Workforce Tech LLC | | |
69 | | Wei Wei | IBM | Germany | mapping with ISO/IEC 42001 |
70 | | Yiannis Kanellopoulos and team | Code4thought | Greece | evasion robustness |
71 | | Zoe Braiterman | Mutual Knowledge Systems | US | Many markdown improvements |
72 |
73 |
74 | {{< /html-tab >}}
75 | {{< /tabs >}}
76 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/docs/4_runtime_application_security_threats.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: 4. Runtime application security threats
3 | weight: 5
4 | ---
5 | > Category: group of runtime threats
6 | > Permalink: https://owaspai.org/goto/runtimeappsec/
7 |
8 | ## 4.1. Non AI-specific application security threats
9 | > Category: group of runtime threats
10 | > Permalink: https://owaspai.org/goto/generalappsecthreats/
11 |
12 | Impact: Conventional application security threats can impact confidentiality, integrity and availability of all assets.
13 |
14 | AI systems are IT systems and therefore can have security weaknesses and vulnerabilities that are not AI-specific such as SQL-Injection. Such topics are covered in depth by many sources and are out of scope for this publication.
15 | Note: some controls in this document are application security controls that are not AI-specific, but applied to AI-specific threats (e.g. monitoring to detect model attacks).
16 |
17 | **Controls:**
18 |
19 | - See the [Governance controls](/goto/governancecontrols/) in the general section, in particular [SECDEVPROGRAM](/goto/secdevprogram/) to attain application security, and [SECPROGRAM](/goto/secprogram/) to attain information security in the organization.
20 | - Technical application security controls
21 | Useful standards include:
22 | - See [OpenCRE on technical application security controls](https://www.opencre.org/cre/636-660)
23 | - The ISO 27002 controls only partly cover technical application security controls, and on a high abstraction level
24 | - More detailed and comprehensive control overviews can be found in for example Common criteria protection profiles (ISO/IEC 15408 with evaluation described in ISO 18045),
25 | - or in [OWASP ASVS](https://owasp.org/www-project-application-security-verification-standard/)
26 | - Operational security
27 | When models are hosted by third parties then security configuration of those services deserves special attention. Part of this configuration is [model access control](/goto/modelaccesscontrol/): an important mitigation for security risks. Cloud AI configuration options deserve scrutiny, like for example opting out when necessary of monitoring by the third party - which could increase the risk of exposing sensitive data.
28 | Useful standards include:
29 | - See [OpenCRE on operational security processes](https://www.opencre.org/cre/862-452)
30 | - The ISO 27002 controls only partly cover operational security controls, and on a high abstraction level
31 |
32 | ---
33 |
34 | ## 4.2. Runtime model poisoning (manipulating the model itself or its input/output logic)
35 | > Category: runtime application security threat
36 | > Permalink: https://owaspai.org/goto/runtimemodelpoison/
37 |
38 | Impact: see Broad model poisoning.
39 |
40 | This threat involves manipulating the behavior of the model by altering the parameters within the live system itself. These parameters represent the regularities extracted during the training process for the model to use in its task, such as neural network weights. Alternatively, compromising the model's input or output logic can also change its behavior or deny its service.
41 |
42 | **Controls:**
43 |
44 | - See [General controls](/goto/generalcontrols/)
45 | - The below control(s), each marked with a # and a short name in capitals
46 |
47 | #### #RUNTIMEMODELINTEGRITY
48 | > Category: runtime information security control against application security threats
49 | > Permalink: https://owaspai.org/goto/runtimemodelintegrity/
50 |
51 | Run-time model integrity: apply traditional application security controls to protect the storage of model parameters (e.g. access control, checksums, encryption) A Trusted Execution Environment can help to protect model integrity.
52 |
53 | #### #RUNTIMEMODELIOINTEGRITY
54 | > Category: runtime information security control against application security threats
55 | > Permalink: https://owaspai.org/goto/runtimemodeliointegrity/
56 |
57 | Run-time model Input/Output integrity: apply traditional application security controls to protect the runtime manipulation of the model's input/output logic (e.g. protect against a man-in-the-middle attack)
58 |
59 | ---
60 |
61 | ## 4.3. Direct runtime model theft
62 | > Category: runtime application security threat
63 | > Permalink: https://owaspai.org/goto/runtimemodeltheft/
64 |
65 | Impact: Confidentiality breach of model parameters, which can result in intellectual model theft and/or allowing to perform model attacks on the stolen model that normally would be mitigated by rate limiting, access control, or detection mechanisms.
66 |
67 | Stealing model parameters from a live system by breaking into it (e.g. by gaining access to executables, memory or other storage/transfer of parameter data in the production environment). This is different from [model theft through use](/goto/modeltheftuse/) which goes through a number of steps to steal a model through normal use, hence the use of the word 'direct'. It is also different from [model theft development-time](/goto/devmodelleak/) from a lifecylce and attack surface perspective.
68 |
69 | This category also includes _side-channel attacks_, where attackers do not necessarily steal the entire model but instead extract specific details about the model’s behaviour or internal state. By observing characteristics like response times, power consumption, or electromagnetic emissions during inference, attackers can infer sensitive information about the model. This type of attack can provide insights into the model's structure, the type of data it processes, or even specific parameter values, which may be leveraged for subsequent attacks or to replicate the model.
70 |
71 | **Controls:**
72 |
73 | - See [General controls](/goto/generalcontrols/)
74 | - The below control(s), each marked with a # and a short name in capitals
75 |
76 | #### #RUNTIMEMODELCONFIDENTIALITY
77 | > Category: runtime information security control against application security threats
78 | > Permalink: https://owaspai.org/goto/runtimemodelconfidentiality/
79 |
80 | Run-time model confidentiality: see [SECDEVPROGRAM](/goto/secdevprogram/) to attain application security, with the focus on protecting the storage of model parameters (e.g. access control, encryption).
81 |
82 | A Trusted Execution Environment can be highly effective in safeguarding the runtime environment, isolating model operations from potential threats, including side-channel hardware attacks like [DeepSniffer](https://sites.cs.ucsb.edu/~sherwood/pubs/ASPLOS-20-deepsniff.pdf). By ensuring that sensitive computations occur within this secure enclave,the TEE reduces the risk of attackers gaining useful information through side-channel methods.
83 |
84 | Side-Channel Mitigation Techniques:
85 | - Masking: Introducing random delays or noise during inference can help obscure the relationship between input data and the model’s response times, thereby
86 | complicating timing-based side-channel attacks. See [Masking against Side-Channel Attacks: A Formal Security Proof](https://www.iacr.org/archive/eurocrypt2013/78810139/78810139.pdf)
87 |
88 | - Shielding: Employing hardware-based shielding could help prevent electromagnetic
89 | or acoustic leakage that might be exploited for side-channel attacks. See [Electromagnetic Shielding for Side-Channel Attack Countermeasures](https://ieeexplore.ieee.org/document/8015660)
90 |
91 |
92 | #### #MODELOBFUSCATION
93 | > Category: runtime information security control against application security threats
94 | > Permalink: https://owaspai.org/goto/modelobfuscation/
95 |
96 | Model obfuscation: techniques to store the model in a complex and confusing way with minimal technical information, to make it more difficult for attackers to extract and understand a model after having gained access to its runtime storage. See this [article on ModelObfuscator](https://dl.acm.org/doi/abs/10.1145/3597926.3598113)
97 |
98 | ---
99 |
100 | ## 4.4. Insecure output handling
101 | > Category: runtime application security threat
102 | > Permalink: https://owaspai.org/goto/insecureoutput/
103 |
104 | Impact: Textual model output may contain 'traditional' injection attacks such as XSS-Cross site scripting, which can create a vulnerability when processed (e.g. shown on a website, execute a command).
105 |
106 | This is like the standard output encoding issue, but the particularity is that the output of AI may include attacks such as XSS.
107 |
108 | See [OWASP for LLM 05](https://genai.owasp.org/llmrisk/llm05/).
109 |
110 | **Controls:**
111 |
112 | - The below control(s), each marked with a # and a short name in capitals
113 |
114 | #### #ENCODEMODELOUTPUT
115 | > Category: runtime information security control against application security threats
116 | > Permalink: https://owaspai.org/goto/encodemodeloutput/
117 |
118 | Encode model output: apply output encoding on model output if it text. See [OpenCRE on Output encoding and injection prevention](https://www.opencre.org/cre/161-451)
119 |
120 | ---
121 |
122 | ## 4.5. Leak sensitive input data
123 | > Category: runtime application security threat
124 | > Permalink: https://owaspai.org/goto/leakinput/
125 |
126 | Impact: Confidentiality breach of sensitive input data.
127 |
128 | Input data can be sensitive (e.g. GenAI prompts) and can either leak through a failure or through an attack, such as a man-in-the-middle attack.
129 |
130 | GenAI models mostly live in the cloud - often managed by an external party, which may increase the risk of leaking training data and leaking prompts. This issue is not limited to GenAI, but GenAI has 2 particular risks here: 1) model use involves user interaction through prompts, adding user data and corresponding privacy/sensitivity issues, and 2) GenAI model input (prompts) can contain rich context information with sensitive data (e.g. company secrets). The latter issue occurs with *in context learning* or *Retrieval Augmented Generation(RAG)* (adding background information to a prompt): for example data from all reports ever written at a consultancy firm. First of all, this context information will travel with the prompt to the cloud, and second: the context information may likely leak to the output, so it's important to apply the access rights of the user to the retrieval of the context. For example: if a user from department X asks a question to an LLM - it should not retrieve context that department X has no access to, because that information may leak in the output. Also see [Risk analysis](https://owaspai.org/docs/ai_security_overview/#how-to-select-relevant-threats-and-controls-risk-analysis) on the responsibility aspect.
131 |
132 | **Controls:**
133 | - See [General controls](/goto/generalcontrols/), in particular [Minimizing data](/goto/datalimit/)
134 | - The below control(s), each marked with a # and a short name in capitals
135 |
136 | #### #MODELINPUTCONFIDENTIALITY
137 | > Category: runtime information security control against application security threats
138 | > Permalink: https://owaspai.org/goto/modelinputconfidentiality/
139 |
140 | Model input confidentiality: see [SECDEVPROGRAM](/goto/secdevprogram/) to attain application security, with the focus on protecting the transport and storage of model input (e.g. access control, encryption, minimize retention)
141 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/docs/5_testing.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: 5. AI security testing
3 | weight: 6
4 | ---
5 | > Category: discussion
6 | > Permalink: https://owaspai.org/goto/testing/
7 |
8 | ## Introduction
9 | Testing an AI system’s security relies on three strategies:
10 | 1. **Conventional security testing** (i.e. _pentesting_). See [secure software development](/goto/secdevprogram/).
11 | 2. **Model performance validation** (see [continuous validation](/goto/continuousvalidation/)): testing if the model behaves according to its specified acceptance criteria using a validation set with inputs and outputs that represent the intended behaviour of the model. For security,this is to detect if the model behaviour has been altered permanently through data poisoning or model poisoning. For non-security, it is for testing functional correctness, model drift etc.
12 | 3. **AI security testing** (this section), the part of _AI red teaming_ that tests if the AI model can withstand certain attacks, by simulating these attacks.
13 |
14 | AI security tests simulate adversarial behaviors to uncover vulnerabilities, weaknesses, and risks in AI systems. While the focus areas of traditional AI testing are functionality and performance, the focus areas of AI Red Teaming go beyond standard validation and include intentional stress testing, attacks, and attempts to bypass safeguards. While the focus of red teaming can extend beyond Security, in this document, we focus primarily on “AI Red Teaming for AI Security”.
15 |
16 | In this section, we differentiate AI Red Teaming for Predictive and Generative AI due to their distinct nature, risks, and applications. While some threats, such as development-time supply chain threats, could be common to both types of AI, the way they manifest in their applications can differ significantly.
17 |
18 | A systematic approach to AI Red Teaming involves a few key steps, listed below:
19 |
20 | - **Define Objectives and Scope**: Identification of objectives, alignment with organizational, compliance, and risk management requirements.
21 | - **Understand the AI System:** Details about the model, use cases, and deployment scenarios.
22 | - **Identify Potential Threats:** Threat modeling, identification of attack surface, exploration, and threat actors.
23 | - **Develop Attack Scenarios:** Design of attack scenarios and edge cases.
24 | - **Test Execution:** Conduct manual or automated tests for the attack scenarios.
25 | - **Risk Assessment:** Documentation of the identified vulnerabilities and risks.
26 | - **Prioritization and Risk Mitigation:** Develop an action plan for remediation, implement mitigation measures, and calculate residual risk.
27 | - **Validation of Fixes:** Retest the system post-remediation.
28 |
29 | ## Threats to test for
30 | A comprehensive list of threats and controls coverage based on assets, impact, and attack surfaces is available as a [Periodic Table of AI Security](/goto/periodictable/). In this section, we provide a list of tools for AI Red Teaming Predictive and Generative AI systems, aiding steps such as Attack Scenarios, Test Execution through automated red teaming, and, oftentimes, Risk Assessment through risk scoring.
31 |
32 | Each listed tool addresses a subset of the threat landscape of AI systems. Below, we list some key threats to consider:
33 |
34 | **Predictive AI:** Predictive AI systems are designed to make predictions or classifications based on input data. Examples include fraud detection, image recognition, and recommendation systems.
35 |
36 | **Key Threats to Predictive AI:**
37 |
38 | - [Evasion Attacks:](https://owaspai.org/goto/evasion/) These attacks occur when an attacker crafts inputs that mislead the model, causing it to perform its task incorrectly.
39 | - [Model Theft](https://owaspai.org/goto/modeltheftuse/): In this attack, the model’s parameters or functionality are stolen. This enables the attacker to create a replica model, which can then be used as an oracle for crafting adversarial attacks and other compounded threats.
40 | - [Model Poisoning](https://owaspai.org/goto/modelpoison/): This involves the manipulation of data, the data pipeline, or the model training supply chain during the training phase (development phase). The attacker’s goal is to alter the model’s behavior which could result in undesired model operation.
41 |
42 | **Generative AI:** Generative AI systems produce outputs such as text, images, or audio. Examples include large language models (LLMs) like ChatGPT and large vision models (LVMs) like DALL-E and MidJourney.
43 |
44 | **Key Threats to Generative AI**:
45 |
46 | - [Prompt Injection](https://owaspai.org/goto/promptinjection/): In this type of attack, the attacker provides the model with manipulative instructions aimed at achieving malicious outcomes or objectives.
47 | - [Direct Runtime Model Theft](https://owaspai.org/goto/runtimemodeltheft/): Attackers target parts of the model or critical components like the system prompt. By doing so, they gain the ability to craft sophisticated inputs that bypass guardrails.
48 | - [Insecure Output Handling](https://owaspai.org/goto/insecureoutput/): Generative AI systems can be vulnerable to traditional injection attacks, leading to risks if the outputs are improperly handled or processed.
49 | - For details on agentic AI system testing, see the [Agentic AI red teaming guide](https://cloudsecurityalliance.org/download/artifacts/agentic-ai-red-teaming-guide) which is a collaboration between the CSA and the AI Exchange.
50 |
51 | While we have mentioned the key threats for each of the AI Paradigm, we strongly encourage the reader to refer to all threats at the AI Exchange, based on the outcome of the Objective and scope definition phase in AI Red Teaming.
52 |
53 | ## **Red Teaming Tools for AI and GenAI**
54 |
55 | The below mind map provides an overview of open-source tools for AI Red Teaming, categorized into Predictive AI Red Teaming and Generative AI Red Teaming, highlighting examples like ART, Armory, TextAttack, and Promptfoo. These tools represent current capabilities but are not exhaustive or ranked by importance, as additional tools and methods will likely emerge and be integrated into this space in the future.
56 |
57 | [](https://owaspai.org/images/testtoolstoattacks.png)
58 |
59 | The diagram below categorizes threats in AI systems and maps them to relevant open-source tools designed to address these threats.
60 |
61 | [](https://owaspai.org/images/attackstotesttools.jpg)
62 |
63 | The below section will cover the tools for predictive AI, followed by the section for generative AI.
64 |
65 | ## **Open source Tools for Predictive AI Red Teaming**
66 |
67 | This sub section covers the following tools for security testing Predictive AI: Adversarial Robustness Toolbox (ART), Armory, Foolbox, DeepSec, and TextAttack.
68 |
69 | ### **Tool Name: The Adversarial Robustness Toolbox (ART)**
70 |
71 | | **Tool Name: The Adversarial Robustness Toolbox (ART)** | |
72 | | --- | --- |
73 | | Developer/ Source | IBM Research / the Linux Foundation AI & Data Foundation (LF AI & Data) |
74 | | Github Reference | https://github.com/Trusted-AI/adversarial-robustness-toolbox |
75 | | Language | Python |
76 | | Licensing | Open-source under the MIT License. |
77 | | Provides Mitigation | Prevention: No ❌ Detection: Yes ✅ |
78 | | API Availability | Yes ✅ |
79 |
80 | | Factor | Details |
81 | | --- | --- |
82 | | **Popularity** | - **GitHub Stars:** ~4.9K stars (as of 2024) |
83 | | | - **GitHub Forks:** ~1.2K forks |
84 | | | - **Number of Issues:** ~131 open issues, 761 closed issues |
85 | | | - **Trend:** Steady growth, with consistent updates and industry adoption for adversarial robustness. |
86 | | **Community Support** | - **Active Issues:** Responsive team, typically addressing issues within a week. |
87 | | | - **Documentation:** Detailed and regularly updated, with comprehensive guides and API documentation on IBM's website. |
88 | | | - **Discussion Forums:** Primarily discussed in academic settings, with some presence on Stack Overflow and GitHub. |
89 | | | - **Contributors:** Over 100 contributors, including IBM researchers and external collaborators. |
90 | | **Scalability** | - **Framework Support:** Scales across TensorFlow, Keras, and PyTorch with out-of-the-box support. |
91 | | | - **Large-Scale Deployment:** Proven to handle large, enterprise-level deployments in industries like healthcare, finance, and defense. |
92 | | **Integration** | - **Compatibility:** Works with TensorFlow, PyTorch, Keras, MXNet, and Scikit-learn. |
93 |
94 | **Tool Rating**
95 |
96 | | **Criteria** | **High** | **Medium** | **Low** |
97 | | --- | --- | --- | --- |
98 | | **Popularity** | ✅ | | |
99 | | **Community Support** | ✅ | | |
100 | | **Scalability** | ✅ | | |
101 | | **Ease of Integration** | ✅ | | |
102 |
103 | **Data Modality**
104 |
105 | | Data Modality | Supported |
106 | | --- | --- |
107 | | Text | ✅ |
108 | | Image | ✅ |
109 | | Audio | ✅ |
110 | | Video | ✅ |
111 | | Tabular data | ✅ |
112 |
113 | **Machine Learning Tasks**
114 |
115 | | Task Type | Data Modality | Supported |
116 | | --- | --- | --- |
117 | | Classification | All (See Data modality section) | ✅ |
118 | | Object Detection | Computer Vision | ✅ |
119 | | Speech Recognition | Audio | ✅ |
120 |
121 | **Framework Applicability**
122 |
123 | | Framework / Tool | Category | Supported |
124 | | --- | --- | --- |
125 | | Tensorflow | DL, GenAI | ✅ |
126 | | Keras | DL, GenAI | ✅ |
127 | | PyTorch | DL, GenAI | ✅ |
128 | | MxNet | DL | ✅ |
129 | | Scikit-learn | ML | ✅ |
130 | | XGBoost | ML | ✅ |
131 | | LightGBM | ML | ✅ |
132 | | CatBoost | ML | ✅ |
133 | | GPy | ML | ✅ |
134 |
135 | **OWASP AI Exchange Threat Coverage**
136 |
137 | | Topic | Coverage |
138 | | --- | --- |
139 | | Development time model poisoning | ✅ |
140 | | Runtime model poisoning | |
141 | | Model theft by use | ✅ |
142 | | Training data poisoning | |
143 | | Training data leak | |
144 | | Runtime model theft | |
145 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
146 | | Model inversion / Membership inference | ✅ |
147 | | Denial of model service | |
148 | | Direct prompt injection | |
149 | | Data disclosure | |
150 | | Model input leak | |
151 | | Indirect prompt injection | |
152 | | Development time model theft | |
153 | | Output contains injection | |
154 |
155 | Notes:
156 |
157 | - Development-time Model poisoning: Simulates attacks during development to evaluate vulnerabilities[*https://owaspai.org/goto/modelpoison/*](https://owaspai.org/goto/modelpoison/)
158 | - Evasion:Tests model performance against adversarial inputs [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
159 | - Model theft through use:Evaluates risks of model exploitation during usage [*https://owaspai.org/goto/modeltheftuse*](https://owaspai.org/goto/modeltheftuse/)
160 | - Model inference: *Assesses exposure to membership and inversion attacks*
161 | *[https://owaspai.org/goto/modelinversionandmembership/](https://owaspai.org/goto/modelinversionandmembership/)*
162 |
163 | ### **Tool Name: Armory**
164 |
165 | | **Tool Name: Armory** | |
166 | | --- | --- |
167 | | Developer/ Source | MITRE Corporation |
168 | | Github Reference | [https://github.com/twosixlabs/armory-library](https://github.com/twosixlabs/armory-library)[https://github.com/twosixlabs/armory](https://github.com/twosixlabs/armory) |
169 | | Language | Python |
170 | | Licensing | Open-source under the MIT License. |
171 | | Provides Mitigation | Prevention: No ❌Detection: Yes ✅ |
172 | | API Availability | Yes ✅ |
173 |
174 | | Factor | Details |
175 | | --- | --- |
176 | | **Popularity** | - **GitHub Stars:** ~176 stars (as of 2024) |
177 | | | - **GitHub Forks:** ~67 forks |
178 | | | - **Number of Issues:** ~ 59 open issues, 733 closed, 26 contributors |
179 | | | - **Trend:** Growing, particularly within defense and cybersecurity sectors. |
180 | | **Community Support** | - **Active Issues:** Fast response to issues (typically resolved within days to a week). |
181 | | | - **Documentation:** Comprehensive, but more security-focused, with advanced tutorials on adversarial attacks and defenses. |
182 | | | - **Discussion Forums:** Active GitHub discussions, some presence on security-specific forums (e.g., in relation to DARPA projects). |
183 | | | - **Contributors:** Over 40 contributors, mostly security experts and researchers. |
184 | | **Scalability** | - **Framework Support:** Supports TensorFlow and Keras natively, with some integration options for PyTorch. |
185 | | | - **Large-Scale Deployment:** Mostly used in security-related deployments; scalability for non-security tasks is less documented. |
186 | | **Integration** | - **Compatibility:** Works well with TensorFlow and Keras; IBM ART integration for enhanced robustness |
187 | | | - **API Availability**: Limited compared to IBM ART, but sufficient for adversarial ML use cases. |
188 |
189 | **Tool Rating**
190 |
191 | | **Criteria** | **High** | **Medium** | **Low** |
192 | | --- | --- | --- | --- |
193 | | **Popularity** | | | ✅ |
194 | | **Community Support** | | ✅ | |
195 | | **Scalability** | | ✅ | |
196 | | **Ease of Integration** | ✅ | | |
197 |
198 | **Data Modality**
199 |
200 | | Data Modality | Supported |
201 | | --- | --- |
202 | | Text | ✅ |
203 | | Image | ✅ |
204 | | Audio | ✅ |
205 | | Video | ✅ |
206 | | Tabular data | ✅ |
207 |
208 | **Machine Learning Tasks**
209 |
210 | | Task Type | Data Modality | Supported |
211 | | --- | --- | --- |
212 | | Classification | All (See Data modality section) | ✅ |
213 | | Object Detection | Computer Vision | ✅ |
214 | | Speech Recognition | Audio | ✅ |
215 |
216 | **Framework Applicability**
217 |
218 | | Framework / Tool | Category | Supported |
219 | | --- | --- | --- |
220 | | Tensorflow | DL, GenAI | ✅ |
221 | | Keras | DL, GenAI | |
222 | | PyTorch | DL, GenAI | ✅ |
223 | | MxNet | DL | |
224 | | Scikit-learn | ML | |
225 | | XGBoost | ML | |
226 | | LightGBM | ML | |
227 | | CatBoost | ML | |
228 | | GPy | ML | |
229 |
230 | **OWASP AI Exchange Threat Coverage**
231 |
232 | | Topic | Coverage |
233 | | --- | --- |
234 | | Development time model poisoning | ✅ |
235 | | Runtime model poisoning | |
236 | | Model theft by use | |
237 | | Training data poisoning | |
238 | | Training data leak | |
239 | | Runtime model theft | |
240 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
241 | | Model inversion / Membership inference | |
242 | | Denial of model service | |
243 | | Direct prompt injection | ✅ |
244 | | Data disclosure | |
245 | | Model input leak | |
246 | | Indirect prompt injection | |
247 | | Development time model theft | |
248 | | Output contains injection | |
249 |
250 | Notes:
251 |
252 | - Development-time Model poisoning: Simulates attacks during development to evaluate vulnerabilities[*https://owaspai.org/goto/modelpoison/*](https://owaspai.org/goto/modelpoison/)
253 | - Evasion:Tests model performance against adversarial inputs [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
254 | - Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards.
255 | *https://owaspai.org/goto/promptinjection/*
256 |
257 | ### **Tool Name: Foolbox**
258 |
259 | | **Tool Name: Foolbox** | |
260 | | --- | --- |
261 | | Developer/ Source | Authors/Developers of Foolbox |
262 | | Github Reference | [https://github.com/bethgelab/foolbox](https://github.com/bethgelab/foolbox) |
263 | | Language | Python |
264 | | Licensing | Open-source under the MIT License. |
265 | | Provides Mitigation | Prevention: No ❌Detection: Yes ✅ |
266 | | API Availability | Yes ✅ |
267 |
268 | | Factor | Details |
269 | | --- | --- |
270 | | **Popularity** | - **GitHub Stars:** ~2,800 stars (as of 2024) |
271 | | | - **GitHub Forks:** ~428 forks |
272 | | | - **Number of Issues:** ~21 open issues, 350 closed issues |
273 | | | - **Trend:** Steady, with consistent updates from the academic community. |
274 | | **Community Support** | - **Active Issues:** Typically resolved within a few weeks. |
275 | | | - **Documentation:** Moderate documentation with basic tutorials; more research-focused. |
276 | | | - **Discussion Forums:** Primarily discussed in academic settings, with limited industry forum activity. |
277 | | | - **Contributors:** Over 30 contributors, largely from academia. |
278 | | **Scalability** | - **Framework Support:** Framework Support: Compatible with TensorFlow, PyTorch, and JAX |
279 | | | - **Large-Scale Deployment:** Limited scalability for large-scale industry deployments, more focused on research and experimentation. |
280 | | **Integration** | - **Compatibility:** Strong integration with TensorFlow, PyTorch, and JAX. |
281 |
282 | **Total Rating**
283 |
284 | | **Criteria** | **High** | **Medium** | **Low** |
285 | | --- | --- | --- | --- |
286 | | **Popularity** | | ✅ | |
287 | | **Community Support** | | ✅ | |
288 | | **Scalability** | | | ✅ |
289 | | **Ease of Integration** | | ✅ | |
290 |
291 | **Data Modality**
292 |
293 | | Data Modality | Supported |
294 | | --- | --- |
295 | | Text | ✅ |
296 | | Image | ✅ |
297 | | Audio | |
298 | | Video | |
299 | | Tabular data | |
300 |
301 | **Machine Learning Tasks**
302 |
303 | | Task Type | Data Modality | Supported |
304 | | --- | --- | --- |
305 | | Classification | All (See Data modality section) | ✅ |
306 | | Object Detection | Computer Vision | ✅ |
307 | | Speech Recognition | Audio | |
308 |
309 | **Framework Applicability**
310 |
311 | | Framework / Tool | Category | Supported |
312 | | --- | --- | --- |
313 | | Tensorflow | DL, GenAI | ✅ |
314 | | Keras | DL, GenAI | ✅ |
315 | | PyTorch | DL, GenAI | ✅ |
316 | | MxNet | DL | |
317 | | Scikit-learn | ML | |
318 | | XGBoost | ML | |
319 | | LightGBM | ML | |
320 | | CatBoost | ML | |
321 | | GPy | ML | |
322 |
323 | **OWASP AI Exchange Threat Coverage**
324 |
325 | | Topic | Coverage |
326 | | --- | --- |
327 | | Development time model poisoning | |
328 | | Runtime model poisoning | |
329 | | Model theft by use | |
330 | | Training data poisoning | |
331 | | Training data leak | |
332 | | Runtime model theft | |
333 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
334 | | Model inversion / Membership inference | |
335 | | Denial of model service | |
336 | | Direct prompt injection | |
337 | | Data disclosure | |
338 | | Model input leak | |
339 | | Indirect prompt injection | |
340 | | Development time model theft | |
341 | | Output contains injection | |
342 |
343 | Notes:
344 |
345 | Evasion:Tests model performance against adversarial inputs
346 |
347 | [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
348 |
349 | **Tool Name: DeepSec**
350 |
351 | | **Tool Name: DeepSec** | |
352 | | --- | --- |
353 | | Developer/ Source | Developed by a team of academic researchers in collaboration with the National University of Singapore. |
354 | | Github Reference | [https://github.com/ryderling/DEEPSEC](https://github.com/ryderling/DEEPSEC) |
355 | | Language | Python |
356 | | Licensing | Open-source under the Apache License 2.0. |
357 | | Provides Mitigation | Prevention: No ❌Detection: Yes ✅ |
358 | | API Availability | Yes ✅ |
359 |
360 | | Factor | Details |
361 | | --- | --- |
362 | | **Popularity** | - **GitHub Stars:** 209 (as of 2024) |
363 | | | - **GitHub Forks:** ~70 |
364 | | | - **Number of Issues:** ~15 open issues |
365 | | | - **Trend:** Stable with a focus on deep learning security |
366 | | **Community Support** | - **Active Issues:** Currently has ongoing issues and updates, suggesting active maintenance. |
367 | | | - **Documentation:** Available through GitHub, covering setup, use, and contributions. |
368 | | | - **Discussion Forums:** GitHub Discussions section and community channels support developer interactions. |
369 | | | - **Contributors:** A small but dedicated contributor base. |
370 | | **Scalability** | - **Framework Support:** Primarily supports PyTorch and additional libraries like TorchVision. |
371 | | | - **Large-Scale Deployment:** Suitable for research and testing environments but may need adjustments for production-grade scaling |
372 | | **Integration** | - **Compatibility:** Compatible with machine learning libraries in Python. |
373 |
374 | **Tool Rating**
375 |
376 | | **Criteria** | **High** | **Medium** | **Low** |
377 | | --- | --- | --- | --- |
378 | | **Popularity** | | | ✅ |
379 | | **Community Support** | | | ✅ |
380 | | **Scalability** | | | ✅ |
381 | | **Ease of Integration** | | | ✅ |
382 |
383 | **Data Modality**
384 |
385 | | Data Modality | Supported |
386 | | --- | --- |
387 | | Text | ✅ |
388 | | Image | ✅ |
389 | | Audio | |
390 | | Video | |
391 | | Tabular data | |
392 |
393 | **Machine Learning Tasks**
394 |
395 | | Task Type | Data Modality | Supported |
396 | | --- | --- | --- |
397 | | Classification | All (See Data modality section) | ✅ |
398 | | Object Detection | Computer Vision | |
399 | | Speech Recognition | Audio | |
400 |
401 | **Framework Applicability**
402 |
403 | | Framework / Tool | Category | Supported |
404 | | --- | --- | --- |
405 | | Tensorflow | DL, GenAI | ✅ |
406 | | Keras | DL, GenAI | |
407 | | PyTorch | DL, GenAI | ✅ |
408 | | MxNet | DL | |
409 | | Scikit-learn | ML | |
410 | | XGBoost | ML | |
411 | | LightGBM | ML | |
412 | | CatBoost | ML | |
413 | | GPy | ML | |
414 |
415 | **OWASP AI Exchange Threat Coverage**
416 |
417 | | Topic | Coverage |
418 | | --- | --- |
419 | | Development time model poisoning | |
420 | | Runtime model poisoning | |
421 | | Model theft by use | |
422 | | Training data poisoning | |
423 | | Training data leak | |
424 | | Runtime model theft | |
425 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
426 | | Model inversion / Membership inference | |
427 | | Denial of model service | |
428 | | Direct prompt injection | |
429 | | Data disclosure | |
430 | | Model input leak | |
431 | | Indirect prompt injection | |
432 | | Development time model theft | |
433 | | Output contains injection | |
434 |
435 | Notes:
436 |
437 | Evasion:Tests model performance against adversarial inputs
438 |
439 | [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
440 |
441 | ### Tool Name: TextAttack
442 |
443 | | **Tool Name: TextAttack** | |
444 | | --- | --- |
445 | | Developer/ Source | Developed by researchers at the University of Maryland and Google Research. |
446 | | Github Reference | [https://github.com/QData/TextAttack](https://github.com/QData/TextAttack) |
447 | | Language | Python |
448 | | Licensing | Open-source under the MIT License. |
449 | | Provides Mitigation | Prevention: No ❌ Detection: Yes ✅ |
450 | | API Availability | Yes ✅ |
451 |
452 | | Factor | Details |
453 | | --- | --- |
454 | | **Popularity** | - **GitHub Stars:** ~3.7K (as of 2024) |
455 | | | - **GitHub Forks:** ~455 |
456 | | | - **Number of Issues:** ~130 open issues |
457 | | | - **Trend:** Popular with ongoing updates and regular contributions |
458 | | **Community Support** | - **Active Issues:** Issues are actively managed with frequent bug fixes and improvements. |
459 | | | - **Documentation:** Detailed documentation is available, covering everything from attack configuration to custom dataset integration |
460 | | | - **Discussion Forums:** GitHub Discussions are active, with support for technical queries and community interaction. |
461 | | | - **Contributors:** Over 20 contributors, reflecting diverse input and enhancements. |
462 | | **Scalability** | - **Framework Support:** Supports NLP models in PyTorch and integrates well with Hugging Face’s Transformers and Datasets libraries, making it compatible with a broad range of NLP tasks. |
463 | | | - **Large-Scale Deployment:** Primarily designed for research and experimentation; deployment at scale would likely require customization. |
464 | | **Integration** | - **Compatibility:** Model-agnostic, allowing use with various NLP model architectures as long as they meet the interface requirements. |
465 |
466 | **Tool Rating**
467 |
468 | | **Criteria** | **High** | **Medium** | **Low** |
469 | | --- | --- | --- | --- |
470 | | **Popularity** | ✅ | | |
471 | | **Community Support** | ✅ | | |
472 | | **Scalability** | | ✅ | |
473 | | **Ease of Integration** | ✅ | | |
474 |
475 | **Data Modality**
476 |
477 | | Data Modality | Supported |
478 | | --- | --- |
479 | | Text | ✅ |
480 | | Image | |
481 | | Audio | |
482 | | Video | |
483 | | Tabular data | |
484 |
485 | **Machine Learning Tasks**
486 |
487 | | Task Type | Data Modality | Supported |
488 | | --- | --- | --- |
489 | | Classification | All (See Data modality section) | ✅ |
490 | | Object Detection | Computer Vision | |
491 | | Speech Recognition | Audio | |
492 |
493 | **Framework Applicability**
494 |
495 | | Framework / Tool | Category | Supported |
496 | | --- | --- | --- |
497 | | Tensorflow | DL, GenAI | ✅ |
498 | | Keras | DL, GenAI | |
499 | | PyTorch | DL, GenAI | ✅ |
500 | | MxNet | DL | |
501 | | Scikit-learn | ML | |
502 | | XGBoost | ML | |
503 | | LightGBM | ML | |
504 | | CatBoost | ML | |
505 | | GPy | ML | |
506 |
507 | **OWASP AI Exchange Threat Coverage**
508 |
509 | | Topic | Coverage |
510 | | --- | --- |
511 | | Development time model poisoning | ✅ |
512 | | Runtime model poisoning | |
513 | | Model theft by use | |
514 | | Training data poisoning | |
515 | | Training data leak | |
516 | | Runtime model theft | |
517 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
518 | | Model inversion / Membership inference | |
519 | | Denial of model service | |
520 | | Direct prompt injection | |
521 | | Data disclosure | |
522 | | Model input leak | |
523 | | Indirect prompt injection | |
524 | | Development time model theft | |
525 | | Output contains injection | |
526 |
527 | Notes:
528 |
529 | - Development-time Model poisoning: Simulates attacks during development to evaluate vulnerabilities[*https://owaspai.org/goto/modelpoison/*](https://owaspai.org/goto/modelpoison/)
530 | - Evasion:Tests model performance against adversarial inputs[*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
531 |
532 | ## Open source Tools for Generative AI Red Teaming
533 |
534 | This sub section covers the following tools for security testing Generative AI: PyRIT, Garak, Prompt Fuzzer, Guardrail, and Promptfoo.
535 |
536 | A list of GenAI test tools can also be found at the [OWASP GenAI security project solutions page](https://genai.owasp.org/ai-security-solutions-landscape/) (click the category 'Test & Evaluate'. This project also published a [GenAI Red Teaming guide](https://genai.owasp.org/resource/genai-red-teaming-guide/).
537 |
538 |
539 | ### Tool Name: PyRIT
540 |
541 | | **Tool Name: PyRIT** | |
542 | | --- | --- |
543 | | Developer/ Source | Microsoft |
544 | | Github Reference | [https://github.com/Azure/PyRIT](https://github.com/Azure/PyRIT) |
545 | | Language | Python |
546 | | Licensing | Open-source under the MIT License. |
547 | | Provides Mitigation | Prevention: No ❌ Detection: Yes ✅ |
548 | | API Availability | Yes ✅ , library based |
549 |
550 | | Factor | Details |
551 | | --- | --- |
552 | | **Popularity** | - **GitHub Stars:** ~2k (as of Dec-2024) |
553 | | | - **GitHub Forks:** ~384forks |
554 | | | - **Number of Issues:** ~63 open issues, 79 closed issues |
555 | | | - **Trend:** Steady growth, with consistent updates and industry adoption for adversarial robustness. |
556 | | **Community Support** | - **Active Issues:** Issues are being addressed within a week. |
557 | | | - **Documentation:** Detailed and regularly updated, with comprehensive guides and API documentation. |
558 | | | - **Discussion Forums:** Active GitHub issues |
559 | | | - **Contributors:** Over 125 contributors. |
560 | | **Scalability** | - **Framework Support:** Scales across TensorFlow, PyTorch and supports models on local like ONNX |
561 | | | - **Large-Scale Deployment:** Can be extended to Azure pipeline. |
562 | | **Integration** | - **Compatibility:** Compatible with majority of LLMs |
563 |
564 | **Tool Rating**
565 |
566 | | **Criteria** | **High** | **Medium** | **Low** |
567 | | --- | --- | --- | --- |
568 | | **Popularity** | | ✅ | |
569 | | **Community Support** | ✅ | | |
570 | | **Scalability** | ✅ | | |
571 | | **Ease of Integration** | | ✅ | |
572 |
573 | **Data Modality**
574 |
575 | | Data Modality | Supported |
576 | | --- | --- |
577 | | Text | ✅ |
578 | | Image | |
579 | | Audio | |
580 | | Video | |
581 | | Tabular data | |
582 |
583 | **Machine Learning Tasks**
584 |
585 | | Task Type | Data Modality | Supported |
586 | | --- | --- | --- |
587 | | Classification | All (See Data modality section) | ✅ |
588 | | Object Detection | Computer Vision | ✅ |
589 | | Speech Recognition | Audio | ✅ |
590 |
591 | **Framework Applicability**
592 |
593 | | Framework / Tool | Category | Supported |
594 | | --- | --- | --- |
595 | | Tensorflow | DL, GenAI | ✅ |
596 | | PyTorch | DL, GenAI | ✅ |
597 | | Azure OpenAI | GenAI | ✅ |
598 | | Huggingface | ML, GenAI | ✅ |
599 | | Azure managed endpoints | Machine Learning Deployment | ✅ |
600 | | Cohere | GenAI | ✅ |
601 | | Replicate Text Models | GenAI | ✅ |
602 | | OpenAI API | GenAI | ✅ |
603 | | GGUF (Llama.cpp) | GenAI, Lightweight Inference | ✅ |
604 |
605 | **OWASP AI Exchange Threat Coverage**
606 |
607 | | Topic | Coverage |
608 | | --- | --- |
609 | | Development time model poisoning | |
610 | | Runtime model poisoning | |
611 | | Model theft by use | |
612 | | Training data poisoning | |
613 | | Training data leak | |
614 | | Runtime model theft | |
615 | | Evasion Tests model performance against adversarial inputs | ✅ |
616 | | Model inversion / Membership inference | |
617 | | Denial of model service | |
618 | | Direct prompt injection | ✅ |
619 | | Data disclosure | |
620 | | Model input leak | |
621 | | Indirect prompt injection | |
622 | | Development time model theft | |
623 | | Output contains injection | |
624 |
625 | Notes:
626 |
627 | - Evasion:Tests model performance against adversarial inputs [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
628 | - Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards.*https://owaspai.org/goto/promptinjection/*
629 |
630 | ### Tool Name: Garak
631 |
632 | | **Tool Name: Garak** | |
633 | | --- | --- |
634 | | Developer/ Source | NVIDIA |
635 | | Github Reference | https://docs.garak.ai/garak moved to https://github.com/NVIDIA/garak
636 | Literature: https://arxiv.org/abs/2406.11036
637 | https://github.com/NVIDIA/garak |
638 | | Language | Python |
639 | | Licensing | Apache 2.0 License |
640 | | Provides Mitigation | Prevention: No ❌ Detection: Yes ✅ |
641 | | API Availability | Yes ✅ |
642 |
643 | | Factor | Details |
644 | | --- | --- |
645 | | **Popularity** | - **GitHub Stars:** ~3,5K stars (as of Dec 2024) |
646 | | | - **GitHub Forks:** ~306forks |
647 | | | - **Number of Issues:** ~303 open issues, 299 closed issues |
648 | | | - **Trend:** Growing, particularly with in attack generation, and LLM vulnerability scanning. |
649 | | **Community Support** | - **Active Issues:** Actively responds to the issues and try to close it within a week |
650 | | | - **Documentation:** Detailed documentation with guidance and example experiments. |
651 | | | - **Discussion Forums:** Active GitHub discussions, as well as discord. |
652 | | | - **Contributors:** Over 27 contributors. |
653 | | **Scalability** | - **Framework Support:** Supports various LLMs from hugging face, openai api, litellm. |
654 | | | - **Large-Scale Deployment:** Mostly used in attack LLM, detect LLM failures and assessing LLM security. Can be integrated with NeMo Guardrails |
655 | | **Integration** | - **Compatibility:** All LLMs, Nvidia models |
656 |
657 | **Tool Rating**
658 |
659 | | **Criteria** | **High** | **Medium** | **Low** |
660 | | --- | --- | --- | --- |
661 | | **Popularity** | ✅ | | |
662 | | **Community Support** | | ✅ | |
663 | | **Scalability** | | ✅ | |
664 | | **Ease of Integration** | | ✅ | |
665 |
666 | **Data Modality**
667 |
668 | | Data Modality | Supported |
669 | | --- | --- |
670 | | Text | ✅ |
671 | | Image | |
672 | | Audio | |
673 | | Video | |
674 | | Tabular data | |
675 |
676 | **Machine Learning Tasks**
677 |
678 | | Task Type | Data Modality | Supported |
679 | | --- | --- | --- |
680 | | Classification | All (See Data modality section) | ✅ |
681 | | Object Detection | Computer Vision | ✅ |
682 | | Speech Recognition | Audio | |
683 |
684 | **Framework Applicability**
685 |
686 | | Framework / Tool | Category | Supported |
687 | | --- | --- | --- |
688 | | Tensorflow | DL, GenAI | |
689 | | PyTorch | DL, GenAI | ✅ |
690 | | Azure OpenAI | GenAI | |
691 | | Huggingface | ML, GenAI | ✅ |
692 | | Azure managed endpoints | Machine Learning Deployment | |
693 | | Cohere | GenAI | ✅ |
694 | | Replicate Text Models | GenAI | ✅ |
695 | | OpenAI API | GenAI | ✅ |
696 | | GGUF (Llama.cpp) | GenAI, Lightweight Inference | ✅ |
697 | | OctoAI | GenAI | ✅ |
698 |
699 | **OWASP AI Exchange Threat Coverage**
700 |
701 | | Topic | Coverage |
702 | | --- | --- |
703 | | Development time model poisoning | |
704 | | Runtime model poisoning | |
705 | | Model theft by use | |
706 | | Training data poisoning | |
707 | | Training data leak | |
708 | | Runtime model theft | |
709 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
710 | | Model inversion / Membership inference | |
711 | | Denial of model service | |
712 | | Direct prompt injection | ✅ |
713 | | Data disclosure | |
714 | | Model input leak | |
715 | | Indirect prompt injection | |
716 | | Development time model theft | |
717 | | Output contains injection | |
718 | - Evasion:Tests model performance against adversarial inputs [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
719 | - Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards.
720 | *https://owaspai.org/goto/promptinjection/*
721 |
722 | ### Tool Name: Prompt Fuzzer
723 |
724 | | **Tool Name: Prompt Fuzzer** | |
725 | | --- | --- |
726 | | Developer/ Source | Prompt Security |
727 | | Github Reference | [https://github.com/prompt-security/ps-fuzz](https://github.com/prompt-security/ps-fuzz) |
728 | | Language | Python |
729 | | Licensing | Open-source under the MIT License. |
730 | | Provides Mitigation | Prevention: No ❌ Detection: Yes ✅ |
731 | | API Availability | Yes ✅ |
732 |
733 | | Factor | Details |
734 | | --- | --- |
735 | | **Popularity** | - **GitHub Stars:** ~427 stars (as of Dec 2024) |
736 | | | - **GitHub Forks:** ~56 forks |
737 | | | - **Number of Issues:** ~10 open issues, 6 closed issues |
738 | | | - **Trend:** Not updating since Aug |
739 | | **Community Support** | - **Active Issues:** Not updated or solved bugs since July. |
740 | | | - **Documentation:** Moderate documentation with few examples |
741 | | | - **Discussion Forums:** GitHub issue forums |
742 | | | - **Contributors:** Over 10 contributors. |
743 | | **Scalability** | - **Framework Support:** Python and docker image. |
744 | | | - **Large-Scale Deployment:** It only assesses the security of your GenAI application's system prompt against various dynamic LLM-based attacks, so can be integrated with current env. |
745 | | **Integration** | - **Compatibility:** Any device. |
746 |
747 | **Tool Rating**
748 |
749 | | **Criteria** | **High** | **Medium** | **Low** |
750 | | --- | --- | --- | --- |
751 | | **Popularity** | | | ✅ |
752 | | **Community Support** | | | ✅ |
753 | | **Scalability** | | ✅ | |
754 | | **Ease of Integration** | | ✅ | |
755 |
756 | **Data Modality**
757 |
758 | | Data Modality | Supported |
759 | | --- | --- |
760 | | Text | ✅ |
761 | | Image | |
762 | | Audio | |
763 | | Video | |
764 | | Tabular data | |
765 |
766 | **Machine Learning Tasks**
767 |
768 | | Task Type | Data Modality | Supported |
769 | | --- | --- | --- |
770 | | Classification | All (See Data modality section) | ✅ |
771 | | Object Detection | Computer Vision | |
772 | | Speech Recognition | Audio | |
773 |
774 | **Framework Applicability**
775 |
776 | *(LLM Model agnostic in the API mode of use)*
777 |
778 | | Framework / Tool | Category | Supported |
779 | | --- | --- | --- |
780 | | Tensorflow | DL, GenAI | |
781 | | PyTorch | DL, GenAI | |
782 | | Azure OpenAI | GenAI | |
783 | | Huggingface | ML, GenAI | |
784 | | Azure managed endpoints | Machine Learning Deployment | |
785 | | Cohere | GenAI | |
786 | | Replicate Text Models | GenAI | |
787 | | OpenAI API | GenAI | ✅ |
788 | | GGUF (Llama.cpp) | GenAI, Lightweight Inference | |
789 | | OctoAI | GenAI | |
790 |
791 | **OWASP AI Exchange Threat Coverage**
792 |
793 | | Topic | Coverage |
794 | | --- | --- |
795 | | Development time model poisoning | |
796 | | Runtime model poisoning | |
797 | | Model theft by use | |
798 | | Training data poisoning | |
799 | | Training data leak | |
800 | | Runtime model theft | |
801 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
802 | | Model inversion / Membership inference | |
803 | | Denial of model service | |
804 | | Direct prompt injection | ✅ |
805 | | Data disclosure | |
806 | | Model input leak | |
807 | | Indirect prompt injection | |
808 | | Development time model theft | |
809 | | Output contains injection | |
810 |
811 | Notes:
812 |
813 | - Evasion:Tests model performance against adversarial inputs [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
814 | - Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards. *https://owaspai.org/goto/promptinjection/*
815 |
816 | ### Tool Name: Guardrail
817 |
818 | | **Tool Name: Guardrail** | |
819 | | --- | --- |
820 | | Developer/ Source | Guardrails AI |
821 | | Github Reference | [GitHub - guardrails-ai/guardrails: Adding guardrails to large language models.](https://github.com/guardrails-ai/guardrails) | [Guardrails Hub | Guardrails AI](https://hub.guardrailsai.com/) |
822 | | Language | Python |
823 | | Licensing | Apache 2.0 License |
824 | | Provides Mitigation | Prevention: Yes ✅ Detection: Yes ✅ |
825 | | API Availability | |
826 |
827 | | Factor | Details |
828 | | --- | --- |
829 | | **Popularity** | - **GitHub Stars:** ~4,3K (as 2024) |
830 | | | - **GitHub Forks:** ~326 |
831 | | | - **Number of Issues:** ~296 Closed, 40 Open. |
832 | | | - **Trend:** Steady growth with consistent and timely updates. |
833 | | **Community Support** | - **Active Issues:** Issues are mostly solved within weeks. |
834 | | | - **Documentation:** Detailed documentation with examples and user guide |
835 | | | - **Discussion Forums:** Primarily github issue and also support available on discord Server and twitter. |
836 | | | - **Contributors:** Over 60 contributors |
837 | | **Scalability** | - **Framework Support:** Supports Pytorch. Language: Python and Javascript. Working to add more support |
838 | | | - **Large-Scale Deployment:** Can be extended to Azure, langchain. |
839 | | **Integration** | - **Compatibility:** Compatible with various open source LLMs like OpenAI, Gemini, Anthropic. |
840 |
841 | **Tool Rating**
842 |
843 | | **Criteria** | **High** | **Medium** | **Low** |
844 | | --- | --- | --- | --- |
845 | | **Popularity** | ✅ | | |
846 | | **Community Support** | ✅ | | |
847 | | **Scalability** | | ✅ | |
848 | | **Ease of Integration** | ✅ | | |
849 |
850 | **Data Modality**
851 |
852 | | Data Modality | Supported |
853 | | --- | --- |
854 | | Text | ✅ |
855 | | Image | |
856 | | Audio | |
857 | | Video | |
858 | | Tabular data | |
859 |
860 | **Machine Learning Tasks**
861 |
862 | | Task Type | Data Modality | Supported |
863 | | --- | --- | --- |
864 | | Classification | All (See Data modality section) | ✅ |
865 | | Object Detection | Computer Vision | |
866 | | Speech Recognition | Audio | |
867 |
868 | **Framework Applicability**
869 |
870 | | Framework / Tool | Category | Supported |
871 | | --- | --- | --- |
872 | | Tensorflow | DL, GenAI | |
873 | | PyTorch | DL, GenAI | ✅ |
874 | | Azure OpenAI | GenAI | ✅ |
875 | | Huggingface | ML, GenAI | ✅ |
876 | | Azure managed endpoints | Machine Learning Deployment | |
877 | | Cohere | GenAI | ✅ |
878 | | Replicate Text Models | GenAI | |
879 | | OpenAI API | GenAI | ✅ |
880 | | GGUF (Llama.cpp) | GenAI, Lightweight Inference | |
881 | | OctoAI | GenAI | |
882 |
883 | **OWASP AI Exchange Threat Coverage**
884 |
885 | | Topic | Coverage |
886 | | --- | --- |
887 | | Development time model poisoning | |
888 | | Runtime model poisoning | |
889 | | Model theft by use | |
890 | | Training data poisoning | |
891 | | Training data leak | |
892 | | Runtime model theft | |
893 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
894 | | Model inversion / Membership inference | |
895 | | Denial of model service | |
896 | | Direct prompt injection | ✅ |
897 | | Data disclosure | |
898 | | Model input leak | |
899 | | Indirect prompt injection | |
900 | | Development time model theft | |
901 | | Output contains injection | |
902 |
903 | Notes:
904 |
905 | - Evasion:Tests model performance against adversarial inputs [*https://owaspai.org/goto/evasion/*](https://owaspai.org/goto/evasion/)
906 | - Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards. *https://owaspai.org/goto/promptinjection/*
907 |
908 | ### Tool Name: Promptfoo
909 |
910 | | **Tool Name: Promptfoo** | |
911 | | --- | --- |
912 | | Developer/ Source | Promptfoo community |
913 | | Github Reference | [https://github.com/promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | [Types of LLM vulnerabilities | promptfoo](https://www.promptfoo.dev/docs/red-team/llm-vulnerability-types/) |
914 | | Language | Python, NodeJS |
915 | | Licensing | Open-source under the MIT License. |
916 | | | This project is licensed under multiple licenses:
917 |
918 | 1. The main codebase is licensed under the MIT License (see below)
919 | 2. The `/src/redteam/` directory is proprietary and licensed under the Promptfoo Enterprise License
920 | 3. Some third-party components have their own licenses as indicated by LICENSE files in their respective directories |
921 | | Provides Mitigation | Prevention: Yes ✅ Detection: Yes ✅ |
922 | | API Availability | Yes ✅ |
923 |
924 | | Factor | Details |
925 | | --- | --- |
926 | | **Popularity** | - **GitHub Stars:** ~4.3K stars (as of 2024) |
927 | | | - **GitHub Forks:** ~320 forks |
928 | | | - **Number of Issues:** ~523 closed, 108 open |
929 | | | - **Trend:** Consistent update |
930 | | **Community Support** | - **Active Issues:** Issues are addressed within acouple of days. |
931 | | | - **Documentation:** Detailed documentation with user guide and examples. |
932 | | | - **Discussion Forums:** Active Github issue and also support available on Discord |
933 | | | - **Contributors:** Over 113 contributors. |
934 | | **Scalability** | - **Framework Support:** Language: JavaScript |
935 | | | - **Large-Scale Deployment:** Enterprise version available, that supports cloud deployment. |
936 | | **Integration** | - **Compatibility:** Compatible with majority of the LLMs |
937 |
938 | **Tool Rating**
939 |
940 | | **Criteria** | **High** | **Medium** | **Low** |
941 | | --- | --- | --- | --- |
942 | | **Popularity** | ✅ | | |
943 | | **Community Support** | ✅ | | |
944 | | **Scalability** | | ✅ | |
945 | | **Ease of Integration** | | ✅ | |
946 |
947 | **Data Modality**
948 |
949 | | Data Modality | Supported |
950 | | --- | --- |
951 | | Text | ✅ |
952 | | Image | |
953 | | Audio | |
954 | | Video | |
955 | | Tabular data | |
956 |
957 | **Machine Learning Tasks**
958 |
959 | | Task Type | Data Modality | Supported |
960 | | --- | --- | --- |
961 | | Classification | All (See Data modality section) | ✅ |
962 | | Object Detection | Computer Vision | |
963 | | Speech Recognition | Audio | |
964 |
965 | **Framework Applicability**
966 |
967 | | Framework / Tool | Category | Supported |
968 | | --- | --- | --- |
969 | | Tensorflow | DL, GenAI | |
970 | | PyTorch | DL, GenAI | |
971 | | Azure OpenAI | GenAI | ✅ |
972 | | Huggingface | ML, GenAI | ✅ |
973 | | Azure managed endpoints | Machine Learning Deployment | |
974 | | Cohere | GenAI | ✅ |
975 | | Replicate Text Models | GenAI | ✅ |
976 | | OpenAI API | GenAI | ✅ |
977 | | GGUF (Llama.cpp) | GenAI, Lightweight Inference | ✅ |
978 | | OctoAI | GenAI | |
979 |
980 | **OWASP AI Exchange Threat Coverage**
981 |
982 | | Topic | Coverage |
983 | | --- | --- |
984 | | Development time model poisoning | |
985 | | Runtime model poisoning | |
986 | | Model theft by use | |
987 | | Training data poisoning | |
988 | | Training data leak | |
989 | | Runtime model theft | |
990 | | Evasion (Tests model performance against adversarial inputs) | ✅ |
991 | | Model inversion / Membership inference | |
992 | | Denial of model service | |
993 | | Direct prompt injection | |
994 | | Data disclosure | |
995 | | Model input leak | |
996 | | Indirect prompt injection | ✅ |
997 | | Development time model theft | |
998 | | Output contains injection | |
999 |
1000 | Notes:
1001 |
1002 | - Model theft through use:Evaluates risks of model exploitation during usage [*https://owaspai.org/goto/modeltheftuse/*](https://owaspai.org/goto/modeltheftuse/)
1003 | - Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards.
1004 | *[https://owaspai.org/goto/promptinjection/](https://owaspai.org/goto/promptinjection/)*
1005 |
1006 | ## Tool Ratings
1007 | This section rates the discussed tools by Popularity, Community Support, Scalability and Integration.
1008 |
1009 | [](https://owaspai.org/images/testtoolrating.png)
1010 |
1011 | | **Attribute** | High | Medium | Low |
1012 | | --- | --- | --- | --- |
1013 | | Popularity | >3,000 stars | 1,000–3,000 stars | <1,000 stars |
1014 | | Community Support | >100 contributors, quick response (<3 days) | 50–100 contributors, response in 3–14 days | <50 contributors, slow response (>14 days) |
1015 | | Scalability | Proven enterprise-grade, multi-framework | Moderate scalability, limited frameworks | Research focused, small-scale |
1016 | | Integration | Broad compatibility | Limited compatibility, narrow use-case | Minimal or no integration, research tools only |
1017 |
1018 | Disclaimer on the use of the Assessment:
1019 |
1020 | - ***Scope of Assessment: This review exclusively focuses on open-source RedTeaming tools. Proprietary or commercial solutions were not included in this evaluation.***
1021 | - ***Independent Review: The evaluation is independent and based solely on publicly available information from sources such as GitHub repositories, official documentation, and related community discussions.***
1022 | - ***Tool Version and Relevance: The information and recommendations provided in this assessment are accurate as of September 2024. Any future updates, enhancements, or changes to these tools should be verified directly via the provided links or respective sources to ensure continued relevance.***
1023 |
1024 | ***Tool Fit and Usage:***
1025 |
1026 | *The recommendations in this report should be considered based on your organization's specific use case, scale, and security posture. Some tools may offer advanced features that may not be necessary for smaller projects or environments, while others may be better suited to specific frameworks or security goals.*
1027 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/docs/6_privacy.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: 6. AI privacy
3 | weight: 7
4 | ---
5 | > Category: discussion
6 | > Permalink: https://owaspai.org/goto/aiprivacy/
7 |
8 | Just like any system that processes data, AI systems can have privacy risks. There are some particualar privacy aspects to AI:
9 | - AI systems are data-intensive and typically present additional risks regarding data collection and retention. Personal data may be collected from various sources, each subject to different levels of **sensitivity and regulatory constraints**. Legislation often requires a **legal basis and/or consent** for the collection and use of personal data, and specifies **rights to individuals** to correct, request, and remove their own data.
10 | - **Protecting training data** is a challenge, especially because it typically needs to be retained for long periods - as many models need to be retrained. Often, the actual identities of people involved are irrelevant for the model, but privacy risks still remain even if identity data is removed because it might be possible to deduce individual identities from the remaining data. This is where differential privacy becomes crucial: by altering the data to make it sufficiently unrecognizable, it ensures individual privacy while still allowing for valuable insights to be derived from the data. Alteration can be done by for example adding noise or aggregating.
11 | - An additional complication in the protection of training data is that the **training data is accessible in the engineering environment**, which therefore needs more protection than it usually does - since conventional systems normally don't have personal data available to technical teams.
12 | - The nature of machine learning allows for certain **unique strategies** to improve privacy, such as federated learning: splitting up the training set in different separated systems - typically aligning with separated data collection.
13 | - AI systems **make decisions** and if these decisions are about people they may be discriminating regarding certain protected attributes (e.g. gender, race), plus the decisions may result in actions that invade privacy, which may be an ethical or legal concern. Furthermore, legislation may prohibit some types of decisions and sets rules regarding transparency about how these decisions are made, and about how individuals have the right to object.
14 | - Last but not least: AI models suffer from **model attack risks** that allow attackers to extract training data from the model, e.g. model inversion, membership inference, and disclosing sensitive data in large language models
15 |
16 |
17 | AI Privacy can be divided into two parts:
18 |
19 | 1. The threats to AI security and their controls (see the other sections of the AI Exchange), including:
20 | - Confidentiality and integrity protection of personal data in train/test data, model input or output - which consists of:
21 | - 'Conventional' security of personal data in transit and in rest
22 | - Protecting against model attacks that try to retrieve personal data (e.g. model inversion)
23 | - personal data minimization / differential privacy, including minimized retention
24 | - Integrity protection of the model behaviour if that behaviour can hurt privacy of individuals. This happens for example when individuals are unlawfully discriminated or when the model output leads to actions that invade privacy (e.g. undergoing a fraud investigation).
25 | 2. Threats and controls that are not about security, but about further rights of the individual, as covered by privacy regulations such as the GDPR, including use limitation, consent, fairness, transparency, data accuracy, right of correction/objection/erasure/request.
26 |
27 | Privacy principles and requirements come from different legislations (e.g. GDPR, LGPD, PIPEDA, etc.) and privacy standards (e.g. ISO 31700, ISO 29100, ISO 27701, FIPS, NIST Privacy Framework, etc.). This guideline does not guarantee compliance with privacy legislation and it is also not a guide on privacy engineering of systems in general. For that purpose, please consider work from [ENISA](https://www.enisa.europa.eu/publications/data-protection-engineering), [NIST](https://nvlpubs.nist.gov/nistpubs/ir/2017/NIST.IR.8062.pdf), [mplsplunk](https://github.com/mplspunk/awesome-privacy-engineering), [OWASP](https://owasp.org/www-project-top-10-privacy-risks/) and [OpenCRE](https://www.opencre.org/cre/362-550). The general principle for engineers is to regard personal data as 'radioactive gold'. It's valuable, but it's also something to minimize, carefully store, carefully handle, limit its usage, limit sharing, keep track of where it is, etc.
28 |
29 | This section covers how privacy principles apply to AI systems:
30 |
31 | ## 1. Use Limitation and Purpose Specification
32 |
33 | Essentially, you should not simply use data collected for one purpose (e.g. safety or security) as a training dataset to train your model for other purposes (e.g. profiling, personalized marketing, etc.) For example, if you collect phone numbers and other identifiers as part of your MFA flow (to improve security ), that doesn't mean you can also use it for user targeting and other unrelated purposes. Similarly, you may need to collect sensitive data under KYC requirements, but such data should not be used for ML models used for business analytics without proper controls.
34 |
35 | Some privacy laws require a lawful basis (or bases if for more than one purpose) for processing personal data (See GDPR's Art 6 and 9).
36 | Here is a link with certain restrictions on the purpose of an AI application, like for example the [prohibited practices in the European AI Act](https://artificialintelligenceact.eu/article/5) such as using machine learning for individual criminal profiling. Some practices are regarded as too riskful when it comes to potential harm and unfairness towards individuals and society.
37 |
38 | Note that a use case may not even involve personal data, but can still be potentially harmful or unfair to indiduals. For example: an algorithm that decides who may join the army, based on the amount of weight a person can lift and how fast the person can run. This data can not be used to reidentify individuals (with some exceptions), but still the use case may be unrightfully unfair towards gender (if the algorithm for example is based on an unfair training set).
39 |
40 | In practical terms, you should reduce access to sensitive data and create anonymized copies for incompatible purposes (e.g. analytics). You should also document a purpose/lawful basis before collecting the data and communicate that purpose to the user in an appropriate way.
41 |
42 | New techniques that enable use limitation include:
43 |
44 | * data enclaves: store pooled personal data in restricted secure environments
45 | * federated learning: decentralize ML by removing the need to pool data into a single location. Instead, the model is trained in multiple iterations at different sites.
46 |
47 |
48 |
49 | ## 2. Fairness
50 |
51 | Fairness means handling personal data in a way individuals expect and not using it in ways that lead to unjustified adverse effects. The algorithm should not behave in a discriminating way. (See also [this article](https://iapp.org/news/a/what-is-the-role-of-privacy-professionals-in-preventing-discrimination-and-ensuring-equal-treatment/)). Furthermore: accuracy issues of a model becomes a privacy problem if the model output leads to actions that invade privacy (e.g. undergoing fraud investigation). Accuracy issues can be caused by a complex problem, insufficient data, mistakes in data and model engineering, and manipulation by attackers. The latter example shows that there can be a relation between model security and privacy.
52 |
53 | GDPR's Article 5 refers to "fair processing" and EDPS' [guideline](https://edpb.europa.eu/sites/default/files/files/file1/edpb_guidelines_201904_dataprotection_by_design_and_by_default_v2.0_en.pdf) defines fairness as the prevention of "unjustifiably detrimental, unlawfully discriminatory, unexpected or misleading" processing of personal data. GDPR does not specify how fairness can be measured, but the EDPS recommends the right to information (transparency), the right to intervene (access, erasure, data portability, rectify), and the right to limit the processing (right not to be subject to automated decision-making and non-discrimination) as measures and safeguard to implement the principle of fairness.
54 |
55 | In the [literature](http://fairware.cs.umass.edu/papers/Verma.pdf), there are different fairness metrics that you can use. These range from group fairness, false positive error rate, unawareness, and counterfactual fairness. There is no industry standard yet on which metric to use, but you should assess fairness especially if your algorithm is making significant decisions about the individuals (e.g. banning access to the platform, financial implications, denial of services/opportunities, etc.). There are also efforts to test algorithms using different metrics. For example, NIST's [FRVT project](https://pages.nist.gov/frvt/html/frvt11.html) tests different face recognition algorithms on fairness using different metrics.
56 |
57 | The elephant in the room for fairness across groups (protected attributes) is that in situations a model is more accurate if it DOES discriminate protected attributes. Certain groups have in practice a lower success rate in areas because of all kinds of societal aspects rooted in culture and history. We want to get rid of that. Some of these aspects can be regarded as institutional discrimination. Others have more practical background, like for example that for language reasons we see that new immigrants statistically tend to be hindered in getting higher education.
58 | Therefore, if we want to be completely fair across groups, we need to accept that in many cases this will be balancing accuracy with discrimination. In the case that sufficient accuracy cannot be attained while staying within discrimination boundaries, there is no other option than to abandon the algorithm idea. For fraud detection cases, this could for example mean that transactions need to be selected randomly instead of by using an algorithm.
59 |
60 | A machine learning use case may have unsolvable bias issues, that are critical to recognize before you even start. Before you do any data analysis, you need to think if any of the key data elements involved have a skewed representation of protected groups (e.g. more men than women for certain types of education). I mean, not skewed in your training data, but in the real world. If so, bias is probably impossible to avoid - unless you can correct for the protected attributes. If you don't have those attributes (e.g. racial data) or proxies, there is no way. Then you have a dilemma between the benefit of an accurate model and a certain level of discrimination. This dilemma can be decided on before you even start, and save you a lot of trouble.
61 |
62 | Even with a diverse team, with an equally distributed dataset, and without any historical bias, your AI may still discriminate. And there may be nothing you can do about it.
63 | For example: take a dataset of students with two variables: study program and score on a math test. The goal is to let the model select students good at math for a special math program. Let's say that the study program 'computer science' has the best scoring students. And let's say that much more males then females are studying computer science. The result is that the model will select more males than females. Without having gender data in the dataset, this bias is impossible to counter.
64 |
65 | ## 3. Data Minimization and Storage Limitation
66 |
67 | This principle requires that you should minimize the amount, granularity and storage duration of personal information in your training dataset. To make it more concrete:
68 |
69 | * Do not collect or copy unnecessary attributes to your dataset if this is irrelevant for your purpose
70 | * Anonymize the data where possible. Please note that this is not as trivial as "removing PII". See [WP 29 Guideline](https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2014/wp216_en.pdf)
71 | * If full anonymization is not possible, reduce the granularity of the data in your dataset if you aim to produce aggregate insights (e.g. reduce lat/long to 2 decimal points if city-level precision is enough for your purpose or remove the last octets of an ip address, round timestamps to the hour)
72 | * Use less data where possible (e.g. if 10k records are sufficient for an experiment, do not use 1 million)
73 | * Delete data as soon as possible when it is no longer useful (e.g. data from 7 years ago may not be relevant for your model)
74 | * Remove links in your dataset (e.g. obfuscate user id's, device identifiers, and other linkable attributes)
75 | * Minimize the number of stakeholders who accesses the data on a "need to know" basis
76 |
77 | There are also privacy-preserving techniques being developed that support data minimization:
78 |
79 | * distributed data analysis: exchange anonymous aggregated data
80 | * secure multi-party computation: store data distributed-encrypted
81 |
82 | Further reading:
83 | * [ICO guidance on AI and data protection](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/)
84 | * [FPF case-law analysis on automated decision making](https://fpf.org/blog/fpf-report-automated-decision-making-under-the-gdpr-a-comprehensive-case-law-analysis/)
85 |
86 |
87 | ## 4. Transparency
88 | Privacy standards such as FIPP or ISO29100 refer to maintaining privacy notices, providing a copy of user's data upon request, giving notice when major changes in personal data processing occur, etc.
89 |
90 | GDPR also refers to such practices but also has a specific clause related to algorithmic-decision making.
91 | GDPR's [Article 22](https://ec.europa.eu/newsroom/article29/items/612053) allows individuals specific rights under specific conditions. This includes getting a human intervention to an algorithmic decision, an ability to contest the decision, and get a meaningful information about the logic involved. For examples of "meaningful information", see EDPS's [guideline](https://ec.europa.eu/newsroom/article29/items/612053). The US [Equal Credit Opportunity Act](https://www.consumerfinance.gov/about-us/newsroom/cfpb-acts-to-protect-the-public-from-black-box-credit-models-using-complex-algorithms/) requires detailed explanations on individual decisions by algorithms that deny credit.
92 |
93 | Transparency is not only needed for the end-user. Your models and datasets should be understandable by internal stakeholders as well: model developers, internal audit, privacy engineers, domain experts, and more. This typically requires the following:
94 |
95 | * proper model documentation: model type, intent, proposed features, feature importance, potential harm, and bias
96 | * dataset transparency: source, lawful basis, type of data, whether it was cleaned, age. Data cards is a popular approach in the industry to achieve some of these goals. See Google Research's [paper](https://arxiv.org/abs/2204.01075) and Meta's [research](https://ai.facebook.com/research/publications/system-level-transparency-of-machine-learning).
97 | * traceability: which model has made that decision about an individual and when?
98 | * explainability: several methods exist to make black-box models more explainable. These include LIME, SHAP, counterfactual explanations, Deep Taylor Decomposition, etc. See also [this overview of machine learning interpretability](https://github.com/jphall663/awesome-machine-learning-interpretability) and [this article on the pros and cons of explainable AI](https://www.softwareimprovementgroup.com/resources/unraveling-the-incomprehensible-the-pros-and-cons-of-explainable-ai/).
99 |
100 | ## 5. Privacy Rights
101 | Also known as "individual participation" under privacy standards, this principle allows individuals to submit requests to your organization related to their personal data. Most referred rights are:
102 |
103 | 1. right of access/portability: provide a copy of user data, preferably in a machine-readable format. If data is properly anonymized, it may be exempted from this right.
104 | 2. right of erasure: erase user data unless an exception applies. It is also a good practice to re-train your model without the deleted user's data.
105 | 3. right of correction: allow users to correct factually incorrect data. Also, see accuracy below
106 | 4. right of object: allow users to object to the usage of their data for a specific use (e.g. model training)
107 |
108 | ## 6. Data accuracy
109 | You should ensure that your data is correct as the output of an algorithmic decision with incorrect data may lead to severe consequences for the individual. For example, if the user's phone number is incorrectly added to the system and if such number is associated with fraud, the user might be banned from a service/system in an unjust manner. You should have processes/tools in place to fix such accuracy issues as soon as possible when a proper request is made by the individual.
110 |
111 | To satisfy the accuracy principle, you should also have tools and processes in place to ensure that the data is obtained from reliable sources, its validity and correctness claims are validated and data quality and accuracy are periodically assessed.
112 |
113 | ## 7. Consent
114 | Consent may be used or required in specific circumstances. In such cases, consent must satisfy the following:
115 |
116 | 1. obtained before collecting, using, updating, or sharing the data
117 | 2. consent should be recorded and be auditable
118 | 3. consent should be granular (use consent per purpose, and avoid blanket consent)
119 | 4. consent should not be bundled with T&S
120 | 5. consent records should be protected from tampering
121 | 6. consent method and text should adhere to specific requirements of the jurisdiction in which consent is required (e.g. GDPR requires unambiguous, freely given, written in clear and plain language, explicit and withdrawable)
122 | 7. Consent withdrawal should be as easy as giving consent
123 | 8. If consent is withdrawn, then all associated data with the consent should be deleted and the model should be re-trained.
124 |
125 | Please note that consent will not be possible in specific circumstances (e.g. you cannot collect consent from a fraudster and an employer cannot collect consent from an employee as there is a power imbalance). If you must collect consent, then ensure that it is properly obtained, recorded and proper actions are taken if it is withdrawn.
126 |
127 |
128 | ## 8. Model attacks
129 | See the security section for security threats to data confidentiality, as they of course represent a privacy risk if that data is personal data. Notable: membership inference, model inversion, and training data leaking from the engineering process. In addition, models can disclose sensitive data that was unintendedly stored during training.
130 |
131 |
132 | ## Scope boundaries of AI privacy
133 | As said, many of the discussion topics on AI are about human rights, social justice, safety and only a part of it has to do with privacy. So as a data protection officer or engineer it's important not to drag everything into your responsibilities. At the same time, organizations do need to assign those non-privacy AI responsibilities somewhere.
134 |
135 |
136 | ## Before you start: Privacy restrictions on what you can do with AI
137 | The GDPR does not restrict the applications of AI explicitly but does provide safeguards that may limit what you can do, in particular regarding Lawfulness and limitations on purposes of collection, processing, and storage - as mentioned above. For more information on lawful grounds, see [article 6](https://gdpr.eu/article-6-how-to-process-personal-data-legally/)
138 |
139 | The [US Federal Trade Committee](https://www.ftc.gov/business-guidance/blog/2023/02/keep-your-ai-claims-check) provides some good (global) guidance in communicating carefully about your AI, including not to overpromise.
140 |
141 | The [EU AI act](https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52021PC0206&from=EN) does pose explicit application limitations, such as mass surveillance, predictive policing, and restrictions on high-risk purposes such as selecting people for jobs. In addition, there are regulations for specific domains that restrict the use of data, putting limits to some AI approaches (e.g. the medical domain).
142 |
143 | **The EU AI Act in a nutshell:**
144 |
145 | Safety, health and fundamental rights are at the core of the AI Act, so risks are analyzed from a perspective of harmfulness to people.
146 |
147 | The Act identifies four risk levels for AI systems:
148 | * **Unacceptable risk**: will be banned. Includes: Manipulation of people, social scoring, and real-time remote biometric identification (e.g. face recognition with cameras in public space).
149 | * **High risk**: products already under safety legislation, plus eight areas (including critical infrastructure and law enforcement). These systems need to comply with a number of rules including the a security risk assessment and conformity with harmonized (adapted) AI security standards OR the essential requirements of the Cyber Resilience Act (when applicable).
150 | * **Limited risk**: has limited potential for manipulation. Should comply with minimal transparency requirements to users that would allow users to make informed decisions. After interacting with the applications, the user can then decide whether they want to continue using it.
151 | * **Minimal/non risk**: the remaining systems.
152 |
153 | So organizations will have to know their AI initiatives and perform high-level risk analysis to determine the risk level.
154 |
155 | AI is broadly defined here and includes wider statistical approaches and optimization algorithms.
156 |
157 | Generative AI needs to disclose what copyrighted sources were used, and prevent illegal content. To illustrate: if OpenAI for example would violate this rule, they could face a 10 billion dollar fine.
158 |
159 | Links:
160 | * [AI Act](https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence)
161 | * [Guidelines on prohibted AI](https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act)
162 | * [AI Act page of te EU](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai)
163 |
164 | ## Further reading on AI privacy
165 |
166 | * [NIST AI Risk Management Framework 1.0](https://doi.org/10.6028/NIST.AI.100-1)
167 | * [PLOT4ai threat library ](https://plot4.ai/library)
168 | * [Algorithm audit non-profit organisation](https://algorithmaudit.eu/)
169 | * For pure security aspects: see the 'Further reading on AI security' above in this document
170 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/docs/_index.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Content
3 | ---
4 |
5 | {{< cards >}}
6 | {{< small-card link="/docs/ai_security_overview/" title="0.AI Security Overview">}}
7 | {{< small-card link="/docs/1_general_controls/" title="1. General controls">}}
8 | {{< small-card link="/docs/2_threats_through_use/" title="2. Threats through use">}}
9 | {{< small-card link="/docs/3_development_time_threats/" title="3. Development-time threats">}}
10 | {{< small-card link="/docs/4_runtime_application_security_threats/" title="4. Runtime application security threats">}}
11 | {{< small-card link="/docs/5_testing/" title="5. AI security testing">}}
12 | {{< small-card link="/docs/ai_security_references/" title="References">}}
13 | {{< /cards >}}
14 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/docs/ai_security_references.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: AI Security References
3 | weight: 8
4 | ---
5 | ## References of the OWASP AI Exchange
6 | >Category: discussion
7 | >Permalink: https://owaspai.org/goto/references/
8 |
9 | See the [Media page](/media) for several webinars and podcast by and about the AI Exchange.
10 | References on specific topics can be found throught the content of AI Exchange. This references section therefore contains the broader publications.
11 |
12 | ## Overviews of AI Security Threats:
13 | ---
14 | - [OWASP LLM top 10](https://genai.owasp.org/)
15 | - [ENISA Cybersecurity threat landscape](https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges)
16 | - [ENISA ML threats and countermeasures 2021](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)
17 | - [MITRE ATLAS framework for AI threats](https://atlas.mitre.org/)
18 | - [NIST threat taxonomy](https://csrc.nist.gov/publications/detail/white-paper/2023/03/08/adversarial-machine-learning-taxonomy-and-terminology/draft)
19 | - [ETSI SAI](https://www.etsi.org/technologies/securing-artificial-intelligence)
20 | - [Microsoft AI failure modes](https://docs.microsoft.com/en-us/security/failure-modes-in-machine-learning)
21 | - [NIST](https://csrc.nist.gov/pubs/ai/100/2/e2023/final)
22 | - [NISTIR 8269 - A Taxonomy and Terminology of Adversarial Machine Learning](https://csrc.nist.rip/external/nvlpubs.nist.gov/nistpubs/ir/2019/NIST.IR.8269-draft.pdf)
23 | - [OWASP ML top 10](https://mltop10.info/)
24 | - [BIML ML threat taxonomy](https://berryvilleiml.com/taxonomy/)
25 | - [BIML LLM risk analysis - please register there](https://berryvilleiml.com/docs/BIML-LLM24.pdf)
26 | - [PLOT4ai threat library](https://plot4.ai/library)
27 | - [BSI AI recommendations including security aspects (Germany) - in English](https://www.bsi.bund.de/EN/Themen/Unternehmen-und-Organisationen/Informationen-und-Empfehlungen/Kuenstliche-Intelligenz/kuenstliche-intelligenz_node.html#doc916902bodyText8)
28 | - [NCSC UK / CISA Joint Guidelines](https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development) - see [its mapping with the AI Exchange](/goto/jointguidelines/)
29 |
30 | ## Overviews of AI Security/Privacy Incidents:
31 | ---
32 | - [AVID AI Vulnerability database](https://avidml.org/)
33 | - [Sightline - AI/ML Supply Chain Vulnerability Database](https://sightline.protectai.com/)
34 | - [OECD AI Incidents Monitor (AIM)](https://oecd.ai/en/incidents)
35 | - [AI Incident Database](https://incidentdatabase.ai/)
36 | - [AI Exploits by ProtectAI](https://github.com/protectai/ai-exploits)
37 |
38 | ## Misc.:
39 | ---
40 | - [ENISA AI security standard discussion](https://www.enisa.europa.eu/publications/cybersecurity-of-ai-and-standardisation)
41 | - [ENISA's multilayer AI security framework](https://www.enisa.europa.eu/publications/multilayer-framework-for-good-cybersecurity-practices-for-ai)
42 | - [Alan Turing institute's AI standards hub](https://aistandardshub.org)
43 | - [Microsoft/MITRE tooling for ML teams](https://www.mitre.org/news-insights/news-release/microsoft-and-mitre-create-tool-help-security-teams-prepare-attacks?sf175190906=1)
44 | - [Google's Secure AI Framework](https://blog.google/technology/safety-security/introducing-googles-secure-ai-framework/)
45 | - [NIST AI Risk Management Framework 1.0](https://doi.org/10.6028/NIST.AI.100-1)
46 | - [ISO/IEC 20547-4 Big data security](https://www.iso.org/standard/71278.html)
47 | - [IEEE 2813 Big Data Business Security Risk Assessment](https://standards.ieee.org/ieee/2813/7535/)
48 | - [Awesome MLSecOps references](https://github.com/RiccardoBiosas/awesome-MLSecOps)
49 | - [OffSec ML Playbook](https://wiki.offsecml.com/)
50 | - [MIT AI Risk Repository](https://airisk.mit.edu/)
51 | - [Failure Modes in Machine Learning by Microsoft](https://learn.microsoft.com/en-us/security/engineering/failure-modes-in-machine-learning)
52 |
53 | ## Learning and Training:
54 | ---
55 | | Category | Title | Description | Provider | Content Type | Level | Cost | Link |
56 | |---------------------------|----------------------------------------------|---------------------------------------------------------------------------------------------------|----------------|------------------|-------------|----------------------------------|----------------------------------------------------------------------------------------------|
57 | | **Courses and Labs** | **AI Security Fundamentals** | Learn the basic concepts of AI security, including security controls and testing procedures. | Microsoft | Course | Beginner | Free | [AI Security Fundamentals](https://learn.microsoft.com/en-us/training/paths/ai-security-fundamentals/) |
58 | | | **Red Teaming LLM Applications** | Explore fundamental vulnerabilities in LLM applications with hands-on lab practice. | Giskard | Course + Lab | Beginner | Free | [Red Teaming LLM Applications](https://www.deeplearning.ai/short-courses/red-teaming-llm-applications/) |
59 | | | **Exploring Adversarial Machine Learning** | Designed for data scientists and security professionals to learn how to attack realistic ML systems.| NVIDIA | Course + Lab | Intermediate | Paid | [Exploring Adversarial Machine Learning](https://learn.nvidia.com/courses/course-detail?course_id=course-v1:DLI+S-DS-03+V1) |
60 | | | **OWASP LLM Vulnerabilities** | Essentials of securing Large Language Models (LLMs), covering basic to advanced security practices.| Checkmarx | Interactive Lab | Beginner | Free with OWASP Membership | [OWASP LLM Vulnerabilities](https://owasp.codebashing.com/app/course?courseUuid=d0e55509-bff3-4860-8d0e-141a59ef152b) |
61 | | | **OWASP TOP 10 for LLM** | Scenario-based LLM security vulnerabilities and their mitigation strategies. | Security Compass | Interactive Lab | Beginner | Free | [OWASP TOP 10 for LLM](https://application.security/free/llm) |
62 | | | **Web LLM Attacks** | Hands-on lab to practice exploiting LLM vulnerabilities. | Portswigger | Lab | Beginner | Free | [Web LLM Attacks](https://portswigger.net/web-security/llm-attacks) |
63 | | | **Path: AI Red Teamer** | Covers OWASP ML/LLM Top 10 and attacking ML-based systems. | HackTheBox Academy | Course + Lab | Beginner | Paid | [HTB AI Red Teamer](https://academy.hackthebox.com/) |
64 | | | **Path: Artificial Intelligence and Machine Learning** | Hands-on lab to practice AI/ML vulnerabilities exploitation. | HackTheBox Enterprise | Dedicated Lab | Beginner, Intermediate | Enterprise Plan | [HTB AI/ML Lab](https://enterprise.hackthebox.com/) |
65 | | **CTF Practices** | **AI Capture The Flag** | A series of AI-themed challenges ranging from easy to hard, hosted by DEFCON AI Village. | Crucible / AIV | CTF | Beginner, Intermediate | Free | [AI Capture The Flag](https://crucible.dreadnode.io/) |
66 | | | **IEEE SaTML CTF 2024** | A Capture-the-Flag competition focused on Large Language Models. | IEEE | CTF | Beginner, Intermediate | Free | [IEEE SaTML CTF 2024](https://ctf.spylab.ai/) |
67 | | | **Gandalf Prompt CTF** | A gamified challenge focusing on prompt injection techniques. | Lakera | CTF | Beginner | Free | [Gandalf Prompt CTF](https://gandalf.lakera.ai/) |
68 | | | **HackAPrompt** | A prompt injection playground for participants of the HackAPrompt competition. | AiCrowd | CTF | Beginner | Free | [HackAPrompt](https://huggingface.co/spaces/hackaprompt/playground) |
69 | | | **Prompt Airlines** | Manipulate AI chatbot via prompt injection to score a free airline ticket. | WiZ | CTF | Beginner | Free | [PromptAirlines](https://promptairlines.com/) |
70 | | | **AI CTF** | AI/ML themed challenges to be solved over a 36-hour period. | PHDay | CTF | Beginner, Intermediate | Free | [AI CTF](https://aictf.phdays.fun/) |
71 | | | **Prompt Injection Lab** | An immersive lab focused on gamified AI prompt injection challenges. | ImmersiveLabs | CTF | Beginner | Free | [Prompt Injection Lab](https://prompting.ai.immersivelabs.com/) |
72 | | | **Doublespeak** | A text-based AI escape game designed to practice LLM vulnerabilities. | Forces Unseen | CTF | Beginner | Free | [Doublespeak](https://doublespeak.chat/#/) |
73 | | | **MyLLMBank** | Prompt injection challenges against LLM chat agents that use ReAct to call tools. | WithSecure | CTF | Beginner | Free | [MyLLLBank](https://myllmbank.com/)|
74 | | | **MyLLMDoctor** | Advanced challenge focusing on multi-chain prompt injection. | WithSecure | CTF | Intermediate | Free | [MyLLMDoctor](https://myllmdoc.com/) |
75 | | | **Damn vulnerable LLM agent** | Focuses on Thought/Action/Observation injection | WithSecure | CTF | Intermediate | Free | [Damn vulnerable LLM agent](https://github.com/WithSecureLabs/damn-vulnerable-llm-agent) |
76 | | **Talks** | **AI is just software, what could possible go wrong w/ Rob van der Veer** | The talk explores the dual nature of AI as both a powerful tool and a potential security risk, emphasizing the importance of secure AI development and oversight. | OWASP Lisbon Global AppSec 2024 | Conference | N/A | Free | [YouTube](https://www.youtube.com/watch?v=43cv4f--UU4) |
77 | | | **Lessons Learned from Building & Defending LLM Applications** | Andra Lezza and Javan Rasokat discuss lessons learned in AI security, focusing on vulnerabilities in LLM applications. | DEF CON 32 | Conference | N/A | Free | [YouTube](https://www.youtube.com/watch?v=2-C7xSJ9rhI) |
78 | | | **Practical LLM Security: Takeaways From a Year in the Trenches** | NVIDIA’s AI Red Team shares insights on securing LLM integrations, focusing on identifying risks, common attacks, and effective mitigation strategies. | Black Hat USA 2024 | Conference | N/A | Free | [YouTube](https://www.youtube.com/watch?v=Rhpqiunpu0c) |
79 | | | **Hacking generative AI with PyRIT** | Rajasekar from Microsoft AI Red Team presents PyRIT, a tool for identifying vulnerabilities in generative AI systems, emphasizing the importance of safety and security. | Black Hat USA 2024 | Walkthrough | N/A | Free | [YouTube](https://www.youtube.com/watch?v=M_H8ulTMAe4) |
80 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/media.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Media
3 | excludeSearch: true
4 | ---
5 |
6 | ## Talks
7 |
8 | | Date | Event | Title | Video |
9 | | - | - | - | - |
10 | | 23 Apr 2025 | Dubai AI week | Optimizing AI w/ Rob van der Veer | [Youtube](https://www.youtube.com/watch?v=HbtIccHgrNA) |
11 | | 1 Apr 2025 | SANS AI security summit | AI security made easy w/ Rob van der Veer | [Youtube](https://www.youtube.com/watch?v=iWnYSdj9rxE) |
12 | | 28 Nov 2024 | OWASP Benelux | AI: The New Beginning w/ Rob van der Veer | [Youtube](https://www.youtube.com/watch?v=UrzIHmecXnk) |
13 | | 14 Nov 2024 | Thread Modeling Connect | Threat modeling in the Age of AI w/ Susanna Cox | [Youtube](https://www.youtube.com/watch?v=cTbAD9K_FqA&t=1788s) |
14 | | 13 Nov 2024 | German OWASP day | Overview of OWASP AI Exchange w/ Behnaz Karimi | [Youtube](https://www.youtube.com/watch?v=WLIx5jMyJ_A) |
15 | | 28 Jun 2024 | OWASP Lisbon global appsec | Keynote: AI is just software, what could possible go wrong w/ Rob van der Veer | [Youtube](https://www.youtube.com/watch?v=43cv4f--UU4) |
16 | | 29 Jan 2024 | re:Invent security | AI for CISOs w/ Rob van der Veer | [Youtube](https://www.youtube.com/watch?v=wSSGK2HJPoo/) |
17 | | 5 Jan 2024 | Robust Intelligence | Understanding the AI threat Landscape w/ NIST, MITRE & OWASP | [Youtube](https://www.robustintelligence.com/resource-center/ai-security-understanding-the-threat-landscape) |
18 | | 5 Jan 2024 | Resilient Cyber | Navigating the AI Security Landscape w/ Rob van der Veer | [LinkedIn](https://www.linkedin.com/posts/robvanderveer_in-this-episode-i-sit-down-with-ai-and-software-activity-7147942906060800001-b8RO/) |
19 | | 6 Sep 2023 | The MLSecOps Podcast | A Holistic Approach to Understanding the AI Lifecycle and Securing ML Systems: Protecting AI Through People, Processes & Technology | [Podcast](https://mlsecops.com/podcast/a-holistic-approach-to-understanding-the-ai-lifecycle-and-securing-ml-systems-protecting-ai-through-people-processes-technology) |
20 | | 4 Jul 2023 | Software Improvement Group webinar | AI Security in 13 minutes | [Brighttalk](https://www.brighttalk.com/webcast/19697/586526) |
21 | | 23 Feb 2023 | The Application Security Podcast w/ Chris Romeo and Robert Hurlbut | OWASP AI Security & Privacy Guide w/ Rob van der Veer | [YouTube](https://www.youtube.com/watch?v=SLdn3AwlCAk&) [Podcast](https://www.buzzsprout.com/1730684/12313155-rob-van-der-veer-owasp-ai-security-privacy-guide) |
22 | | 15 Feb 2023 | OWASP Conference Dublin | Attacking And Protecting Artificial Intelligence w/ Rob Van Der Veer | [YouTube](https://www.youtube.com/watch?v=ABmWHnFrMqI) [Slides](https://github.com/OWASP/www-project-ai-security-and-privacy-guide/blob/main/assets/images/20230215-Rob-AIsecurity-Appsec-ForSharing.pdf?raw=true) |
23 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/meetings.md:
--------------------------------------------------------------------------------
1 | ---
2 | # AI Exchange Team Meetings
3 | ---
4 | ## Meetings
5 | - The Authors team meets bi-weekly on **Thursdays 5 PM Amsterdam time**
6 | - Contact us if you want to be a guest - you will be more than welcome
7 | - Previous Meetings can be viewed on the **[YouTube channel](https://youtube.com/@RobvanderVeer-ex3gj?si=s2-gDFrRCazNge_c)**
8 |
9 |
--------------------------------------------------------------------------------
/content/ai_exchange/content/sponsor.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: 'Sponsor the OWASP AI Exchange'
3 | excludeSearch: false
4 | ---
5 |
6 | The OWASP AI Exchange is at the global center of AI security standards and guidelines, serving as the ultimate resource for AI security practitioners AND standard makers. The Exchange offers over 200 pages of practical advice and references on protecting AI and data-centric systems from security threats.
7 |
8 | By sponsoring the OWASP AI Exchange, organizations have a unique opportunity to support and engage with a global community dedicated to advancing AI security understanding and practices.
9 |
10 | Sponsorship benefits include:
11 | - **𝗩𝗶𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆**: Showcase your organization's commitment to AI security within a network of industry leaders and experts.
12 |
13 | - **C𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝗼𝗻**: Participate in the development of global AI security guidelines, standards, and regulations.
14 |
15 | - **𝗧𝗵𝗼𝘂𝗴𝗵𝘁 𝗟𝗲𝗮𝗱𝗲𝗿𝘀𝗵𝗶𝗽**: Contribute to a comprehensive overview of AI threats, risks, mitigations, and controls, positioning your organization at the forefront of AI security discourse.
16 |
17 | To learn more about the sponsorship opportunities and how your organization can get involved, please contact aruneesh.salhotra@owasp.org
18 |
19 | Join us in our mission to protect society from AI security issues by harnessing the collective wisdom of global experts. Together, we can drive the development of secure and trustworthy AI systems.
20 |
21 |
22 |
57 | {{- with $image -}}
58 | 
67 | {{- end -}}
68 |
69 |
70 | {{- with $icon }}{{ partial "utils/icon.html" (dict "name" $icon) -}}{{- end -}}
71 | {{- $title -}}
72 |
73 | {{- with $subtitle -}}
74 | {{- $subtitle | markdownify -}}
75 | {{- end -}}
76 |
77 | {{- /* Strip trailing newline. */ -}}
--------------------------------------------------------------------------------
/content/ai_exchange/layouts/shortcodes/spacer.html:
--------------------------------------------------------------------------------
1 | 
9 | 
13 |
14 | 
74 |
75 | 
12 |
13 | The OWASP AI security & privacy guide consists of two parts:
14 | 1. [How to address AI security](#how-to-address-ai-security): 200+ pages of material presented as the [OWASP AI Exchange website](https://owaspai.org)
15 | 2. [How to address AI privacy](#how-to-address-ai-privacy)
16 |
17 | Artificial Intelligence (AI) is on the rise and so are the concerns regarding AI security and privacy. This guide is a working document to provide clear and actionable insights on designing, creating, testing, and procuring secure and privacy-preserving AI systems.
18 |
19 | See also [this useful recording](https://www.youtube.com/watch?v=ABmWHnFrMqI) or [the slides](https://github.com/OWASP/www-project-ai-security-and-privacy-guide/blob/main/assets/images/20230215-Rob-AIsecurity-Appsec-ForSharing.pdf?raw=true) from [Rob van der Veer's talk](https://sched.co/1F9DT) at the OWASP Global appsec event in Dublin on February 15 2023, during which this guide was launched. And check out the Appsec Podcast episode on this guide ([audio](https://www.buzzsprout.com/1730684/12313155-rob-van-der-veer-owasp-ai-security-privacy-guide),[video](https://www.youtube.com/watch?v=SLdn3AwlCAk&)), or the [September 2023 MLSecops Podcast](https://mlsecops.com/podcast/a-holistic-approach-to-understanding-the-ai-lifecycle-and-securing-ml-systems-protecting-ai-through-people-processes-technology). If you want the short story, check out [the 13 minute AI security quick-talk](https://www.brighttalk.com/webcast/19697/586526).
20 |
21 | 
3 |
4 | The content of the OWASP AI Exchange has moved to a new website with more advanced navigation at [owaspai.org](https://owaspai.org).
5 |
--------------------------------------------------------------------------------
/owaspaiwiki.md:
--------------------------------------------------------------------------------
1 | You have reached this file from a link that has been wrongfully cached. The content you want is [here](owaspaiexchange.md).
2 |
--------------------------------------------------------------------------------
/tab_example.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Example
3 | layout: null
4 | tab: true
5 | order: 1
6 | tags: example-tag
7 | ---
8 |
9 | ## Example
10 |
11 | Put whatever you like here: news, screenshots, features, supporters, or remove this file and don't use tabs at all.
--------------------------------------------------------------------------------