CWE-1434: Insecure Setting of Generative AI/ML Model Inference Parameters
The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.
Last updated
Overview
Generative AI/ML models, such as those used for text generation, image synthesis, and other creative tasks, rely on inference parameters that control model behavior, such as temperature, Top P, and Top K. These parameters affect the model's internal decision-making processes, learning rate, and probability distributions. Incorrect settings can lead to unusual behavior such as text "hallucinations," unrealistic images, or failure to converge during training. The impact of such misconfigurations can compromise the integrity of the application. If the results are used in security-critical operations or decisions, then this could violate the intended security policy, i.e., introduce a vulnerability.
Common consequences
What can happen when CWE-1434 is exploited.
Varies by Context, Unexpected State
Affects: Integrity, Other
The product can generate inaccurate, misleading, or nonsensical information.
Alter Execution Logic, Unexpected State, Varies by Context
Affects: Other
If outputs are used in critical decision-making processes, errors could be propagated to other systems or components.
How it happens
When it is introduced
Typically introduced during these phases of the software lifecycle.
Applies to
Technologies
How to prevent it
Practical mitigations for CWE-1434, grouped by where in the lifecycle they apply.
Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
Implement feedback mechanisms to continuously assess and adjust model performance.
Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.
How to detect it
Automated Dynamic Analysis
Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
Effectiveness: Moderate
Manual Dynamic Analysis
Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
Effectiveness: Moderate
Code examples
Illustrative examples from MITRE showing how the weakness appears in code.
Assume the product offers an LLM-based AI coding assistant to help users to write code as part of an Integrated Development Environment (IDE). Assume the model has been trained on real-world code, and the model behaves normally under its default settings. Suppose there is a default temperature of 1, with a range of temperature values from 0 (most deterministic) to 2.
Consider the following configuration.
Vulnerable example
"model": "my-coding-model",Frequently asked questions
Common questions about CWE-1434.
- What is CWE-1434?
- The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.
- How do you prevent CWE-1434?
- Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
- How is CWE-1434 detected?
- Automated Dynamic Analysis: Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
- What are the consequences of CWE-1434?
- Exploiting CWE-1434 can lead to: Varies by Context, Unexpected State, Alter Execution Logic.
References
- MITRE CWE definition (CWE-1434) (opens in a new tab)
- CWE-1434 vulnerabilities on NVD (opens in a new tab)
- Learn: What is a CWE?
Weakness data is sourced from the MITRE CWE catalog (v4.20). CVE associations are aggregated and kept current by RadicalNotion.AI.
Stay ahead of CWE-1434
Get alerted the moment a new CWE-1434 vulnerability affects your stack, with AI-written analysis, severity context, and remediation guidance.