Model Format Vulnerability Form beta

What's a Model File Vulnerability?

A model format vulnerability refers to a security flaw that arises from the way an AI/ML model is stored or serialized in a specific file format. Exploiting these flaws can lead to real-world impacts, such as unauthorized model manipulation or malicious code execution.

Currently, we have identified two broad categories of model format vulnerabilities: Deserialization and Backdoors.

  • Deserialization vulnerabilities occur when improperly handled serialized data allows attackers to inject malicious payloads during model loading.
  • Backdoors involve the intentional embedding of hidden malicious functionality within the model itself.

These categories are not exhaustive and we are constantly on the lookout for new threat vectors. If you have discovered a vulnerability that doesn't fall within these categories, please submit it anyway.

What are we looking for?

High-Value Targets

Supported Formats: .safetensors, .gguf, .keras, .joblib

  • Arbitrary Code Execution (ACE) through a file format vulnerability
  • Backdoors or output manipulation triggered by malicious model files
  • Unique methods to bypass our existing model scanning tools

Medium to Low Value Targets

Supported Formats: .pkl, other small or less-common formats

  • Zip/Tar-based directory traversal (Zip/TarSlip)
  • Library-specific code execution not coming directly from the model file load or inference
  • Denial of Service (DoS) attacks through malformed model files

What We're Looking For

  • Attacks that cause code execution at model load time by manipulating headers, metadata, or custom operators.
  • Embedded backdoors within model architectures that alter inference outputs under specific conditions.
  • Techniques that trick or evade our automated scanners, allowing malicious model files to go undetected.
  • Vulnerabilities in model file parsing leading to memory corruption (heap overflows, integer overflows, etc.).
  • Exploits via custom layers, operators, or code embedded in model formats that execute during model loading.

Submission Requirements

  • Provide a proof-of-concept (PoC) model file uploaded to a public HuggingFace repository and reproduction steps.
  • Clearly explain the vulnerability, how you've created the PoC model file, affected file format, and conditions required to trigger it.
  • Demonstrate the security impact (e.g., code execution, silent output manipulation, scanner bypass).
Examples

Here are some examples of model file vulnerabilities we've identified internally:

These examples were created internally to demonstrate the types of vulnerabilities we're interested in. Your submission could be similar to these or could uncover entirely new types of vulnerabilities in model formats.

Notes and Guidelines
  • Rewards: As we're hashing out a systematic approach to price model format vulnerabilities, the final determination for the bounty amounts will be subjective and vary based on the severity and impact of the vulnerability. For vulnerabilities in pickle files, rewards are up to $1,500, and for all other formats it's up to $3,000.
  • Beta Label: As this is a beta program, we are currently unable to confirm CVE assignments or public disclosure timelines.
  • Review Timeline: We aim to review all submissions within 45 days of receipt (as usual).
  • Scope: All model formats are in scope. If we're missing any, please reach out.
  • Who can participate: All huntr users (new and existing) are welcome.
  • Resources: For inspiration and guidance please refer to Protect AI's Knowledge Base and the ModelScan Repository. New content will be added regularly.
Your expertise is vital in enhancing the security landscape of AI/ML models. Thank you for contributing to this important initiative. Good luck ☘️

For more information read our participation guidelines.

Please log in to continue. By logging in you agree to our terms of service.