---
title: "Reverse Engineering Benchmarks for LLMs: A Practical Guide"
description: "What reverse engineering benchmarks test, how they score LLMs and agents, and why binary analysis is a good stress test for reasoning."
updated: 2026-11-07
url: https://lforla.org/blog/reverse-engineering-benchmarks
---
Reverse engineering is a useful stress test for language models. The task is open-ended, the feedback is concrete, and a plausible-sounding answer is easy to produce but easy to disprove. That combination makes it a good benchmark for agents that need to reason about real systems.

The live results are on the [reverse engineering benchmark page](/benchmarks/reverse-engineering).

## What the benchmark tests

A reverse engineering benchmark can target several skills:

- reading assembly and decompiled code;
- inferring a file format or protocol from samples;
- identifying a vulnerability or a hidden check;
- reconstructing logic from observed inputs and outputs;
- writing a working script or patch.

The strongest benchmarks score an **artefact**, not an explanation. A model that writes a parser that works is more convincing than a model that describes a parser.

## How scoring works

Common metrics:

- **exact match** for a recovered value or flag;
- **pass rate** for a script that must run on held-out inputs;
- **functional similarity** for a patch or reconstruction;
- **time and tool budget** for agentic runs.

A benchmark that only checks the final answer can reward guessing. A benchmark that runs the artefact on unseen samples is harder to game.

## Why binary analysis is hard for LLMs

- **Long context:** binaries and decompiled output are large.
- **Low tolerance for hallucination:** a wrong offset or type breaks the result.
- **Tool use:** the model often needs a disassembler, a debugger or a sandbox.
- **Feedback loops:** the agent must run, observe, and revise.
- **Security:** executing untrusted binaries requires isolation.

These are the same properties that make reverse engineering a good proxy for real agent work: planning, tool use, verification and recovery from failure.

## How to read the results

Do not compare a raw percentage across benchmark versions. Check:

- the task family (format inference, vulnerability, patching);
- the tool budget and time limit;
- whether the score is pass@1 or pass@k;
- whether the artefact was executed on held-out inputs;
- the model's cost per solved task.

A model with a lower score but a much lower cost can be the better production choice.

## How to run your own

1. Collect 20 to 50 small, legally shareable binaries or protocols.
2. Write a deterministic checker that runs the artefact.
3. Fix the tool budget and time limit.
4. Run each model three times and report variance.
5. Publish the prompts and the checker.

The [how to create an AI benchmark](/blog/how-to-create-an-ai-benchmark) guide covers the general method. For agent evaluation, add tool-call success and recovery metrics.

## FAQ

**What is a reverse engineering benchmark?** A task set where a model or agent must recover logic, a format or a vulnerability from code, binaries or samples, scored on a working artefact.

**Can LLMs reverse engineer binaries?** They can help with decompiled code, pattern recognition and scripting, but results depend heavily on tooling and task size. Treat it as an agentic task, not a single prompt.

**Why is pass@k reported?** Because a model may solve a task only after several attempts. Pass@k measures that, but it also changes the cost comparison.

**Is reverse engineering benchmark data public?** On LFORLA, the benchmark page describes the task and links to the results. Check the page for the current task and run details.