---
title: "FreeCAD AI Benchmark: Can LLMs Fix Real CAD Models?"
description: "A CAD repair task is a concrete test of spatial reasoning, scripting and verification. Here is what the FreeCAD benchmark measures and how to read it."
updated: 2026-11-08
url: https://lforla.org/blog/freecad-benchmark
---
CAD is a good antidote to vague AI benchmarks. A model either produces a model that opens, respects the constraints and passes the checks, or it does not. The [FreeCAD benchmark](/benchmarks/freecad-fix) uses that concreteness to test agents on a real engineering task.

## The task

The benchmark starts from a broken or incomplete FreeCAD model and asks the agent to repair it. The agent can inspect the model, reason about the geometry, write Python or use the FreeCAD API, and iterate. The result is scored by running the repaired model through a set of checks.

That makes it an **agentic** benchmark: the model needs to plan, use tools, observe errors and recover. A single answer is rarely enough.

## What it measures

- **spatial and geometric reasoning;**
- **API and scripting knowledge;**
- **debugging from error messages;**
- **constraint satisfaction;**
- **persistence across iterations.**

These are transferable skills for engineering agents, not just CAD.

## How scoring works

The exact rubric is on the benchmark page, but the principles are:

1. The repaired model must open without errors.
2. Required dimensions and constraints must be satisfied.
3. The result is checked programmatically, not by a judge reading a description.
4. The agent's tool budget and time limit are fixed.

A model that writes a convincing explanation but produces a broken model scores zero. That is the point.

## Why CAD is hard for LLMs

- Geometry is unforgiving: a small constraint error breaks the model.
- The API has many version-specific details.
- Error messages are terse and sometimes misleading.
- The agent must maintain state across attempts.
- Visual feedback is limited unless the harness renders the model.

This is why the benchmark is a useful stress test for agents that claim to "use tools" and "recover from errors".

## How to read the results

Look at:

- the pass rate on held-out models;
- the number of attempts and tool calls;
- the cost per solved task;
- the failure modes (opening, constraints, dimensions, scripting).

A high pass rate with a huge tool budget is not the same as a high pass rate with a tight budget. Compare both.

## How to run your own CAD benchmark

Start with 10 to 20 small models that you can share. Define the checks before you look at model outputs. Fix the FreeCAD version, the tool budget and the time limit. Run each model at least three times. Publish the starting models and the checker.

The [how to create an AI benchmark](/blog/how-to-create-an-ai-benchmark) guide covers the general method.

## FAQ

**What is the FreeCAD benchmark?** An agentic benchmark where a model must repair a broken FreeCAD model and pass programmatic checks.

**Do LLMs need vision for CAD?** Not always. Many tasks can be solved through the API and error messages, but rendering helps with spatial errors. The benchmark page describes the current setup.

**Is this a test of FreeCAD knowledge or general reasoning?** Both. The task rewards API knowledge, but also planning, debugging and persistence.

**Can I submit my own CAD task?** The benchmark page explains the current task and how results are produced. Check it for submission details.