qcbench leaderboard
qcbench scores LLM-written OpenQASM circuits the only way that can't be fooled: simulate them and check a declared physical invariant. "Verified" = simulator-confirmed. Arms are inference-time strategies measured with qicode. Dataset: verified-openqasm-circuits. Submit results via GitHub issue.
claude-opus (CLI)entry 52 · n=1best 100% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (baseline) | 94.4% | 95.4% | 18 |
| verify & retry (harness) | 100.0% | 99.3% | 18 |
claude-sonnet (CLI)entry 54 · n=1best 100% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (control) | 94.4% | 93.9% | 18 |
| retrieval-augmented (augmented) | 100.0% | 99.3% | 18 |
claude-sonnet-4.6 (Bedrock)entry 51+53 · 3 seedsbest 87% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (baseline) | 50.0% | 54.2% | 54 |
| few-shot (fewshot) | 75.9% | 88.1% | 54 |
| verify & retry (harness) | 83.3% | 92.1% | 54 |
| few-shot + retry (fewshot-harness) | 87.0% | 91.7% | 54 |
qwen2.5-coder-14b + LoRA-100it (local MLX)entry 58 · n=1best 44% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (control) | 22.2% | 32.0% | 18 |
| retrieval-augmented (augmented) | 38.9% | 51.6% | 18 |
| verify & retry (harness) | 22.2% | 37.0% | 18 |
| retrieval + retry (augmented-harness) | 44.4% | 54.4% | 18 |
qwen2.5-coder-14b (local MLX)entry 56 · n=1best 39% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (control) | 22.2% | 34.4% | 18 |
| retrieval-augmented (augmented) | 33.3% | 52.7% | 18 |
| verify & retry (harness) | 22.2% | 40.9% | 18 |
| retrieval + retry (augmented-harness) | 38.9% | 58.2% | 18 |
qwen2.5-coder-14b base (extended-22)entry 60+61 · extended suite (22 problems), n=1best 36% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (baseline) | 31.8% | 40.4% | 22 |
| verify & retry (harness) | 36.4% | 45.5% | 22 |
qwen2.5-coder-14b + LoRA-r3-trajectories (extended-22)entry 61 · extended suite (22 problems), n=1best 36% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (baseline) | 31.8% | 40.3% | 22 |
| verify & retry (harness) | 36.4% | 45.9% | 22 |
qwen2.5-coder-14b + LoRA-r2-200it (extended-22)entry 60 · extended suite (22 problems), n=1best 32% verified
| intervention | verified | mean score | n |
|---|
| plain 1-shot (baseline) | 31.8% | 37.3% | 22 |
Built by Qly — a quantum computing IDE. n=1 rows are directional; see the research log in the repo for caveats.