Benchmark / 02
DuckDB CLI vs MCP for SQL agents
Sample
792observationsEval breakdown
- Exact analytics
- 360 runs · 15 tasks/scenarios
- Paired robustness
- 144 runs · 12 tasks/scenarios
- Repair and safety
- 192 runs · 16 tasks/scenarios
- Repository workflow
- 48 runs · 4 tasks/scenarios
- Stateful SQL analyst
- 48 runs · 4 tasks/scenarios
Interfaces
| Metric | CLI | MCP |
|---|---|---|
| Mean quality | 0.944 | 0.942 |
| Full pass | 75.8% | 77.9% |
| Mean latency | 40.84 s | 36.45 s |
| Mean tokens | 56,920 | 41,214 |
| Mean steps | 7.89 | 7.71 |
| Mean tool calls | 8.22 | 9.61 |
| Protocol rate | 99.7% | 100.0% |
Models
| Metric | deepseek-v4-flash | glm-5.2 | gpt-5.6-luna |
|---|---|---|---|
| Mean quality | 0.942 | 0.934 | 0.953 |
| Full pass | 79.8% | 74.6% | 76.3% |
| Mean latency | 28.57 s | 60.54 s | 26.82 s |
| Mean tokens | 61,947 | 45,193 | 40,059 |
| Mean steps | 7.97 | 8.07 | 7.35 |
| Mean tool calls | 9.83 | 8.69 | 8.21 |
| Protocol rate | 99.8% | 100.0% | 99.7% |
Harnesses
| Metric | OpenCode | Pi |
|---|---|---|
| Mean quality | 0.943 | 0.943 |
| Full pass | 77.2% | 76.6% |
| Mean latency | 38.48 s | 38.82 s |
| Mean tokens | 67,065 | 31,068 |
| Mean steps | 7.94 | 7.65 |
| Mean tool calls | 8.96 | 8.87 |
| Protocol rate | 99.9% | 99.8% |
Setup
- DuckDB CLI
- v1.5.5 (Variegata) d8cdaa33fd
- DuckDB Python
- 1.5.5
- MCP server
- 1.1.0
- OpenCode
- 1.18.16
- Pi
- 0.84.1
- Models
- opencode-go/gpt-5.6-luna · opencode-go/deepseek-v4-flash · opencode-go/glm-5.2
- Bootstrap
- 10,000 cluster resamples
- Captured
- 2026-08-12T08:15:23.171290-04:00