coding

Modality: llm_chat · full deep dive — every ranked model, test result, and artifact.

Models

342

Benchmark Results

Media Artifacts

Resolutions

Ranked Models

Ranked by confidence-adjusted score (single/zero-sample, non-curated scores floored; curated empirical scores trusted as-is).

#	Model	Provider	Adj. Score	Raw	Evidence
1	deepseek-chat	deepseek	0.996	0.997	n=12836
2	claude-sonnet-4-6	anthropic	0.920	0.920	curated
3	claude-opus-4-6	anthropic	0.840	0.840	curated
4	claude-sonnet-4-6	claude_desktop	0.773	0.850	n=50
5	deepseek-ai/DeepSeek-R1	together	0.753	0.753	curated
6	command-r-08-2024	cohere	0.753	0.753	curated
7	Qwen/Qwen2.5-7B-Instruct-Turbo	together	0.753	0.753	curated
8	deepseek-ai/DeepSeek-V3.1	together	0.753	0.753	curated
9	claude-haiku-4-5-20251001	anthropic	0.734	0.734	n=16760
10	deepseek-reasoner	deepseek	0.700	0.700	curated
11	magistral-small-latest	mistral	0.583	0.583	curated
12	claude-sonnet-4-6::think=fast	claude_code	0.545	1.000	n=6
13	claude-sonnet-4-6::think=deep	claude_code	0.545	1.000	n=6
14	gpt-5.5::think=fast	codex	0.525	0.963	n=6
15	gpt-5.5::think=deep	codex	0.525	0.963	n=6
16	claude-opus-4-7::think=fast	claude_code	0.477	0.875	n=6
17	claude-opus-4-7::think=deep	claude_code	0.477	0.875	n=6
18	claude-opus-4-8::think=fast	claude_code	0.430	0.788	n=6
19	mistral-large-latest	mistral	0.417	0.417	curated
20	claude-opus-4-8::think=deep	claude_code	0.382	0.700	n=6
21	gpt-5.3-codex-spark::think=deep	codex	0.382	0.700	n=6
22	claude-sonnet-4-5-20250929	anthropic	0.363	0.363	n=12137
23	gpt-4.1-mini	openai	0.344	0.344	curated
24	gpt-4o-mini	openai	0.344	0.344	curated
25	gpt-5.4::think=fast	codex	0.334	0.612	n=6
26	gpt-5.4::think=deep	codex	0.334	0.612	n=6
27	codex-auto-review	codex	0.286	0.525	n=6
28	gpt-5.3-codex-spark::think=fast	codex	0.286	0.525	n=6
29	grok-4.20-multi-agent-0309	xai	0.178	0.400	n=4
30	sonar	perplexity	0.167	0.377	n=4
31	sonar-pro	perplexity	0.167	0.377	n=4
32	grok-3	xai	0.167	0.377	n=4
33	gpt-4o	openai	0.150	0.150	curated
34	meta-llama/Llama-3.3-70B-Instruct-Turbo	together	0.150	0.150	curated
35	gemini-2.5-flash	google_gemini	0.142	0.319	n=4
36	gemini-3.1-pro	gemini	0.120	0.800	provisional (n≤1)
37	antigravity:gemini-2.5-pro	antigravity	0.117	0.780	provisional (n≤1)
38	gemini-2.5-pro	google_gemini	0.109	0.109	curated
39	antigravity:gemini-2.5-flash	antigravity	0.105	0.700	provisional (n≤1)
40	antigravity:gemini-2.0-flash-thinking	antigravity	0.105	0.700	provisional (n≤1)
41	o3	openai	0.000	0.000	provisional (n≤1)
42	o3-mini	openai	0.000	0.000	provisional (n≤1)
43	gpt-5.4	codex	0.000	0.000	n=50
44	claude-sonnet-4-6	claude_code	0.000	0.000	n=50
45	claude-opus-4-7	claude_code	0.000	0.000	n=50
46	gpt-5.5	codex	0.000	0.000	n=50
47	claude-opus-4-8	claude_code	0.000	0.000	n=50
48	claude-opus-4-6	claude_code	0.000	0.000	n=50
49	claude-opus-4-5-20251101	claude_code	0.000	0.000	n=50
50	claude-sonnet-4-5-20250929	claude_code	0.000	0.000	n=50
51	claude-opus-4-1-20250805	claude_code	0.000	0.000	n=50
52	claude-opus-4-20250514	claude_code	0.000	0.000	n=50
53	claude-sonnet-4-20250514	claude_code	0.000	0.000	n=50
54	gpt-5.3-codex-spark	codex	0.000	0.000	n=50
55	copilot:gpt-4o	github_copilot	0.000	0.000	provisional (n≤1)
56	copilot:gpt-4o-mini	github_copilot	0.000	0.000	provisional (n≤1)
57	copilot:claude-3.5-sonnet	github_copilot	0.000	0.000	provisional (n≤1)
58	copilot:claude-3.7-sonnet	github_copilot	0.000	0.000	provisional (n≤1)
59	copilot:gemini-1.5-pro	github_copilot	0.000	0.000	provisional (n≤1)
60	copilot:gemini-2.0-flash	github_copilot	0.000	0.000	provisional (n≤1)
61	copilot:o1	github_copilot	0.000	0.000	provisional (n≤1)
62	copilot:o3-mini	github_copilot	0.000	0.000	provisional (n≤1)

Test Results

Showing top 25 of 342 benchmark outputs by weighted score.

claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0 · $0.000107 · 709ms

```python def add(a, b): return a + b ```

claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000238 · 633ms

```python class UserProfile: def __init__(self): self.email = None def update_email(self, new_email): self.email = new_email ```

mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 899ms

```python def add(a, b): return a + b ```

claude-opus-4-6 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.00327 · 1714ms

```python class UserProfile: def update_email(self, new_email): self.email = new_email ```

claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0 · $0.000107 · 881ms

```python def add(a, b): return a + b ```

gpt-4.1-mini · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1556ms

```python class UserProfile: def update_email(self, new_email): self.email = new_email ```

claude-sonnet-4-6 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1420ms

```python class UserProfile: def update_email(self, new_email): self.email = new_email ```

gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $2.9e-05 · 939ms

```python class UserProfile: def __init__(self, email): self.email = email def update_email(self, new_email): self.email = new_email ```

deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0 · $8e-06 · 872ms

```python def add(a, b): return a + b ```

gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $6.6e-05 · 863ms

```python def add(a, b): return a + b ```

sonar · perplexity · weighted 10.0 · syntactic_validity=10.0 · $4.5e-05 · 6096ms

```python def add(a, b): return a + b ```[1][3][5]

mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 896ms

```python def add(a, b): return a + b ```

gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 894ms

```python def add(a, b): return a + b ```

deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0 · $8e-06 · 787ms

```python def add(a, b): return a + b ```

deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $1.2e-05 · 979ms

```python class UserProfile: def update_email(self, new_email): self.email = new_email ```

gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 745ms

```python def add(a, b): return a + b ```

gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $6.6e-05 · 633ms

```python def add(a, b): return a + b ```

command-r-08-2024 · cohere · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 769ms

```python def add(a, b): return a + b ```

mistral-large-latest · mistral · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 819ms

```python class UserProfile: def update_email(self, new_email): self.email = new_email ```

command-r-08-2024 · cohere · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 963ms

```python def add(a, b): return a + b ```

grok-3 · xai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000396 · 1812ms

class UserProfile: def update_email(self, new_email): self.email = new_email

mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1790ms

```python class UserProfile: def update_email(self, new_email): self.email = new_email ```

gpt-4o · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000485 · 744ms

```python class UserProfile: def __init__(self, email): self.email = email def update_email(self, new_email): self.email = new_email ```

sonar · perplexity · weighted 10.0 · syntactic_validity=10.0 · $4.3e-05 · 2045ms

```python def add(a, b): return a + b ```[3][5]

grok-3-mini · xai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 2937ms

def add(a, b): return a + b

Resolved Here

Requests semantically routed into this niche.

Requested	Method	Confidence	When
`code_writing`	lexical	0.99	2026-05-20 17:25
`code_refactor`	lexical	0.99	2026-05-20 17:25
`code_refactor`	lexical	0.99	2026-05-20 17:24