← Niche Catalog
python_code_generation
Modality: llm_chat · full deep dive — every ranked model, test result, and artifact.
Ranked Models
Ranked by confidence-adjusted score (single/zero-sample, non-curated scores floored; curated empirical scores trusted as-is).
| # | Model | Provider | Adj. Score | Raw | Evidence |
|---|
| 1 | claude-sonnet-4-6 | claude_desktop | 0.773 | 0.850 | n=50 |
| 2 | gpt-5.5::think=fast | codex | 0.525 | 0.963 | n=6 |
| 3 | gpt-5.5::think=deep | codex | 0.525 | 0.963 | n=6 |
| 4 | claude-opus-4-8::think=fast | claude_code | 0.430 | 0.788 | n=6 |
| 5 | gpt-5.3-codex-spark::think=deep | codex | 0.382 | 0.700 | n=6 |
| 6 | claude-opus-4-8::think=deep | claude_code | 0.382 | 0.700 | n=6 |
| 7 | deepseek-reasoner | deepseek | 0.356 | 0.800 | n=4 |
| 8 | grok-3-mini | xai | 0.356 | 0.800 | n=4 |
| 9 | claude-opus-4-6 | anthropic | 0.356 | 0.800 | n=4 |
| 10 | grok-4.20-multi-agent-0309 | xai | 0.356 | 0.800 | n=4 |
| 11 | gemini-2.5-pro | google_gemini | 0.346 | 0.778 | n=4 |
| 12 | gemini-2.5-flash | google_gemini | 0.338 | 0.760 | n=4 |
| 13 | gpt-4o | openai | 0.336 | 0.756 | n=4 |
| 14 | meta-llama/Llama-3.3-70B-Instruct-Turbo | together | 0.336 | 0.756 | n=4 |
| 15 | deepseek-ai/DeepSeek-R1 | together | 0.335 | 0.753 | n=4 |
| 16 | sonar | perplexity | 0.335 | 0.753 | n=4 |
| 17 | sonar-pro | perplexity | 0.335 | 0.753 | n=4 |
| 18 | grok-3 | xai | 0.335 | 0.753 | n=4 |
| 19 | gpt-4.1-mini | openai | 0.335 | 0.753 | n=4 |
| 20 | command-r-08-2024 | cohere | 0.335 | 0.753 | n=4 |
| 21 | Qwen/Qwen2.5-7B-Instruct-Turbo | together | 0.335 | 0.753 | n=4 |
| 22 | gpt-4o-mini | openai | 0.335 | 0.753 | n=4 |
| 23 | deepseek-ai/DeepSeek-V3.1 | together | 0.335 | 0.753 | n=4 |
| 24 | gpt-5.4::think=fast | codex | 0.334 | 0.612 | n=6 |
| 25 | gpt-5.4::think=deep | codex | 0.334 | 0.612 | n=6 |
| 26 | codex-auto-review | codex | 0.286 | 0.525 | n=6 |
| 27 | gpt-5.3-codex-spark::think=fast | codex | 0.286 | 0.525 | n=6 |
| 28 | magistral-small-latest | mistral | 0.259 | 0.583 | n=4 |
| 29 | mistral-large-latest | mistral | 0.185 | 0.417 | n=4 |
| 30 | claude-sonnet-4-6 | anthropic | 0.120 | 0.803 | provisional (n≤1) |
| 31 | claude-haiku-4-5-20251001 | anthropic | 0.120 | 0.803 | provisional (n≤1) |
| 32 | claude-sonnet-4-5-20250929 | claude_code | 0.117 | 0.778 | provisional (n≤1) |
| 33 | claude-opus-4-7 | anthropic | 0.113 | 0.753 | provisional (n≤1) |
| 34 | grok-4 | xai | 0.113 | 0.753 | provisional (n≤1) |
| 35 | gpt-4.1 | openai | 0.113 | 0.753 | provisional (n≤1) |
| 36 | gpt-5.1 | openai | 0.113 | 0.753 | provisional (n≤1) |
| 37 | o4-mini | openai | 0.113 | 0.753 | provisional (n≤1) |
| 38 | gpt-5-nano | openai | 0.113 | 0.753 | provisional (n≤1) |
| 39 | claude-sonnet-4-20250514 | claude_code | 0.107 | 0.714 | provisional (n≤1) |
| 40 | claude-opus-4-1-20250805 | claude_code | 0.103 | 0.690 | provisional (n≤1) |
| 41 | claude-opus-4-20250514 | claude_code | 0.088 | 0.586 | provisional (n≤1) |
| 42 | claude-opus-4-5-20251101 | claude_code | 0.086 | 0.574 | provisional (n≤1) |
| 43 | claude-opus-4-8 | claude_code | 0.063 | 0.422 | provisional (n≤1) |
| 44 | deepseek-chat | deepseek | 0.000 | 0.000 | n=4 |
| 45 | mistral-small-latest | mistral | 0.000 | 0.000 | n=4 |
| 46 | gpt-5 | openai | 0.000 | 0.000 | provisional (n≤1) |
| 47 | gpt-5-mini | openai | 0.000 | 0.000 | provisional (n≤1) |
| 48 | gpt-5.4 | codex | 0.000 | 0.000 | n=50 |
| 49 | gpt-5.5 | codex | 0.000 | 0.000 | provisional (n≤1) |
| 50 | gpt-5.3-codex-spark | codex | 0.000 | 0.000 | n=50 |
Test Results
Showing top 25 of 342 benchmark outputs by weighted score.
claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0 · $0.000107 · 709ms
```python
def add(a, b):
return a + b
```
claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000238 · 633ms
```python
class UserProfile:
def __init__(self):
self.email = None
def update_email(self, new_email):
self.email = new_email
```
mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 899ms
```python
def add(a, b):
return a + b
```
claude-opus-4-6 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.00327 · 1714ms
```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0 · $0.000107 · 881ms
```python
def add(a, b):
return a + b
```
gpt-4.1-mini · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1556ms
```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
claude-sonnet-4-6 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1420ms
```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $2.9e-05 · 939ms
```python
class UserProfile:
def __init__(self, email):
self.email = email
def update_email(self, new_email):
self.email = new_email
```
deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0 · $8e-06 · 872ms
```python
def add(a, b):
return a + b
```
gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $6.6e-05 · 863ms
```python
def add(a, b):
return a + b
```
sonar · perplexity · weighted 10.0 · syntactic_validity=10.0 · $4.5e-05 · 6096ms
```python
def add(a, b):
return a + b
```[1][3][5]
mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 896ms
```python
def add(a, b):
return a + b
```
gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 894ms
```python
def add(a, b):
return a + b
```
deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0 · $8e-06 · 787ms
```python
def add(a, b):
return a + b
```
deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $1.2e-05 · 979ms
```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 745ms
```python
def add(a, b):
return a + b
```
gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $6.6e-05 · 633ms
```python
def add(a, b):
return a + b
```
command-r-08-2024 · cohere · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 769ms
```python
def add(a, b):
return a + b
```
mistral-large-latest · mistral · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 819ms
```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
command-r-08-2024 · cohere · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 963ms
```python
def add(a, b):
return a + b
```
grok-3 · xai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000396 · 1812ms
class UserProfile:
def update_email(self, new_email):
self.email = new_email
mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1790ms
```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
gpt-4o · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000485 · 744ms
```python
class UserProfile:
def __init__(self, email):
self.email = email
def update_email(self, new_email):
self.email = new_email
```
sonar · perplexity · weighted 10.0 · syntactic_validity=10.0 · $4.3e-05 · 2045ms
```python
def add(a, b):
return a + b
```[3][5]
grok-3-mini · xai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 2937ms
def add(a, b):
return a + b