TAAS
Overview Niche Catalog
Generated Objects Routing & Audit ← Niche Catalog
python_code_generation Modality: llm_chat · full deep dive — every ranked model, test result, and artifact.
Ranked Models Ranked by confidence-adjusted score (single/zero-sample, non-curated scores floored; curated empirical scores trusted as-is).
# Model Provider Adj. Score Raw Evidence 1 deepseek-reasoner deepseek 0.356 0.800 n=4 2 grok-3-mini xai 0.356 0.800 n=4 3 claude-opus-4-6 anthropic 0.356 0.800 n=4 4 gemini-2.5-pro google_gemini 0.346 0.778 n=4 5 gemini-2.5-flash google_gemini 0.338 0.760 n=4 6 gpt-4o openai 0.336 0.756 n=4 7 meta-llama/Llama-3.3-70B-Instruct-Turbo together 0.336 0.756 n=4 8 deepseek-ai/DeepSeek-R1 together 0.335 0.753 n=4 9 sonar perplexity 0.335 0.753 n=4 10 sonar-pro perplexity 0.335 0.753 n=4 11 grok-3 xai 0.335 0.753 n=4 12 gpt-4.1-mini openai 0.335 0.753 n=4 13 command-r-08-2024 cohere 0.335 0.753 n=4 14 Qwen/Qwen2.5-7B-Instruct-Turbo together 0.335 0.753 n=4 15 gpt-4o-mini openai 0.335 0.753 n=4 16 deepseek-ai/DeepSeek-V3.1 together 0.335 0.753 n=4 17 magistral-small-latest mistral 0.259 0.583 n=4 18 mistral-large-latest mistral 0.185 0.417 n=4 19 claude-sonnet-4-6 anthropic 0.120 0.803 provisional (n≤1) 20 claude-haiku-4-5-20251001 anthropic 0.120 0.803 provisional (n≤1) 21 claude-opus-4-7 anthropic 0.113 0.753 provisional (n≤1) 22 grok-4 xai 0.113 0.753 provisional (n≤1) 23 gpt-4.1 openai 0.113 0.753 provisional (n≤1) 24 gpt-5.1 openai 0.113 0.753 provisional (n≤1) 25 o4-mini openai 0.113 0.753 provisional (n≤1) 26 gpt-5-nano openai 0.113 0.753 provisional (n≤1) 27 deepseek-chat deepseek 0.000 0.000 n=4 28 mistral-small-latest mistral 0.000 0.000 n=4 29 gpt-5 openai 0.000 0.000 provisional (n≤1) 30 gpt-5-mini openai 0.000 0.000 provisional (n≤1)
Test Results Showing top 40 of 342 benchmark outputs by weighted score.
deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0 · $8e-06 · 787ms ```python
def add(a, b):
return a + b
```
claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000238 · 633ms ```python
class UserProfile:
def __init__(self):
self.email = None
def update_email(self, new_email):
self.email = new_email
```
mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 899ms ```python
def add(a, b):
return a + b
```
deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $1.2e-05 · 979ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
gpt-4o · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000485 · 744ms ```python
class UserProfile:
def __init__(self, email):
self.email = email
def update_email(self, new_email):
self.email = new_email
```
gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $6.6e-05 · 633ms ```python
def add(a, b):
return a + b
```
Qwen/Qwen2.5-7B-Instruct-Turbo · together · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 657ms ```python
class UserProfile:
def __init__(self, email):
self.email = email
def update_email(self, new_email):
self.email = new_email
```
This class includes an initializer to set the initial email and a method to update the email.
gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $6.6e-05 · 684ms ```python
def add(a, b):
return a + b
```
gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $6.6e-05 · 863ms ```python
def add(a, b):
return a + b
```
command-r-08-2024 · cohere · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 963ms ```python
def add(a, b):
return a + b
```
sonar · perplexity · weighted 10.0 · syntactic_validity=10.0 · $4.5e-05 · 6096ms ```python
def add(a, b):
return a + b
```[1][3][5]
mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 896ms ```python
def add(a, b):
return a + b
```
gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 894ms ```python
def add(a, b):
return a + b
```
gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 745ms ```python
def add(a, b):
return a + b
```
command-r-08-2024 · cohere · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 671ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
claude-sonnet-4-6 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1420ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
mistral-large-latest · mistral · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 745ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
command-r-08-2024 · cohere · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 769ms ```python
def add(a, b):
return a + b
```
deepseek-reasoner · deepseek · weighted 10.0 · syntactic_validity=10.0 · $0.000123 · 1284ms ```python
def add(a, b):
return a + b
```
grok-3 · xai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000396 · 1812ms class UserProfile:
def update_email(self, new_email):
self.email = new_email
grok-3-mini · xai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $2.3e-05 · 4812ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
deepseek-chat · deepseek · weighted 10.0 · syntactic_validity=10.0 · $8e-06 · 872ms ```python
def add(a, b):
return a + b
```
sonar-pro · perplexity · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000513 · 2247ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```[2][5]
claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0 · $0.000107 · 881ms ```python
def add(a, b):
return a + b
```
mistral-large-latest · mistral · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 819ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
gpt-4.1-mini · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1556ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
gpt-4o-mini · openai · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $2.9e-05 · 939ms ```python
class UserProfile:
def __init__(self, email):
self.email = email
def update_email(self, new_email):
self.email = new_email
```
gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000205 · 668ms ```python
class UserProfile:
def __init__(self, username, email):
self.username = username
self.email = email
def update_email(self, new_email):
self.email = new_email
```
claude-opus-4-6 · anthropic · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.00327 · 1714ms
```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
meta-llama/Llama-3.3-70B-Instruct-Turbo · together · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1254ms ```python
class UserProfile:
def __init__(self, email):
self.email = email
def update_email(self, new_email):
self.email = new_email
```
deepseek-ai/DeepSeek-V3.1 · together · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1402ms ```python
class UserProfile:
def __init__(self):
self.email = None
def update_email(self, new_email):
self.email = new_email
```
gemini-2.5-flash · google_gemini · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.000205 · 910ms ```python
class UserProfile:
def __init__(self, username, email):
self.username = username
self.email = email
def update_email(self, new_email):
self.email = new_email
```
claude-haiku-4-5-20251001 · anthropic · weighted 10.0 · syntactic_validity=10.0 · $0.000107 · 709ms ```python
def add(a, b):
return a + b
```
deepseek-reasoner · deepseek · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.001169 · 7214ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
gemini-2.5-pro · google_gemini · weighted 10.0 · syntactic_validity=10.0 · $0.004649 · 6438ms ```python
def add(a, b):
return a + b
```
claude-sonnet-4-6 · anthropic · weighted 10.0 · syntactic_validity=10.0 · $0.0 · 1479ms ```python
def add(a, b):
return a + b
```
sonar · perplexity · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $5.4e-05 · 2068ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
mistral-small-latest · mistral · weighted 10.0 · syntactic_validity=10.0, naming_conventions=10.0 · $0.0 · 1790ms ```python
class UserProfile:
def update_email(self, new_email):
self.email = new_email
```
sonar · perplexity · weighted 10.0 · syntactic_validity=10.0 · $4.3e-05 · 2045ms ```python
def add(a, b):
return a + b
```[3][5]
grok-3-mini · xai · weighted 10.0 · syntactic_validity=10.0 · $1.4e-05 · 2937ms def add(a, b):
return a + b