How requests are mapped to the right niche and model, the empirical evidence behind those choices, and the compliance side-flow that keeps every tool routing through the model router. The audit is the product.
Snapshot 2026-07-12 05:04 UTC · auto-refreshed
Head-to-head results that drive niche scores — not vendor claims, actual outputs verified by TAAS.
Code-refactor routing: which model produces valid Python · coding / code_refactor · 2026-05-20
| Model | Result | Detail |
|---|---|---|
| claude-sonnet-4-6 | PASS | 2338 tokens, end_turn, valid Python |
| meta-llama/Llama-3.3-70B-Instruct-Turbo | FAIL | invalid Python (unterminated string) |
| gpt-4o | FAIL | invalid Python (unterminated string) |
| claude-haiku-4-5 | FAIL | invalid Python (unterminated string) |
→ Re-scored the coding niche: claude-sonnet-4-6 ranked #1; the three failing models demoted. code_refactor added as a first-class scored niche.
Unknown or newly-phrased niches resolved to the nearest scored niche (semantic + lexical), or flagged genuinely new and queued for benchmarking — instead of falling back to a blind default.
| Requested | Resolved → niche | Method | Confidence | When |
|---|---|---|---|---|
research_synthesis | research (research) | lexical | 0.817 | 2026-07-12 04:42 |
legal_analysis | NEW — queued for scoring | new_niche | — | 2026-07-12 04:16 |
Every tool must route LLM/AI calls through the model router. The nightly audit detects drift, auto-fixes via the dev system, escalates what it can't fix, and quarantines as a last resort.
| Run | Scanned | Compliant | Violators | Fixed | Escalated | When |
|---|---|---|---|---|---|---|
| v5_audit_20260609_030205 | 7726 | 7607 | 119 | 2 | 0 | 2026-06-09 03:02 |
| v5_audit_20260608_030008 | 7634 | 7517 | 117 | 0 | 0 | 2026-06-08 03:00 |
| v5_audit_20260607_030208 | 7586 | 7468 | 118 | 0 | 1 | 2026-06-07 03:02 |
| v5_audit_20260606_030209 | 7514 | 7396 | 118 | 0 | 2 | 2026-06-06 03:02 |
| v5_audit_20260605_030247 | 7459 | 7319 | 140 | 24 | 1 | 2026-06-05 03:02 |
| v5_audit_20260601_030006 | 6912 | 6815 | 97 | 2 | 0 | 2026-06-01 03:00 |
| v5_audit_20260531_030205 | 6911 | 6791 | 120 | 25 | 0 | 2026-05-31 03:02 |
| v5_audit_20260530_030006 | 6728 | 6721 | 7 | 4 | 0 | 2026-05-30 03:00 |
| v5_audit_20260529_030212 | 6704 | 6692 | 12 | 9 | 3 | 2026-05-29 03:02 |
| v5_audit_20260528_032041 | 6646 | 6646 | 0 | 0 | 0 | 2026-05-28 03:20 |
| v5_audit_20260528_031508 | 6646 | 6646 | 0 | 0 | 0 | 2026-05-28 03:15 |
| v5_audit_20260527_030755 | 6516 | 6516 | 0 | 0 | 0 | 2026-05-27 03:07 |
| v5_audit_20260526_030714 | 6374 | 6374 | 0 | 0 | 0 | 2026-05-26 03:07 |
| v5_audit_20260525_030611 | 6374 | 6374 | 0 | 0 | 0 | 2026-05-25 03:06 |
| Tool | Outcome | Violations | Attempts | When |
|---|---|---|---|---|
generated.mistral_api_v1 | needs_escalation | model_literal | 3 | 2026-07-03 03:00 |
case_assistant.rag_ingest_v1 | fixed | 1 | 2026-07-02 03:00 | |
generated.mistral_api_v1 | needs_escalation | model_literal | 3 | 2026-06-29 03:32 |
generated.mistral_api_v1 | needs_escalation | model_literal | 3 | 2026-06-24 07:52 |
gemini.capability_verify_v1 | fixed | 1 | 2026-06-24 07:52 | |
gemini.capability_verify_v1 | fixed | 1 | 2026-06-24 07:51 | |
gemini.capability_verify_v1 | fixed | 1 | 2026-06-24 07:51 | |
gemini.capability_verify_v1 | fixed | 1 | 2026-06-24 07:51 | |
fal.threed.generate_v1 | fixed | 1 | 2026-06-23 03:01 | |
fal.threed.cinematic_pipeline_v1 | fixed | 1 | 2026-06-21 03:00 | |
education.vision_portal_runner_v1 | fixed | 1 | 2026-06-20 03:00 | |
agent.auth_tiered_router | fixed | 1 | 2026-06-19 03:00 | |
fal.threed.character_animation_v1 | fixed | 1 | 2026-06-16 03:01 | |
fal.image.generate_v1 | fixed | 1 | 2026-06-15 03:00 | |
fal.capability_verify_v1 | fixed | 1 | 2026-06-14 03:00 | |
anthropic.catalog_ingest_v1 | fixed | 1 | 2026-06-13 03:00 | |
account.capability_harvest_complete_v1 | fixed | 1 | 2026-06-12 03:00 | |
anthropic.capability_verify_v1 | fixed | 1 | 2026-06-11 03:00 | |
anthropic.capability_verify_v1 | needs_escalation | rest_anthropic | 3 | 2026-06-10 03:00 |
trellison.agi_audit.benchmark_niche_models | fixed | 2 | 2026-06-09 03:02 |