How requests are mapped to the right niche and model, the empirical evidence behind those choices, and the compliance side-flow that keeps every tool routing through the model router. The audit is the product.
Snapshot 2026-10-14 05:00 UTC · auto-refreshed
Head-to-head results that drive niche scores — not vendor claims, actual outputs verified by TAAS.
Code-refactor routing: which model produces valid Python · coding / code_refactor · 2026-05-20
| Model | Result | Detail |
|---|---|---|
| claude-sonnet-4-6 | PASS | 2338 tokens, end_turn, valid Python |
| meta-llama/Llama-3.3-70B-Instruct-Turbo | FAIL | invalid Python (unterminated string) |
| gpt-4o | FAIL | invalid Python (unterminated string) |
| claude-haiku-4-5 | FAIL | invalid Python (unterminated string) |
→ Re-scored the coding niche: claude-sonnet-4-6 ranked #1; the three failing models demoted. code_refactor added as a first-class scored niche.
Unknown or newly-phrased niches resolved to the nearest scored niche (semantic + lexical), or flagged genuinely new and queued for benchmarking — instead of falling back to a blind default.
| Requested | Resolved → niche | Method | Confidence | When |
|---|---|---|---|---|
research_synthesis | research (research) | lexical | 0.817 | 2026-10-14 04:49 |
legal_analysis | NEW — queued for scoring | new_niche | — | 2026-10-14 04:47 |
data_classification | NEW — queued for scoring | new_niche | — | 2026-10-14 04:44 |
Every tool must route LLM/AI calls through the model router. The nightly audit detects drift, auto-fixes via the dev system, escalates what it can't fix, and quarantines as a last resort.
| Run | Scanned | Compliant | Violators | Fixed | Escalated | When |
|---|---|---|---|---|---|---|
| v5_audit_20261014_030029 | 11037 | 10912 | 125 | 0 | 0 | 2026-10-14 03:00 |
| v5_audit_20261013_030020 | 11037 | 10912 | 125 | 0 | 0 | 2026-10-13 03:00 |
| v5_audit_20261012_030022 | 11037 | 10912 | 125 | 0 | 0 | 2026-10-12 03:00 |
| v5_audit_20261011_030018 | 11037 | 10912 | 125 | 0 | 0 | 2026-10-11 03:00 |
| v5_audit_20261010_030018 | 11037 | 10912 | 125 | 0 | 0 | 2026-10-10 03:00 |
| v5_audit_20261009_030027 | 11037 | 10912 | 125 | 0 | 0 | 2026-10-09 03:00 |
| v5_audit_20261008_030644 | 11038 | 10913 | 125 | 0 | 0 | 2026-10-08 03:06 |
| v5_audit_20261007_030021 | 11038 | 10913 | 125 | 0 | 0 | 2026-10-07 03:00 |
| v5_audit_20261005_030022 | 11035 | 10910 | 125 | 0 | 0 | 2026-10-05 03:00 |
| v5_audit_20261004_030029 | 11035 | 10910 | 125 | 0 | 0 | 2026-10-04 03:00 |
| v5_audit_20261003_030018 | 11035 | 10910 | 125 | 0 | 0 | 2026-10-03 03:00 |
| v5_audit_20261002_030041 | 11037 | 10912 | 125 | 0 | 0 | 2026-10-02 03:00 |
| v5_audit_20261001_030012 | 11036 | 10911 | 125 | 0 | 0 | 2026-10-01 03:00 |
| v5_audit_20260930_030021 | 11035 | 10910 | 125 | 0 | 0 | 2026-09-30 03:00 |
| Tool | Outcome | Violations | Attempts | When |
|---|---|---|---|---|
tools.seed_data_classification_niche_register_v1 | needs_escalation | model_literal | 3 | 2026-09-29 03:00 |
tools.failure_triage_data_classification_v1 | needs_escalation | model_literal | 3 | 2026-09-27 03:01 |
tools.seed_data_classification_niche_v2 | needs_escalation | model_literal | 3 | 2026-09-26 03:01 |
tools.register_data_classification_niche_v1 | needs_escalation | model_literal | 3 | 2026-09-24 03:01 |
pm.decompose_program | needs_escalation | model_literal | 3 | 2026-09-23 03:00 |
uda.select_framework_v1 | needs_escalation | model_literal | 3 | 2026-09-22 03:03 |
uda.generated_d9a4bac72f8e | needs_escalation | model_literal | 3 | 2026-09-22 03:03 |
uda.generated_4334a904a0fd | needs_escalation | model_literal | 3 | 2026-09-22 03:03 |
tools.seed_data_classification_niche_v1 | needs_escalation | model_literal | 3 | 2026-09-22 03:02 |
subscription.credential_pool_v1 | needs_escalation | rest_openai | 3 | 2026-09-22 03:02 |
stub.threed.terrain_relief | needs_escalation | rest_fal | 3 | 2026-09-22 03:02 |
uda.select_framework_v1 | needs_escalation | model_literal | 3 | 2026-09-22 03:02 |
uda.generated_d9a4bac72f8e | needs_escalation | model_literal | 3 | 2026-09-22 03:01 |
uda.generated_4334a904a0fd | needs_escalation | model_literal | 3 | 2026-09-22 03:01 |
uda.code_generation_v1 | needs_escalation | model_literal | 3 | 2026-09-22 03:01 |
tools.seed_data_classification_niche_v1 | needs_escalation | model_literal | 3 | 2026-09-22 03:01 |
subscription.credential_pool_v1 | needs_escalation | rest_openai | 3 | 2026-09-22 03:00 |
stub.threed.terrain_relief | needs_escalation | rest_fal | 3 | 2026-09-22 03:00 |
pm.decompose_program | needs_escalation | model_literal | 3 | 2026-09-22 03:00 |
uda.code_generation_v1 | needs_escalation | model_literal | 3 | 2026-09-21 03:04 |