test: track LLM gateway 5xx separately + lock cascade integrity
build-and-publish / test (push) Has been cancelled
build-and-publish / image (push) Has been cancelled

Oracle wave-4 watch-outs:
- '5xx count: 1 should not be hidden'
- 'Add one future regression check that fails if both primary and
   fallback LLM paths fail, since the fallback tunnel is now part
   of the live behavior'

E2E now reports llm-gateway 5xx and system 5xx as two distinct
counts. The canvas-primary 503 is acknowledged + counted; only
system 5xx must be zero. New no_system_5xx assertion guards that.

wave4 adds gap1_cascade_did_not_exhaust: fails if telemetry shows
'[llm] all gateways exhausted' on the live assistant path. Locks
in that at least one gateway (primary or fallback-dev) is
returning real model output.
This commit is contained in:
canvas-bot
2026-06-15 22:31:05 +04:00
parent e29667b348
commit 99203267aa
2 changed files with 22 additions and 3 deletions
+12
View File
@@ -53,6 +53,18 @@ console.log("\n=== Gap 1: Real LLM-backed answer ===");
if (gatewayOk && !fallback && last.length > 30) pass("gap1_real_llm_answer", `${gatewayOk.text.match(/provider=\w+/)?.[0]} · ${last.length}ch`);
else fail("gap1_real_llm_answer", `fallback=${fallback}, gateway_ok=${!!gatewayOk}, len=${last.length}`);
await p.screenshot({ path: `${OUT}/g1-llm-answer.png`, fullPage: false });
// Oracle wave-4 action #5: regression check — if both primary and
// fallback gateways fail, the assistant must NOT produce a real reply.
// Equivalent: a real LLM reply implies at least one gateway succeeded.
// We assert the inverse by inspecting telemetry: a successful 'gap1'
// run produces exactly one "ok" telemetry line; the cascade-exhausted
// failure mode emits "all gateways exhausted". If we ever see both
// primary AND fallback-dev fail (cascade exhausted) AND a real reply,
// the test fails because the contract is broken.
const cascadeExhausted = msgs.some(m => /\[llm\] all gateways exhausted/.test(m.text));
if (!cascadeExhausted) pass("gap1_cascade_did_not_exhaust");
else fail("gap1_cascade_did_not_exhaust", "both primary and fallback-dev failed");
await p.context().close();
}