test: track LLM gateway 5xx separately + lock cascade integrity
Oracle wave-4 watch-outs: - '5xx count: 1 should not be hidden' - 'Add one future regression check that fails if both primary and fallback LLM paths fail, since the fallback tunnel is now part of the live behavior' E2E now reports llm-gateway 5xx and system 5xx as two distinct counts. The canvas-primary 503 is acknowledged + counted; only system 5xx must be zero. New no_system_5xx assertion guards that. wave4 adds gap1_cascade_did_not_exhaust: fails if telemetry shows '[llm] all gateways exhausted' on the live assistant path. Locks in that at least one gateway (primary or fallback-dev) is returning real model output.
This commit is contained in:
@@ -53,6 +53,18 @@ console.log("\n=== Gap 1: Real LLM-backed answer ===");
|
||||
if (gatewayOk && !fallback && last.length > 30) pass("gap1_real_llm_answer", `${gatewayOk.text.match(/provider=\w+/)?.[0]} · ${last.length}ch`);
|
||||
else fail("gap1_real_llm_answer", `fallback=${fallback}, gateway_ok=${!!gatewayOk}, len=${last.length}`);
|
||||
await p.screenshot({ path: `${OUT}/g1-llm-answer.png`, fullPage: false });
|
||||
|
||||
// Oracle wave-4 action #5: regression check — if both primary and
|
||||
// fallback gateways fail, the assistant must NOT produce a real reply.
|
||||
// Equivalent: a real LLM reply implies at least one gateway succeeded.
|
||||
// We assert the inverse by inspecting telemetry: a successful 'gap1'
|
||||
// run produces exactly one "ok" telemetry line; the cascade-exhausted
|
||||
// failure mode emits "all gateways exhausted". If we ever see both
|
||||
// primary AND fallback-dev fail (cascade exhausted) AND a real reply,
|
||||
// the test fails because the contract is broken.
|
||||
const cascadeExhausted = msgs.some(m => /\[llm\] all gateways exhausted/.test(m.text));
|
||||
if (!cascadeExhausted) pass("gap1_cascade_did_not_exhaust");
|
||||
else fail("gap1_cascade_did_not_exhaust", "both primary and fallback-dev failed");
|
||||
await p.context().close();
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user