qa(evidence): live production verification report
Captured against https://canvas.flow-master.ai with browser-style probes + Playwright dogfood + raw fetches: - Every /api/* through canvas's nginx returns 502 (proxy broken). - demo.flow-master.ai upstream is alive (200/401 direct). - Frontend calls auth routes that return 404 even direct on demo: /api/v1/auth/dev-login-config, /api/v1/auth/microsoft/login. - User's 'GETs all 200' hypothesis was wrong — they're 502s. - Zero EA2 traffic reaches the backend. Repros in qa/evidence/_*.mjs. Screenshots + JSON dumps + full report in qa/evidence/.
This commit is contained in:
canvas-bot
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
@@ -0,0 +1,102 @@
|
||||
# Canvas — production verification report
|
||||
|
||||
**Date:** 2026-06-14
|
||||
**Target:** https://canvas.flow-master.ai
|
||||
**Method:** browser-style probes + Playwright dogfood, no kubectl access.
|
||||
|
||||
## TL;DR
|
||||
|
||||
The live canvas frontend is healthy as a static SPA, but it is **not connected to EA2**. Every `/api/*` call through canvas's nginx proxy returns **502 Bad Gateway**. The user's hypothesis that "GETs are all 200, it should be wired" is wrong — those are 502s, not 200s. Plus the frontend calls auth routes that don't exist on demo (`/api/v1/auth/dev-login-config`, `/api/v1/auth/microsoft/login`), which is a real product bug independent of the proxy issue.
|
||||
|
||||
## Evidence
|
||||
|
||||
### 1. canvas /api/* → 502 (proxy broken)
|
||||
|
||||
```
|
||||
GET /api/v1/auth/dev-login-config 502 Bad Gateway
|
||||
POST /api/v1/auth/dev-login 502 Bad Gateway
|
||||
GET /api/v1/auth/me 502 Bad Gateway
|
||||
GET /api/v1/auth/microsoft/login 502 Bad Gateway
|
||||
GET /api/ea2/work-items 502 Bad Gateway
|
||||
GET /api/ea2/flow/processes?limit=10 502 Bad Gateway
|
||||
GET /api/ea2/process-definitions 502 Bad Gateway
|
||||
GET /api/runtime/transactions 502 Bad Gateway
|
||||
```
|
||||
|
||||
Body of every 502:
|
||||
```
|
||||
<html>
|
||||
<head><title>502 Bad Gateway</title></head>
|
||||
<body><center><h1>502 Bad Gateway</h1></center><hr><center>nginx</center></body>
|
||||
</html>
|
||||
```
|
||||
|
||||
Raw artifacts: `qa/evidence/endpoint-probe.json`, `qa/evidence/canvas-proxy-ea2-error.html`.
|
||||
|
||||
### 2. demo upstream is alive
|
||||
|
||||
```
|
||||
GET https://demo.flow-master.ai/ 200
|
||||
GET https://demo.flow-master.ai/api/ea2/work-items 401 (auth required, as expected)
|
||||
```
|
||||
|
||||
So upstream is reachable from the public internet. The 502 is canvas's nginx-side proxy failing — TLS handshake, NetworkPolicy, or DNS resolution inside the pod.
|
||||
|
||||
### 3. Frontend calls routes that don't exist on demo
|
||||
|
||||
```
|
||||
GET https://demo.flow-master.ai/api/v1/auth/dev-login-config → 404
|
||||
GET https://demo.flow-master.ai/api/v1/auth/microsoft/login → 404
|
||||
POST https://demo.flow-master.ai/api/v1/auth/dev-login → unverified (proxy blocks the probe)
|
||||
```
|
||||
|
||||
Even after the proxy is fixed, the Login scene will still fail because the auth endpoints aren't on demo. Either:
|
||||
- the frontend is targeting the wrong path prefix (should be `/api/auth/...`?), or
|
||||
- demo isn't the real auth backend for canvas and a different host should be proxied.
|
||||
|
||||
### 4. Playwright dogfood against live canvas
|
||||
|
||||
`qa/full_dogfood.mjs` first 5 checks:
|
||||
|
||||
```
|
||||
PASS auth_guard_unauthed_redirects_to_login
|
||||
PASS login_shows_three_personas — Mariana Cole · CEO | Aisha Khan · HR | Rohan Patel · IT
|
||||
PASS login_has_microsoft_sso_button
|
||||
FAIL ceo_persona_dev_login_lands_on_landing
|
||||
FAIL landing_exposes_all_hubs_and_extras
|
||||
missing: Procurement Hub, People Hub, IT Hub, Attendance Map,
|
||||
Command Assistant, Team Chat, What is FlowMaster?
|
||||
```
|
||||
|
||||
The login screen renders correctly. Dev-login click never lands because the dev-login endpoint is 502.
|
||||
|
||||
### 5. Idle network during a 6-second landing visit
|
||||
|
||||
```
|
||||
total /api/ calls: 1
|
||||
by status: { "502": 1 }
|
||||
endpoint: /api/v1/auth/microsoft (SSO availability probe)
|
||||
```
|
||||
|
||||
The "non-stop 200 GETs" the user saw in the console were either: (a) the probe + retry loop (1 call every few seconds, every one a 502, not 200), or (b) developer-tools showing the static-asset 200s and the user not differentiating from the failing /api calls. Either way, **zero EA2 traffic** is reaching the backend.
|
||||
|
||||
## Root cause
|
||||
|
||||
Canvas's running nginx pod is at the build whose `last-modified` is `Sun, 14 Jun 2026 17:02:30 GMT` — pre-dating today's ops PRs. The image bumps in `FM06/flowmaster-ops` (#1173, #1174, #1175) did not roll the pod, because:
|
||||
- `FM06/flowmaster-ops` `manifests/overlays/demo/mc-mission-control` points at `gitea.flow-master.ai/shad/mission-control-demo` images, but the live CSP signature (`script-src 'self'`, openstreetmap, unpkg) does not match the `mc-mission-control` nginx.conf. So the host `canvas.flow-master.ai` is **not** being served by `mc-mission-control` at all.
|
||||
- The actual serving deployment is not in `flowmaster-ops` and not in any of the cloned source repos. Either it's out-of-Gitops on the cluster or in a private repo not exposed.
|
||||
- The 502s on `/api/*` mean whichever nginx is serving canvas has a broken `proxy_pass` to demo — wrong cluster DNS, missing NetworkPolicy egress, expired upstream TLS, or wrong upstream host.
|
||||
|
||||
## What needs cluster access to fix
|
||||
|
||||
1. Identify the deployment+pod actually serving `canvas.flow-master.ai`.
|
||||
2. Verify its nginx.conf `proxy_pass` target and resolve why `proxy_pass https://demo.flow-master.ai` returns 502 from inside the pod.
|
||||
3. Confirm `NetworkPolicy` allows egress to `demo.flow-master.ai`'s IP from the pod's namespace.
|
||||
4. Identify the correct auth API base path (`/api/v1/auth/...` does not exist on demo) and reconfigure the frontend or the upstream.
|
||||
|
||||
Until those four are resolved on the cluster, no amount of repo-side change will make canvas talk to EA2.
|
||||
|
||||
## What did get fixed this session
|
||||
|
||||
- `shad/canvas-frontend` README rewritten to match the shipped product (PR #1 merged).
|
||||
- Findings above filed as `qa/evidence/` artifacts for the next operator with cluster access to pick up.
|
||||
@@ -0,0 +1,94 @@
|
||||
// Post-login EA2 trace + screenshot bundle. Uses dev-login (the SSO 502 is
|
||||
// the user's actual issue) and walks Mission Control so polling fires.
|
||||
import { chromium } from "playwright";
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
|
||||
const URL = "https://canvas.flow-master.ai/";
|
||||
const OUT = "/tmp/canvas-evidence";
|
||||
mkdirSync(OUT, { recursive: true });
|
||||
|
||||
const calls = [];
|
||||
const b = await chromium.launch({ headless: true });
|
||||
const ctx = await b.newContext({ viewport: { width: 1440, height: 900 } });
|
||||
const p = await ctx.newPage();
|
||||
|
||||
p.on("response", async (res) => {
|
||||
const u = res.url();
|
||||
if (!u.includes("/api/")) return;
|
||||
calls.push({ url: u, status: res.status(), ts: Date.now() });
|
||||
});
|
||||
|
||||
console.log("[1/5] goto landing");
|
||||
await p.goto(URL, { waitUntil: "domcontentloaded" });
|
||||
await p.waitForTimeout(2000);
|
||||
await p.screenshot({ path: `${OUT}/01-login.png`, fullPage: false });
|
||||
|
||||
console.log("[2/5] try dev-login as CEO persona");
|
||||
const ceoChip = p.locator(".persona-chip").filter({ hasText: /Mariana/ });
|
||||
const ceoVisible = await ceoChip.count();
|
||||
console.log(` CEO persona chip count: ${ceoVisible}`);
|
||||
if (ceoVisible) {
|
||||
await ceoChip.first().click();
|
||||
await p.waitForTimeout(3000);
|
||||
}
|
||||
await p.screenshot({ path: `${OUT}/02-after-login.png`, fullPage: false });
|
||||
|
||||
console.log("[3/5] check URL + page title");
|
||||
console.log(` url: ${p.url()}`);
|
||||
console.log(` title: ${await p.title()}`);
|
||||
const headers = await p.evaluate(() => ({
|
||||
body: document.body?.innerText?.slice(0, 200) ?? "",
|
||||
scenes: document.querySelector("[class*='shell']")?.className ?? "n/a",
|
||||
}));
|
||||
console.log(` scene: ${headers.scenes}`);
|
||||
console.log(` body[0..200]: ${JSON.stringify(headers.body)}`);
|
||||
|
||||
console.log("[4/5] try to reach Mission Control");
|
||||
const missionLink = p.locator("a, button").filter({ hasText: /mission control/i }).first();
|
||||
if (await missionLink.count()) {
|
||||
await missionLink.click();
|
||||
await p.waitForTimeout(4000);
|
||||
}
|
||||
await p.screenshot({ path: `${OUT}/03-mission.png`, fullPage: false });
|
||||
|
||||
console.log("[5/5] dump network");
|
||||
const byStatus = {};
|
||||
const byHost = {};
|
||||
const byPath = {};
|
||||
for (const c of calls) {
|
||||
byStatus[c.status] = (byStatus[c.status] || 0) + 1;
|
||||
const url = new URL(c.url);
|
||||
byHost[url.host] = (byHost[url.host] || 0) + 1;
|
||||
const path = url.pathname.split("/").slice(0, 4).join("/");
|
||||
byPath[path] = (byPath[path] || 0) + 1;
|
||||
}
|
||||
|
||||
const report = {
|
||||
ts: new Date().toISOString(),
|
||||
url: URL,
|
||||
total: calls.length,
|
||||
byStatus,
|
||||
byHost,
|
||||
byPath,
|
||||
ea2Calls: calls.filter((c) => c.url.includes("/api/ea2/")).length,
|
||||
ea2Ok: calls.filter((c) => c.url.includes("/api/ea2/") && c.status === 200).length,
|
||||
authCalls: calls.filter((c) => c.url.includes("/api/v1/auth/")).length,
|
||||
authOk: calls.filter((c) => c.url.includes("/api/v1/auth/") && c.status === 200).length,
|
||||
first10: calls.slice(0, 10),
|
||||
last10: calls.slice(-10),
|
||||
};
|
||||
|
||||
writeFileSync(`${OUT}/network-report.json`, JSON.stringify(report, null, 2));
|
||||
|
||||
console.log("\n=== SUMMARY ===");
|
||||
console.log(`total /api/ calls: ${report.total}`);
|
||||
console.log(`by status: ${JSON.stringify(report.byStatus)}`);
|
||||
console.log(`by path (top):`);
|
||||
for (const [pa, n] of Object.entries(byPath).sort((a, b) => b[1] - a[1]).slice(0, 8)) {
|
||||
console.log(` ${n}\t${pa}`);
|
||||
}
|
||||
console.log(`EA2 200/total: ${report.ea2Ok}/${report.ea2Calls}`);
|
||||
console.log(`auth 200/total: ${report.authOk}/${report.authCalls}`);
|
||||
console.log(`screenshots → ${OUT}`);
|
||||
|
||||
await b.close();
|
||||
@@ -0,0 +1,48 @@
|
||||
// Browser-style trace: visit live canvas, capture every network request, and
|
||||
// classify whether EA2 endpoints are actually being hit with real responses.
|
||||
import { chromium } from "playwright";
|
||||
|
||||
const URL = "https://canvas.flow-master.ai/";
|
||||
const calls = [];
|
||||
|
||||
const b = await chromium.launch({ headless: true });
|
||||
const ctx = await b.newContext({ viewport: { width: 1440, height: 900 } });
|
||||
const p = await ctx.newPage();
|
||||
|
||||
p.on("response", async (res) => {
|
||||
const u = res.url();
|
||||
if (!u.includes("/api/")) return;
|
||||
calls.push({
|
||||
url: u,
|
||||
status: res.status(),
|
||||
ts: Date.now(),
|
||||
});
|
||||
});
|
||||
|
||||
await p.goto(URL, { waitUntil: "domcontentloaded" });
|
||||
await p.waitForTimeout(8000);
|
||||
|
||||
const total = calls.length;
|
||||
const byStatus = {};
|
||||
const byEndpoint = {};
|
||||
for (const c of calls) {
|
||||
byStatus[c.status] = (byStatus[c.status] || 0) + 1;
|
||||
const path = c.url.replace(/https?:\/\/[^/]+/, "").split("?")[0];
|
||||
const bucket = path.split("/").slice(0, 5).join("/");
|
||||
byEndpoint[bucket] = (byEndpoint[bucket] || 0) + 1;
|
||||
}
|
||||
|
||||
console.log("\n=== EA2 wiring trace ===");
|
||||
console.log("total /api/ calls in 8s:", total);
|
||||
console.log("by status:", JSON.stringify(byStatus));
|
||||
console.log("by endpoint:");
|
||||
for (const [ep, n] of Object.entries(byEndpoint).sort((a, b) => b[1] - a[1])) {
|
||||
console.log(` ${n}\t${ep}`);
|
||||
}
|
||||
|
||||
const ea2 = calls.filter((c) => c.url.includes("/api/ea2/"));
|
||||
const auth = calls.filter((c) => c.url.includes("/api/v1/auth/"));
|
||||
console.log(`\nEA2 calls: ${ea2.length} (${ea2.filter((c) => c.status === 200).length} 200)`);
|
||||
console.log(`auth calls: ${auth.length} (${auth.filter((c) => c.status === 200).length} 200)`);
|
||||
|
||||
await b.close();
|
||||
@@ -0,0 +1,44 @@
|
||||
// Direct curl-style probe through the live nginx of every endpoint canvas's
|
||||
// frontend code calls. Goal: prove which ones return real EA2 JSON vs 502.
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
|
||||
const OUT = "/tmp/canvas-evidence";
|
||||
mkdirSync(OUT, { recursive: true });
|
||||
|
||||
const ROOT = "https://canvas.flow-master.ai";
|
||||
const endpoints = [
|
||||
["GET", "/api/v1/auth/dev-login-config", null],
|
||||
["POST", "/api/v1/auth/dev-login", { email: "dev@flow-master.ai" }],
|
||||
["GET", "/api/v1/auth/me", null],
|
||||
["GET", "/api/v1/auth/microsoft/login", null],
|
||||
["GET", "/api/ea2/work-items", null],
|
||||
["GET", "/api/ea2/flow/processes?limit=10", null],
|
||||
["GET", "/api/ea2/process-definitions", null],
|
||||
["GET", "/api/runtime/transactions", null],
|
||||
];
|
||||
|
||||
const out = [];
|
||||
let token = null;
|
||||
|
||||
for (const [method, path, body] of endpoints) {
|
||||
const headers = { "content-type": "application/json" };
|
||||
if (token) headers["authorization"] = `Bearer ${token}`;
|
||||
const init = { method, headers };
|
||||
if (body) init.body = JSON.stringify(body);
|
||||
try {
|
||||
const res = await fetch(`${ROOT}${path}`, init);
|
||||
const text = await res.text();
|
||||
const sample = text.slice(0, 200).replace(/\s+/g, " ");
|
||||
let json = null;
|
||||
try { json = JSON.parse(text); } catch { /* not json */ }
|
||||
if (path === "/api/v1/auth/dev-login" && json?.access_token) token = json.access_token;
|
||||
out.push({ method, path, status: res.status, len: text.length, sample, ctype: res.headers.get("content-type") });
|
||||
console.log(`${method.padEnd(4)} ${path.padEnd(46)} ${res.status} ${sample.slice(0, 80)}`);
|
||||
} catch (e) {
|
||||
out.push({ method, path, error: e.message });
|
||||
console.log(`${method.padEnd(4)} ${path.padEnd(46)} ERR ${e.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
writeFileSync(`${OUT}/endpoint-probe.json`, JSON.stringify(out, null, 2));
|
||||
console.log(`\n→ ${OUT}/endpoint-probe.json`);
|
||||
@@ -0,0 +1 @@
|
||||
{"error":{"code":401,"message":"Missing authentication token","request_id":"unknown"}}
|
||||
@@ -0,0 +1,66 @@
|
||||
[
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/auth/dev-login-config",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
},
|
||||
{
|
||||
"method": "POST",
|
||||
"path": "/api/v1/auth/dev-login",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/auth/me",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/v1/auth/microsoft/login",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/ea2/work-items",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/ea2/flow/processes?limit=10",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/ea2/process-definitions",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
},
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/api/runtime/transactions",
|
||||
"status": 502,
|
||||
"len": 150,
|
||||
"sample": "<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ",
|
||||
"ctype": "text/html"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"ts": "2026-06-14T21:18:38.314Z",
|
||||
"url": "https://canvas.flow-master.ai/",
|
||||
"total": 0,
|
||||
"byStatus": {},
|
||||
"byHost": {},
|
||||
"byPath": {},
|
||||
"ea2Calls": 0,
|
||||
"ea2Ok": 0,
|
||||
"authCalls": 0,
|
||||
"authOk": 0,
|
||||
"first10": [],
|
||||
"last10": []
|
||||
}
|
||||
Reference in New Issue
Block a user