Trace showed every Assistant submit triggered 60+ sequential
GET /api/ea2/flow/{key} requests via agentMemory.recall before
llmClient.chat ran. With a slow EA2 demo, recall took longer
than the test wait so the LLM fetch never started, the agent
reply never rendered, and [llm] telemetry never logged.
Two fixes:
1. agentMemory.listAll(cap) - default 200, recall uses 30.
Edges are returned newest-first by EA2 so the 30-cap keeps the
most recent vault entries and drops the long-tail history that
was pinning the network.
2. routeAgentInput wraps recall in a 2s AbortController. If recall
doesn't finish in 2s, hints[] stays empty and llmFallback runs
immediately. Recall is best-effort context, never a blocker.
Bug: 'tell me a haiku about flowcharts' matched send_chat with
target='me a haiku' / body='flowcharts', producing the bogus
"I don't know who 'me a haiku' is" reply and silently swallowing
all non-tool prompts that started with tell/ask. The LLM fallback
never ran, so [llm] telemetry from wave 3 never fired in tests.
Fix: tighten the matcher to require a single-word [A-Za-z]+ name
between the verb and that/about/:. Drop the substring search in the
directory; require an exact match. Now 'tell me a haiku about X'
falls through to llmFallback as intended.
Investigation: the wave-3 [llm] telemetry warns weren't firing on live
because the fetch had no client-side timeout. When demo's
llm-integration-service was slow/stuck, fetch hung indefinitely past
the test's 12s wait, so the fallback path never ran and the user saw
no agent reply at all.
Add a 12s AbortController-backed deadline. On timeout: console.warn
'[llm] gateway timed out after Xms (deadline 12000ms)' + return the
graceful 'temporarily unavailable' fallback that renders the tool
menu in the chat surface. Uses AbortSignal.any when available to
preserve any caller-supplied signal.
Oracle's wave-2 watch-outs:
1. Wizard config persistence — the follow-up PUT was silently swallowed.
Empirical curl reproducer confirmed config.wizard DOES land after the
split create/PUT (config.wizard.marker=EA2_DRAFT_PROCESS visible on
re-GET). But to satisfy the 'no silent loss' concern: the catch now
warns to console AND surfaces a soft toast 'Draft saved, but its
wizard state didn't attach. You can keep working — save will retry
on Confirm.' so failure is loud without blocking the user.
2. LLM telemetry — fallback was masking infra health. llmClient now
logs every outcome with elapsed time: 503/404/502/504/non-2xx/parse
failures all console.warn with reason + ms. Success path
console.info with provider + content length. Friendly fallback to
the user stays the same; ops/devs see the real story.
3. Catalog hygiene — duplicate 'Laptop Procurement' rows. list_processes
now dedupes by normalized display_name after the existing _key
dedupe. EA2 seeds + tenant imports both publishing the same name
collapse to one row in the assistant output.
4. Mission intuitiveness — user said 'I don't understand anything that's
going on there.' Added a one-line first-visit primer strip
('What you're looking at. Each tab below is a process running in
your company...') dismissible with an X, sticky to localStorage.
Also fixed the stale 'showing snapshot' fallback copy in the
liveError banner (snapshot mode was removed in wave 1; banner now
says 'last known state shown').
30/30 vitest pass. tsc + vite build green.
Three Oracle-flagged gaps:
1. Studio createDraft 500 (was masked by retries). Empirical curl
reproducer proved POST /api/ea2/flow returns 201 with the minimal
shape but intermittently 500s when config.wizard.* is in the
initial body. Split: POST creates the draft minimally, then a
follow-up PUT attaches the wizard config best-effort. Draft
creation now lands reliably.
2. LLM 'temporarily unavailable' was a dead end for the user. Now
the assistant returns ok:true with a concrete menu of tools the
user CAN use right now ('list processes', 'start <name>',
'open <hub>', 'tell <name> that ...', 'remember ...', 'recall ...').
The reply renders as a normal assistant message, not a red error.
3. Topbar UI/UX Pro Max pass:
- Added aria-label to the ⌘K and avatar buttons (icon-only a11y)
- Removed the duplicate topbar-mid chips on Mission (Mission scene
already shows family/defName/version inline). Cleaner topbar.
- Added focus-visible outlines (2px amber, 2px offset) on all
topbar action buttons + user-menu items
- Hover/active scale on the avatar (1.04 / 0.97) with 160ms ease
- Tabular numerals for the ⌘K kbd and notification badge
- prefers-reduced-motion respected
30/30 vitest pass. tsc + vite build green.
POST /api/ea2/flow currently returns 500 after 19s on demo upstream.
Retry once with backoff, then surface a clear 'EA2 backend temporarily
unable to create new processes' message naming the status. The user
gets actionable info instead of a raw 500.
- chatApi.listThreads: catch on the edges query so a fresh empty
inbox doesn't bubble a 500 to the UI. New users see 'No
conversations yet' instead of the error banner.
- agentTools.llmFallback: when the LLM gateway returns 503/500,
show a clear 'plain-language replies temporarily unavailable'
message naming the reason, listing the deterministic capabilities
that still work. No more silent null returns.
If buildLiveScenariosFromApi hangs or errors on any EA2 read,
loginAs would await it forever, leaving the user stuck on
'VERIFYING...' even after auth+identity succeed. Make refreshLive
fire-and-forget after the toast; the Mission scene re-fetches
on mount and surfaces failures via liveError.
CoreDNS at 10.43.0.10 returns SERVFAIL for demo.flow-master.ai in
this environment. Including it in the resolver list caused nginx
to query it first, get SERVFAIL, and return 502 before falling
back to 1.1.1.1. Pinning to public resolvers eliminates the cold
miss. 300s TTL caches the result.
Restored $variable form for proxy_pass because the static-hostname
form failed pod readiness (DNS query at config-load time hits
the same SERVFAIL).
The previous $variable form forced lazy per-request DNS resolution.
Per-request lookups against 10.43.0.10 → SERVFAIL → cold-cache 502
even when downstream was healthy. Switching to a literal hostname
makes nginx resolve once at startup and reuse the cached upstream IP
for the lifetime of the worker. Cloudflare anycast is stable so this
is safe.
Oracle round-9 watch-out: same liveMeta.fetchedFrom leak class
existed in two more user-visible surfaces:
- src/components/Telemetry.tsx: 'source' block rendered the host
string. Now hard-coded 'EA2'.
- src/scenes/Landing.tsx: hero paragraph said 'runs end-to-end
through EA2 on <host>'. Dropped the host suffix entirely.
liveMeta import removed from Telemetry (no other refs). Landing
still uses liveMeta for the workItems/distinctDefs stat counters.
Oracle round-9 third blocker: App.tsx topbar tag rendered
liveMeta.fetchedFrom?.replace('https://', '') which surfaced
'demo.flow-master.ai' in the topbar chip in live mode.
Replaced with the buyer-safe brand label 'live · EA2'. liveMeta
import removed (no other references). Wizard preview pane label
change and CSP recovery from prior commit stay.
When the K3s node recovers from an outage, cluster DNS (CoreDNS at
10.43.0.10) can be briefly unavailable, and nginx's startup-time
resolution of demo.flow-master.ai + canvas-llm-proxy.demo.svc...
fails the conf check, leaving the pod in CrashLoopBackOff. The
hostnames sat in proxy_pass directives that nginx resolves at boot.
Switched to:
- resolver 10.43.0.10 valid=30s ipv6=off;
- set $upstream_demo / $upstream_llm; proxy_pass $variable;
This forces lazy per-request DNS so transient cluster-DNS hiccups
don't kill the canvas-frontend pod. Verified during a real K3s node
outage today.
The seedHeadsIfMissing path I added fired N listMessages calls on
first refresh after auth, blowing idle-poll budget (landing went to
11 reqs at 30s). Reverting to the prior behavior:
- Topbar Chat badge shows real unread when localStorage heads are
populated (i.e. after the user has visited Chat at least once)
- Cold-start (fresh browser / cleared storage) the badge shows 0
until Chat is opened — same accepted caveat Oracle named on
sha-c3515707 and PASS'd
Idle-poll budget protected; Wizard 'Manual review' label change
from prior commit stays.
Close Oracle's two standing caveats so they can't gate a future loop:
1. Wizard dispatch label 'Person' was vague (Oracle: could imply chat
recipient, not work-routing). Now 'Manual review' in both dropdown
and preview-pane badge. dispatch_kind storage value unchanged
('human'/'agent'/'system' for EA2).
2. Chat unread cold-start. Topbar badge depended entirely on
localStorage being populated by Chat scene's first visit. Fresh
browser / private window / cleared storage saw 0 unread until then.
Added seedHeadsIfMissing in App.tsx topbar poll: on first refresh
(and only when localStorage heads count < thread count), fetches
last message per thread ONCE and persists to localStorage. Subsequent
ticks reuse the cache — same idle-poll budget (test still PASS).
Oracle narrow FAIL on sha-a02cc70e: Inspector.tsx:91 still emitted
'Switch to LIVE mode + sign in to execute "..." against
/api/runtime/transactions/{id}/actions/{actionId}.'
Rewritten as 'Switch to live mode and sign in to run "..." on this
case.' Matches the LeftRail and Approvals toast voice.
Source grep for user-visible /api/ in pushToast/title/JSX text returns
zero matches.
Oracle PASS-with-caveats; closing all 3:
1. Wizard dispatch labels were 'human'/'agent'/'system' (impl-ish).
Now 'Person'/'Assistant'/'System' in both the dropdown and the
preview-pane badge.
2. Chat badge was thread COUNT (a buyer with 5 chats sees '5' forever).
Now real unread count, computed locally from localStorage heads +
read maps that the Chat scene already persists. App.tsx topbar
poll still only fires 2 network calls (listThreads + workItems);
per-thread unread math is zero-network.
3. LeftRail toast leaked 'POST /api/runtime/transactions/{id}/actions/
submit' / save_draft. Rewritten as 'Switch to live mode and sign
in to submit this action against EA2.'
Plus engineering-string sweep elsewhere: 'demo.flow-master.ai' /
'bundled JSON' / 'in-browser fetch' references removed from:
- App.tsx mode toggle title
- MissionControl loading spinner
- Landing mode-button title
- state/store snapshot toast
- data/live.ts + data/synthetic.ts tour-step bodies
- buildScenarios.ts tagline
Source-grep confirms no remaining user-visible engineering jargon
referencing the backend hostname or raw endpoints.
The new topbar badge on the Chat tab (chatThreadCount > 0) broke the
old /^\\s*Chat\\s*$/ regex. Switched to a Playwright filter chain:
.tab filtered by hasText 'Chat' AND hasNotText 'Assistant' (which also
contains 'Chat' substring elsewhere in DOM). All 4 QA suites green:
- 36/36 dogfood (chat_sidebar threads=8 previews=8 times=8)
- 12/12 buyer-script (Approvals action disabled? true)
- 8/8 idle-poll (0 reqs at 30s on every scene)
- 14/14 mobile (390px clean)
User brief: 'normal businessman could use this'. The command palette
was already wired but leaked engineering jargon and missed half the
scenes.
- 'Scenes' renamed 'Go to'; now lists all 14 user-reachable scenes
(was 5). Hubs, Documents, Approvals, Chat, Assistant, Explainer,
Geo - all keyboard-reachable.
- Endpoint strings ('POST /api/runtime/transactions', 'demo.flow-
master.ai', 'bundled JSON') removed from hints. Replaced with
buyer-safe equivalents like 'writes to EA2' / 'for developers'.
- Real-actions group: hints simplified, less raw IDs.
- Dropped 'Dispatch sidekick agent · coming soon' (dead stub).
- 'Data mode' group renamed 'Preferences' and simplified to 3
options (refresh live, theme toggle, dev console).
Idle-poll audit caught a regression I introduced in the prior commit:
the topbar badge poll iterated chatApi.listMessages per thread on each
60s tick, firing N requests per refresh. Landing scene showed 9 reqs
in a 30s idle window (failed the <=5 budget).
Now the topbar badge just shows the THREAD COUNT (not per-thread
unread), which is a single chatApi.listThreads call. Per-thread unread
math stays in the Chat scene where it belongs (already wired:
chat-thread-row.unread + .chat-unread-summary).
App.tsx polls every 60s when authed + tenant known:
- chatApi.listThreads + listMessages -> count threads with last message
from someone else after our localStorage read mark
- api.workItems filtered by tenant -> queue count
Badge UI:
- Chat tab shows amber unread pill (99+ cap)
- Approvals tab (new in topbar) shows navy queue-count pill
Also promotes Approvals to a first-class topbar tab (was landing-chip
only). 60s poll is identical to existing live-tick cadence so this
adds no extra polling load.
User brief: 'recreate dev.flow-master.ai Process Creation Wizard'.
Until now the Wizard wrote EA2 correctly but felt like a form. New:
- Stepper shows checkmarks on past steps (was just step numbers).
Step labels rewritten in human language: Describe / Steps / Form
fields / Rules / Review / Done.
- Sticky preview pane on the right shows the live process taking
shape: name + purpose + step chain (numbered + dispatch-kind
badges) + form field list + rules list. Collapses to a single
column at <=1000px.
- Analyze phase: each step gets up/down arrows + remove. New
'Add step' button.
- Generate phase: clearer placeholder ('e.g. requester name'), Yes/No
type label.
- Validate phase: friendlier placeholder.
- Review (Draft saved) phase: replaced the 3-number summary card
with a labelled review-row table that shows process name, purpose,
step chain, field count, rule count.
- Removed all the 'title=POST /api/...' attribute leaks that exposed
endpoint paths on hover.
- Helper functions: moveStep / removeStep / addStep keep node order
intact so the EA2 child-edge chain stays correct.
EA2 write contract unchanged: same 5 phases (Intake / Analyze /
Generate / Validate / Draft saved / Publish), same wizardApi.ops
emit shape, same per-phase apply-batch + governs edge.
When the QA runs against a fresh CEO session (no pre-existing chat
threads), the new-thread create + inbox-edge + message send chain is
4-8 EA2 round-trips. Prior waits (8s thread + 2.5s send + 12s
condition) were occasionally too tight. Bumped to 15s + 4s + 20s.
36/36 dogfood stable across the slow-path.
Oracle round-8 polish caveat: the 503 body still named OPENAI_API_KEY
and ANTHROPIC_API_KEY in the 'hint' field. Buyer-visible in devtools
even though no UI renders it. Replaced with the same buyer-safe copy
the Agent OFF pill uses: 'natural language replies are not enabled in
this environment'.
Image bumped to canvas-llm-proxy:sha-002. Live verified.
Two Oracle round-8 blockers:
1. Approvals action buttons stayed enabled even though the actual
runtime engine submit path is known-500. The runtime-values GET
probe proved EA2 reachability, not action-path readiness — Oracle
rejected 'fail-once-then-disable' for a primary buyer CTA.
Now: actionsEnabled starts FALSE. Approvals shows a 'Read-only
view' note with an explicit 'Enable live actions' checkbox. Action
buttons stay disabled until the operator opts in. If a click then
500s, actionsEnabled flips back off and the toast says 'Live actions
have been turned back off.' No first-click-into-error.
2. Agent OFF hint exposed raw env var names OPENAI_API_KEY /
ANTHROPIC_API_KEY. Replaced with buyer-safe 'Natural language
replies are not enabled in this environment.'
Also removed the runtimeHealth probe entirely (Oracle's nit: the
probe's semantics were misleading — it only proved reachability,
not action-path readiness).
Previous regex was anchored to whole string. Real EA2 display_names
embed the 32-hex case-key as a prefix ('25f89310... #7a9cd1d1'), so
they slipped through. Now any 20-char hex run anywhere in the string
flips the value to machine-shaped.
Buyer-script audit showed queue row titles falling through to raw
case-key hex when business_subject was empty / set to the case key
itself. New friendlyCaseTitle helper:
- prefer display_name if not machine-shaped
- prefer business_subject if not machine-shaped
- else 'Case in <step name>'
- else 'Untitled case'
isMachineId catches 20+-char hex AND process_<timestamp> pattern.
Buyer-script QA caught two regressions:
1. Toast on action failure leaked 'runtime-values isn't reachable' and
'Backend can't accept' — engineering jargon a buyer would read as
'product is broken'.
2. After a real 500, the button stayed enabled so a buyer could click
again into the same error.
Now:
- Failure toast says 'The runtime engine couldn't accept that decision.
Action buttons are now disabled until it recovers.' (no jargon, no
raw error strings).
- handleAction sets runtimeHealth='down' on the failure, which flips
the runtime-note banner on and disables every action button.
- Non-runtime failures get the even-shorter 'Something went wrong
handling that action. Try again in a moment.'
Oracle round-7 watch-out: button title said 'Disabled in this
environment' but disabled prop only blocked when !a.enabled, so a
buyer could still click it and get the 500.
Approvals now probes GET /api/ea2/runtime-values on mount. Treats
HTTP 422 (validation, missing query) and 2xx as 'up' — that proves
the service is reachable. Anything else flips runtimeHealth to
'down', which (a) hides the yellow runtime-note, (b) actually
disables the button, and (c) changes the title attribute to the
honest reason. Removes the misleading '· preview' suffix when the
runtime is fine.
Live probe confirms /api/ea2/runtime-values returns 422 for a bare
GET (validation error), so under normal operation the button stays
enabled. The original 500 came from the runtime engine's internal
cluster DNS to ea2.baseline.svc which is a different code path.
Oracle PASS→FAIL flip identified three blockers; closing all three.
1. Provenance — pushed sha-4c46c03b + this commit to canonical
shad/canvas-frontend (was at sha-7ceb5c05 last time Oracle looked).
2. Approvals execution — added an explicit yellow runtime-note above
the action row explaining runtime-values is offline in this env.
Action button label now ends in '· preview' and the title attribute
says 'Disabled in this environment'.
3. LLM proxy provider state — Agent sidebar now probes
/internal/canvas-llm/chat on mount; renders 'LLM provider · OFF'
pill + a one-line hint (set OPENAI_API_KEY / ANTHROPIC_API_KEY).
Replaces the previous sub-text that buried the off state.
The prior loop tightened Approvals.loadQueue to early-return when
tenantId is null. But the standard 'user reloads page with cached
session token' path didn't populate tenantId — it was only set during
loginAs or the live-fetch path.
Added one effect in App: when isAuthed && !tenantId, call api.ping
which proxies to /api/v1/auth/me and pulls user_id + tenant_id +
email into the store. This unblocks tenant-scoped surfaces
(Approvals) after a page reload.
1. Approvals tenantId hydration race
loadQueue used to fall back to 'show all rows' when tenantId was
null. Now early-returns [] until tenantId hydrates, and the effect
depends on tenantId so it reloads once auth completes.
2. Settings persona quick-switch threw 'Password required'
signIn() in Settings called loginAs(email) without options.method.
Now explicit { method: 'dev' } so the documented dev-login path
is used.
Closes the 'fork Pi coding agent' integration loop with a tiny
broker that mirrors the @earendil-works/pi-ai request shape:
POST /internal/canvas-llm/chat
body: { messages: [{role, content}], max_tokens }
The proxy speaks to OpenAI or Anthropic depending on which env key
is set. With no key, returns 503 so the frontend's deterministic
fallback in routeAgentInput fires gracefully.
- llm-proxy/main.py: FastAPI + httpx (~150 LOC). System messages get
split out for Anthropic (separate 'system' field) and inlined for
OpenAI.
- llm-proxy/Dockerfile: python:3.12-slim, uvicorn on :8080.
- Deployment + Service in demo namespace (verified Running).
- nginx.conf: /internal/canvas-llm/ proxies to the in-cluster service.
Provider keys are set to empty in the manifest by design: customer
flip is a single 'kubectl set env' or sealed-secret update.
New Documents scene at landing chip + scene route. Walks the
curated startable flows, fetches each one's graph, surfaces every
data_definition as a row with its source process. Search filter,
two-column list+detail layout. Mobile collapses to single column
at 700px.
User's brief explicitly banned 'random strings of numbers and letters'.
Settings was leaking a 12-char user_id slice next to 'Current'. Removed.
- Identity card: Signed in / Email / Status (no raw user_id)
- COMMON_EMAILS: replaced 4 dead demo identities with the 3 seeded
personas (CEO/HR/IT) so quick-switch matches what the login page
already advertises
- Sign out button: clears the bearer + sessionStorage and returns to /
- About copy: refreshed; stale gitea repo link removed (the new canon
is shad/canvas-frontend)
canvas-bot
shad