Vision-AI Domain (Master)

Consolidated reusable-domain spec for the vision-AI service, cherry-picked from three feature branches.

27 min read diagramsUpdated 2026-05-25docs/architecture/vision-ai-domain-master.md

Status: Consolidated design — cherry-picked from three feature branches into one reusable-domain spec. Source branches (unmerged on main as of 2026-05-25):

origin/claude/surgical-equipment-counter-ai-LU0GW — device screen reader + medication safety/planner/verify (LLM-driven)

origin/claude/surgical-ai-yolo-pharmacy-XB3Sx — pharmacy/blood-bank/specimen/sterilization/wound/wristband (multi-vertical verify)

origin/claude/nutrition-ai-meal-analysis-SLbto — nutrition LLM + device-vision API + robotics integration + module catalog

Already on main: services/vision/ (surgicalCount only) + surgical_count_scans migration

This doc supersedes the branch-local summaries (vision-ai-complete-summary.md, vision-ai-module-catalog.md, vision-ai-robotics-integration.md, device-screen-reader.md, medication-ai-planner.md, medication-safety-agent.md) — those remain as deep-dive references; this one is the bounded-context contract + integration map.

1. TL;DR

The vision-AI domain wraps camera-derived clinical inference as a first-class platform capability. It composes two inference primitives (YOLO object detection + multimodal LLM reasoning), persists a uniform *_scans / *_verifications / *_verdicts audit trail, emits typed manifest.* events into the existing encounter-orchestrator, and surfaces results through medical-kit AiAssist components plus a device-facing REST surface for autonomous carts and robot dogs.

Layer	What’s on main today	What the three branches add
Vision service	1 module (surgicalCount), 1 adapter (YOLO stub→onnx→remote)	10 more modules, 2nd adapter (LLM), and 1 composed adapter (OCR+LLM)
Public API	—	`device-vision/` module with token-auth REST + heartbeat + round profiles
Supabase	1 audit table	10 audit + registry tables + 4 policy-gate seeds
Orchestrator	—	1 handler (`handleMedicationSafetyVerdict`) + 2 new `ManifestEventType` entries
Frontend	—	8 medical-kit `*AiAssist` packages, 8 service-layer clients, 2 miniapps, 6 sandbox targets
Docs	—	5 architecture docs

The work fits the existing backend arch cleanly at the seams (Moleculer service, public-api module, hospital_events, encounter-orchestrator, policy_gates, AcknowledgementRequest, market-packs) but introduces three internal sprawl problems that must be resolved before consolidation: (a) three competing LLM abstractions, (b) only one orchestrator handler for many emitted event types, © parallel provider-config between services/vision/ and the new services/llm/.

2. Domain Definition (Bounded Context)

2.1 Ubiquitous Language

Term	Meaning
Capture	Raw camera frame (base64) + minimal metadata (frameWidth, captureTimestamp). Untyped, never persisted as-is.
Inference	Stateless function: `Capture → InferenceResult`. Either YOLO detection or LLM-vision reasoning. Provider-agnostic.
Scan	A persisted `Capture + Inference + Context` row. Always has a `scan_uid` and `analyzed_at`. The unit of audit.
Verdict	A scan promoted to a clinical decision (`safe \| caution \| warning \| critical`) with a `checks[]` array. Verdicts can gate workflow; scans cannot.
Capability Module	A `services/vision/modules/<name>/` folder shipping one Moleculer mixin + one inference adapter + one Supabase audit table.
Capability	The Moleculer action name, e.g. `vision.nutritionAnalysis.analyze`. Listed in `VISION_MODULES` registry.
Round	A device-driven sequence of scans across multiple beds, defined by a `ward_round_profile`.
Device	A non-human caller of the vision API (robot dog, cart camera, wall camera, tablet, IoT sensor) authenticated by a long-lived API token, not a user JWT.

2.2 Bounded Context Boundary

                        ┌─────────────────────────────────────────┐
                        │      VISION-AI DOMAIN (bounded)         │
                        │                                         │
       capture ─────────┤  inference → scan → verdict             ├──── manifest.* event
                        │                                         │
                        │  owns: vision/* actions, *_scans tables │
                        │  owns: devices, device_rounds, profiles │
                        └────────────────┬────────────────────────┘
                                         │
                       ──────────────────┼──────────────────
                       outside the boundary:
                       • clinical decision (CDS engine)
                       • policy gating (policy_gates)
                       • escalation (AcknowledgementRequest)
                       • patient context (read models, FHIR, Mongo)
                       • storage of media (IPFS via filestore)

The domain never makes clinical decisions. It produces structured observations + verdicts. Downstream systems (CDS, policy gates, ack inbox) act on them.

2.3 What is NOT in scope

Streaming pipelines — Stage 1 supports single-frame REST only. WebSocket streaming is sketched in the device-vision controller (POST /device/v1/vision/stream/start) but not implemented; deferred to phase 4.
On-device inference — The robot is “a dumb camera with legs.” YOLO/LLM run server-side. (Future: optional Jetson-side YOLO for framing assist, but the canonical inference is always server-side.)
Generic LLM chat — That’s services/llm/. Vision-AI is single-purpose multimodal inference with strict JSON schemas. See §6.7 for the dependency.

3. Reusable Primitives (The Layer Cake)

The pattern below is what every capability module instantiates. All five layers are required; missing one means the capability can’t ship.

┌─────────────────────────────────────────────────────────────────────────┐
│ Layer 5 — UI Surface                                                    │
│   medical-kit/{capability}/{Capability}AiAssist.tsx                     │
│   + sandbox target + (optional) miniapp + DynamicContentRenderer hook   │
└──────────────────────────────┬──────────────────────────────────────────┘
                               │ POST /api/v2/vision/{capability}/{verb}
┌──────────────────────────────▼──────────────────────────────────────────┐
│ Layer 4 — Frontend Service                                              │
│   web/src/services/{capability}-vision-ai.service.ts                    │
│   • multipart→base64 + stripDataUri                                     │
│   • optimistic local stub fallback when backend off                     │
│   • IPFS upload of raw frame (via filestore service)                    │
└──────────────────────────────┬──────────────────────────────────────────┘
                               │ REST → ApiGateway → Moleculer
┌──────────────────────────────▼──────────────────────────────────────────┐
│ Layer 3 — Capability Mixin (Moleculer action)                           │
│   services/vision/.../modules/{capability}/{capability}.controller.mixin.ts │
│   • parameter validation                                                │
│   • orchestrate: fetch context → run inference → persist → emit         │
│   • single transaction boundary; fail-soft on telemetry                 │
└──────────────────────────────┬──────────────────────────────────────────┘
                               │
┌──────────────────────────────▼──────────────────────────────────────────┐
│ Layer 2 — Inference Adapter                                             │
│   services/vision/.../modules/_shared/{kind}Adapter.ts                  │
│   • provider-agnostic; backend chosen by ever.config                    │
│   • yolo: stub | onnx | remote(Triton)                                  │
│   • llm:  stub | openai-compatible | ollama                             │
│   • Composed adapters chain both (e.g. medicationInferenceAdapter)      │
└──────────────────────────────┬──────────────────────────────────────────┘
                               │
┌──────────────────────────────▼──────────────────────────────────────────┐
│ Layer 1 — Persistence + Eventing                                        │
│   • supabase.from('{capability}_scans').upsert(...)  (audit, fail-soft) │
│   • ctx.broker.broadcast('VISION_{CAPABILITY}_*', payload) (NATS)       │
│   • supabase.from('hospital_events').insert(...) (orchestrator feed)    │
└─────────────────────────────────────────────────────────────────────────┘

3.1 The Two-Tier Inference Abstraction (target shape)

Today (across the three branches) we have FOUR overlapping abstractions that must collapse:

File	Purpose	Backends	Used by
`_shared/inferenceAdapter.ts` (on main)	YOLO object detection	stub, onnx, remote	surgicalCount
`_shared/llmAdapter.ts` (nutrition branch)	LLM vision, nutrition-specific JSON schema	stub, anthropic, openai	nutritionAnalysis
`_shared/llmClient.ts` (safety branch)	Generic LLM chat client	openai, ollama	medicationSafety, deviceReader
`_shared/medicationInferenceAdapter.ts` (safety branch)	OCR-only stub, drug-label specific	stub	medicationVerify, medicationSafety

Target shape (proposed):

_shared/
  llmGateway.ts          ← single multimodal LLM client (chat + vision). Wraps services/llm/ if available, otherwise direct provider call. Provider-agnostic.
  yoloAdapter.ts         ← (renamed from inferenceAdapter.ts) YOLO/object-detection only
  ocrAdapter.ts          ← OCR-only — Paddle / Tesseract / cloud-vision. Currently only in deviceReader; promote to _shared
  composed/
    medicationOcrLlm.ts  ← (renamed from medicationInferenceAdapter.ts) chains ocrAdapter + llmGateway
    nutritionLlm.ts      ← (extracted from llmAdapter.ts) — schema-specific wrapper around llmGateway

Every capability mixin picks one of yoloAdapter, ocrAdapter, llmGateway, or a composed variant. No mixin defines its own provider config.

3.2 Mixin Contract

Every capability mixin MUST expose exactly:

'<capability>.<verb>': {
  rest: 'POST /<capability>/<verb>',
  params: { /* validator schema */ },
  async handler(ctx): Promise<<Capability>Result> {
    // 1. Validate + assemble context (fail-fast)
    // 2. Run inference via adapter
    // 3. Persist scan to Supabase (fail-soft — never block on telemetry)
    // 4. Emit NATS event (broker.broadcast)
    // 5. Insert hospital_events row (orchestrator picks up async)
    // 6. Return structured result
  }
}

Invariants:

Idempotent on scan_uid — callers may supply their own; mixin generates one if absent. Re-POST with same scan_uid is a no-op for persistence but re-runs inference (caller’s choice).
Fail-soft persistence — Supabase write errors are logged but never raised. Inference is the value; the audit is the bonus.
No clinical action — the mixin returns the verdict but does NOT trigger acknowledgements, send messages, or write to MedicationAdministration. That’s the orchestrator’s job.
All capture media → IPFS — raw frames go through services/filestore/ first; only the IPFS CID is stored in frame_uri. (Currently spotty across branches; needs uniformity.)
PHI minimization in LLM prompts — patient context passed to LLM contains age/sex/weight/labs only; no name, HN, or DOB. Already enforced in patientContextFetcher.ts.

3.3 Frontend Service Contract

Every web/src/services/{capability}-vision-ai.service.ts MUST expose:

export interface {Capability}AiService {
  analyze(input: { imageBase64: string; context: ContextShape }): Promise<{Capability}Result>;
  // Optional, when the capability supports it:
  history(query: { encounterId?: string; patientId?: string; limit?: number }): Promise<{Capability}Result[]>;
}

With three baked-in concerns:

Stub fallback — when VITE_VISION_AI_BACKEND=stub or fetch fails, return a realistic local stub. Demos must never break on backend outage.
IPFS upload — uploads the raw frame to filestore and includes the CID in the request; backend persists only the CID, not the bytes.
Network telemetry — logs inferenceMs to console + (optional) telemetry sink. Helps tune timeouts for slow LLM backends.

3.4 AiAssist Component Contract

The medical-kit/{capability}/{Capability}AiAssist.tsx component MUST:

Accept { encounterId, patientId, onResult } props minimum
Embed a (camera or upload) + (canonical 3-pane: image / structured fields / actions)
Use design-kit wrappers (no raw MUI) — see feedback_design_kit_wrappers rule
Be sandbox-renderable — every capability ships a sandbox/targets/{Capability}AiAssistTarget.tsx

4. Capability Catalog

Status legend: ✅ shipped (main) · 🔵 ready on branch · ⚪ designed only.

#	Capability	Adapter kind	Audit table	NATS event	Source
1	surgicalCount	yolo	`surgical_count_scans`	`manifest.surgery.count_scanned`	✅ main
2	nutritionAnalysis	llm	`nutrition_meal_analysis`	`VISION_NUTRITION_ANALYZED`	🔵 nutrition branch
3	medicationVerify	ocr+llm composed	`medication_scan_verifications`	`manifest.medication.scan_verified`	🔵 surgical-equipment
4	medicationSafety	ocr+llm composed + context	`medication_safety_verdicts`	`manifest.medication.safety_verdict`	🔵 surgical-equipment
5	medicationPlanner	llm (text+lab pattern)	`prescription_patterns`	(none emitted yet)	🔵 surgical-equipment
6	deviceReader	ocr (vitals/ventilator screens)	`device_screen_readings`	(none emitted yet)	🔵 surgical-equipment
7	bloodBankVerify	ocr+llm	`blood_bag_verifications`	`VISION_BLOOD_BAG_VERIFIED`	🔵 yolo-pharmacy
8	pharmacyVerify	ocr+llm	`pharmacy_dispense_verifications`	`VISION_PHARMACY_DISPENSE_VERIFIED`	🔵 yolo-pharmacy
9	specimenQa	ocr+llm	`specimen_label_qa`	`VISION_SPECIMEN_QA`	🔵 yolo-pharmacy
10	sterilizationQa	llm (CSSD indicator OCR)	`sterilization_qa_scans`	`VISION_STERILIZATION_QA`	🔵 yolo-pharmacy
11	woundAssess	llm (segmentation + measurement)	`wound_assess_scans`	`VISION_WOUND_ASSESSED`	🔵 yolo-pharmacy
12	wristbandId	ocr	`wristband_verifications`	`VISION_WRISTBAND_VERIFIED`	🔵 yolo-pharmacy
13	drugVialOcr	ocr+llm	—	—	⚪ catalog spec
14	vitalsMonitorOcr	ocr	(use `observations` via FHIR)	(use ORU-style)	⚪ catalog spec
15	orderCardOcr	llm (handwriting)	`order_card_ocr`	(none)	⚪ catalog spec
16	inventoryShelfScan	yolo	`inventory_scans`	`VISION_INVENTORY_COUNTED`	⚪ catalog spec

4.1 The “verdict” sub-family

Capabilities 4, 7, 8 (medicationSafety, bloodBankVerify, pharmacyVerify) produce verdicts — they may carry severity = critical, which means they MUST be picked up by the orchestrator and surfaced via encounter_journey_cache.active_alerts. Currently only #4 has a handler. This is a gap (see §9.2).

4.2 The “device-only” sub-family

Capabilities 6, 14, 16 (deviceReader, vitalsMonitorOcr, inventoryShelfScan) are primarily invoked by autonomous devices, not by humans through a dialog. They reach the vision service through services/public-api/device-vision/ rather than the user-facing gateway.

5. End-to-End Data Flow

5.1 Human-initiated flow (e.g. nurse uses MedicationSafetyAgent in e-MAR)

┌──────────────┐
│  e-MAR task  │  user clicks "Safety Check" in administration dialog
│  dialog      │
└──────┬───────┘
       │ 1. capture frame
       ▼
┌──────────────────────────────────┐
│  MedicationSafetyAgent.tsx       │  packages/miniapps/e-mar/
│  (medical-kit AiAssist)          │  components
└──────┬───────────────────────────┘
       │ 2. POST { imageBase64, context }
       ▼
┌──────────────────────────────────┐
│  vision-ai.service.ts            │  web/src/services/
│  • multipart→base64              │
│  • IPFS upload → CID             │
│  • optimistic stub fallback      │
└──────┬───────────────────────────┘
       │ 3. HTTPS /api/v2/vision/medication/safety
       ▼
┌──────────────────────────────────┐
│  ApiGateway (Moleculer-NATS)     │  services/gateway/
│  → vision.medicationSafety.scan  │
└──────┬───────────────────────────┘
       │
       ▼
┌──────────────────────────────────┐
│  medicationSafety.controller     │  services/vision/.../
│  .mixin.ts                       │  modules/medicationSafety/
│                                  │
│  STAGE 1 — fetchPatientContext   │ (Supabase read models, no PHI in prompt)
│  STAGE 2 — medicationInfAdapter  │ (OCR detect drug+dose)
│  STAGE 3 — llmGateway.chat       │ (clinical reasoning, JSON-mode)
│  STAGE 4 — persist verdict       │ (medication_safety_verdicts table)
│  STAGE 5 — broker.broadcast      │ ('manifest.medication.safety_verdict')
│  STAGE 6 — hospital_events insert│
└──────┬───────────────────────────┘
       │ verdict returned (sync) ────────────────────────────────┐
       │                                                          │
       ▼                                                          ▼
┌──────────────────────────┐                          ┌──────────────────────────┐
│  AiAssist renders the    │                          │  hospital_events row     │
│  verdict inline; if      │                          │  consumed by             │
│  severity=critical,      │                          │  encounter-orchestrator  │
│  blocks the Administer   │                          │  (Deno edge fn)          │
│  button until MD override│                          │                          │
└──────────────────────────┘                          │  → handleMedicationSafety│
                                                       │     Verdict              │
                                                       │  → encounter_journey_    │
                                                       │     cache.active_alerts  │
                                                       │  → (future) Acknowledge  │
                                                       │     mentRequest to       │
                                                       │     charge nurse + pharm │
                                                       └──────────────────────────┘

5.2 Device-initiated flow (e.g. robo-dog round)

┌──────────────┐
│  Robot Dog   │  patrols ward, beacon detects bed location
│  / Cart      │
└──────┬───────┘
       │ 1. POST /device/v1/round/start { profile_id: 'morning-med-pass' }
       │    Authorization: Bearer <device-api-token>
       ▼
┌──────────────────────────────────┐
│  device-vision.controller        │  services/public-api/.../
│  (public-api NestJS)             │  modules/device-vision/
│  • DeviceAuthGuard validates     │
│    token against `devices` table │
│  • loads ward_round_profile      │
│  • returns ordered module list   │
└──────┬───────────────────────────┘
       │
       │ for each bed × each module in profile.sequence:
       │
       ▼
┌──────────────────────────────────┐
│  POST /device/v1/vision/analyze  │
│  { moduleType, imageBase64,      │
│    context: { wardId, bedId } }  │
└──────┬───────────────────────────┘
       │
       │ device-vision.controller dispatches to broker:
       ▼
┌──────────────────────────────────┐
│  broker.call(VISION_MODULES[m]   │  same Moleculer action as
│         .moleculerAction, ...)   │  human-initiated flow
└──────┬───────────────────────────┘
       │
       ▼ (same persistence + event chain as 5.1)
       │
       ▼
┌──────────────────────────────────┐
│  Round summary returned          │
│  POST /device/v1/round/end       │
│  • aggregates all scans          │
│  • emits ROUND_COMPLETED         │
│  • nurse tablet shows summary    │
└──────────────────────────────────┘

Key point: human-flow and device-flow converge at the Moleculer action. The same mixin handles both. Authn differs (JWT vs device token), routing differs (gateway vs public-api), but the inference + persistence + eventing layer is shared.

6. Backend Integration Map

This section answers “does it fit?” by mapping every vision-AI touchpoint to an existing platform primitive.

6.1 NATS / Moleculer

Vision-AI need	Existing platform mechanism	Status
Service registration	Moleculer service mesh (`SERVICE_NAME = 'vision'`)	✅ fits — already on main
Cross-service calls (fetch patient context, post HL7)	`broker.call('clinical.encounter.get', ...)` etc.	✅ fits — `patientContextFetcher.ts` already calls Supabase directly; could optionally call clinical service for ground-truth
Event broadcast	`broker.broadcast('manifest.medication.safety_verdict', payload)`	✅ fits — same shape as `manifest.rx.prescribed` etc.
`ManifestEventType` registry	`infrastructure/medbase/functions/_shared/event-contract.ts`	🟡 partial — branch adds 2 entries (`medication.scan_verified`, `medication.safety_verdict`); other capability events not yet enumerated

Action items:

Add all 12 manifest.vision.* (or per-domain) event types to event-contract.ts
Decide naming: manifest.medication.safety_verdict (domain-prefixed) vs manifest.vision.medication_safety (capability-prefixed). The branches use the former; recommend keeping it because it co-locates with other medication events.

6.2 Encounter Orchestrator (Deno edge function)

The encounter-orchestrator is the only place where vision-AI events trigger downstream side-effects. The branches show one wired-up handler; the rest are missing.

Event	Handler exists?	Should write to
`manifest.medication.safety_verdict`	✅ `handleMedicationSafetyVerdict.ts`	`encounter_journey_cache.active_alerts` (action: BLOCK on critical, WARN on warning)
`manifest.medication.scan_verified`	❌ missing	`encounter_journey_cache.active_alerts` (lower severity), `audit_log`
`manifest.surgery.count_scanned`	❌ missing	`encounter_journey_cache.surgery_status`, OR procedureRequest update
`VISION_NUTRITION_ANALYZED`	❌ missing	nutritionRequest fulfillment % update, encounter_journey_cache.nutrition_intake
`VISION_BLOOD_BAG_VERIFIED`	❌ missing	blood_bag_dispense status, encounter_journey_cache.active_alerts on mismatch
`VISION_PHARMACY_DISPENSE_VERIFIED`	❌ missing	productDispense status, queue placement update
`VISION_SPECIMEN_QA`	❌ missing	labRequest specimen status (chain into `handleLabSpecimenPipeline`)
`VISION_STERILIZATION_QA`	❌ missing	sterilization_load status (CSSD inventory)
`VISION_WOUND_ASSESSED`	❌ missing	observations (FHIR Observation type) + wound_progression table
`VISION_WRISTBAND_VERIFIED`	❌ missing	active_alerts on mismatch (patient-safety-grade)

Action items:

Stub all missing handlers in a new inpatient-handlers/handleVision*.ts family or co-locate with existing domain handlers (e.g. wound goes near observation handler, specimen goes near handleLabSpecimenPipeline).
Add the case arms in encounter-orchestrator/index.ts.

Reference: docs/architecture/encounter-orchestrator-triggers.md (master inventory) + docs/architecture/lab-data-pipeline.md (specimen handler precedent).

6.3 Policy Gates

The branches seed policy-gate rows but the integration is not uniform:

Capability	Gate name	Seeded?	Enforced in UI?
surgicalCount	`surgical_count.close_or`	partial	yes (CountAiAssist)
medicationVerify	`medication.administration.tray_verified`	yes	yes (e-MAR)
medicationSafety	`administer_medication`	re-uses existing	yes (e-MAR blocks on critical)
bloodBankVerify	`blood_bank.transfusion.bag_verified`	designed	not wired
pharmacyVerify	`pharmacy.dispense.bag_verified`	yes	yes (PharmacyScannerViewport)
woundAssess	`wound.assessment.daily_completed`	no	no
(others)	—	no	no

Action items:

Add a seed migration per capability (e.g. 20260524*_<capability>_policy_gates_seed.sql) consistent with existing pattern in infrastructure/medbase/migrations/
Reference: docs/architecture/policy-gates.md and docs/architecture/policy-gates-coverage.md (visual map of every gate point)
Decide: should every vision capability automatically get a gate registered with action=OBSERVE, then operators escalate to WARN / BLOCK via the /admin/policy-gates UI? Recommended yes — gives a uniform default and avoids forgotten gates.

6.4 AcknowledgementRequest

The medication-safety doc says “trigger AcknowledgementRequest to charge nurse + pharmacist on critical verdict.” This needs a uniform mechanism:

Vision verdict (severity=critical)
    → hospital_events
        → encounter-orchestrator handler
            → encounter_journey_cache.active_alerts (UI surface)
            → AcknowledgementRequest.create({
                  templateId: 'vision_critical_verdict',
                  recipient: { type: 'role', value: 'charge_nurse' },
                  payload: { capability, scan_uid, verdict, ... },
                  escalationChain: [
                    { afterMin: 5, recipient: { type: 'user', value: pharmacist_id } },
                    { afterMin: 15, recipient: { type: 'role', value: 'attending_physician' } }
                  ]
              })

Reference: docs/architecture/acknowledgement-system.md — the universal AcknowledgementRequest (FHIR R4 Task wrapper) already supports role + role-escalation + multi-channel dispatch.

Action items:

Add vision_*_critical_verdict acknowledgement template to seed data
Wire orchestrator handlers (§6.2) to call AcknowledgementRequest.create for all severity=critical verdicts uniformly

6.5 CDS Engine

Vision verdicts that map to clinical observations (wound, vitals OCR, nutrition intake) should be able to fire CDS rules — not just hardcoded action=BLOCK alerts.

Capability	Should write to `observations` table?	CDS rule eligibility
woundAssess	yes (wound size, tissue type, drainage)	wound deterioration rule (size growing, infection signs)
vitalsMonitorOcr	yes (HR, RR, SpO2, BP, Temp)	NEWS2 / MEWS / qSOFA — already in baseline CDS library
nutritionAnalysis	yes (percentage eaten as observation)	low-intake escalation rule
medicationSafety	no — verdict is its own object, not an observation	n/a

Reference: docs/architecture/cds-vital-signs-rules.md — every observation write fires the CDS engine, frontend via recordObservation, backend via orchestrator handlers.

Action items:

Wound-assess + vitals-monitor + nutrition-analysis handlers should call recordObservation() (FHIR Observation insert) in addition to writing their domain-specific *_scans row. CDS fires automatically.

6.6 RUDS (Rogue User Detection)

Vision-AI generates user-action telemetry that should feed the platform-wide threat detection:

Action class: vision_scan
  Subject: user_id (or device_id)
  Object: patient_id + capability
  Risk factors:
    • scanning many patients in short window (out-of-pattern)
    • scanning patients off-roster
    • repeated critical verdicts ignored
    • device tokens used outside expected ward

Reference: docs/architecture/rogue-user-detection-system.md — user_action_events hypertable consolidates audit silos. Vision should append rows here.

Action items:

Add user_action_events insert (or NATS event RUDS consumes) in every mixin handler. Currently only writes to capability-specific table.
Devices count as a special principal class in RUDS — needs a device_id field beside user_id.

6.7 LLM Service (`services/llm/`)

The recently-added services/llm/ (self-hosted Ollama platform — see project_llm_platform memory) and the vision-service’s LLM adapters overlap on provider abstraction.

Current duplication:

services/vision/.../_shared/llmClient.ts defines LlmChatMessage, LlmResponse, callOpenAi, callOllama
services/llm/ presumably defines the same things for its own purposes

Recommended consolidation:

The vision service’s llmGateway.ts (proposed in §3.1) should call services/llm/ via Moleculer when present, falling back to direct provider call only in standalone deploys.
services/llm/ owns provider config (endpoint, key, model, fallback). Vision asks: “give me a chat completion with this prompt + image, in JSON mode, with this schema.”
Per-use-case model selection (in services/llm/) means vision capabilities can each request use_case: 'medication_safety', use_case: 'nutrition_analysis', etc. and the LLM service picks the appropriate model.

Action items:

Decide: is services/llm/ a hard dependency of services/vision/, or a soft one? Recommend soft — config flag VISION_LLM_VIA_GATEWAY=true, default true in production, false in standalone vision-only deploys.

6.8 FHIR & HL7v2

Vision-AI verdicts that represent observations are FHIR-shaped:

Capability	FHIR resource	HL7v2 message
woundAssess	`Observation` (clinical-finding category) + `Media` (the photo)	ORU^R01 (skin assessment)
vitalsMonitorOcr	`Observation` × N (vital-signs panel)	ORU^R01
nutritionAnalysis	`Observation` (nutrition-intake category)	ORU^R01
medicationVerify, medicationSafety	`Provenance` or custom `Task`	(not a standard message)
surgicalCount	`Procedure.note` or custom audit	(not a standard message)

Reference: docs/architecture/fhir-transformer-module.md — adding new resources/fields has a checklist.

Action items:

Wound + vitals + nutrition: emit as Observation via the existing FHIR write API, not into their own audit tables only. Audit table becomes secondary; FHIR Observation is the source of truth for downstream FHIR subscribers.
Add Media resource support for photo attachments (IPFS CID → Media.content.url).

6.9 Market Packs (Multi-Region)

Vision-AI prompts and seed data must be bilingual / multi-region:

Concern	Mechanism	Notes
LLM system prompts	Currently hardcoded English with `nameLocal` outputs	OK for nutrition (Thai); needs ja/fil variants for Japan/Philippines deploys
OCR language hint	`ocrAdapter` config	Paddle supports multilingual; pass `lang: 'th'` / `lang: 'ja'` from `ever.config`
Drug catalog for LASA	`lasaDictionary.ts` is currently single-language	Promote to a market-pack-driven seed table `lasa_dictionary` keyed by tenant/region
Ward round profiles	`ward_round_profiles` seeded with Thai descriptions	Seed JP/PH variants via market packs

Reference: infrastructure/market-packs/{region}/ — every region has its own seed-pathology, seed-kaigo-rates, etc.

Action items:

Add seed-vision-ai.sql (or .ts) per market pack for: LASA dictionary, ward round profiles, vision policy gates.
LLM system prompts should accept a locale parameter; provide th, ja, fil, en variants in _shared/prompts/.

6.10 Cron Jobs Registry

Vision-AI doesn’t add new cron jobs directly, BUT:

Device-vision rounds may be scheduled by a cron (e.g. “run morning-med-pass profile across all wards at 06:00”).
Reference: docs/architecture/cron-jobs-registry.md — all schedules go through the cron_jobs table, not ad-hoc cron.schedule(...).

Action items:

Add a device.scheduledRound cron entry per region as part of market-pack seed.

7. Frontend Integration Map

7.1 Patient Profile

DynamicContentRenderer.tsx already gained two new module entries (on the nutrition branch):

modules.NutritionIntakeAI
modules.DeviceScreenReader (designed)

Pattern: each capability that’s patient-scoped registers itself in DynamicCoreApp enum + a switch case in DynamicContentRenderer. Then it’s available for drag-drop placement on patient profile via the existing PatientProfileDisplayRGL.

7.2 Miniapps

web/packages/miniapps/nutrition-intake-ai/ — full miniapp + dialog wrapper (nutrition branch)
web/packages/miniapps/e-mar/components/ — 4 in-MAR-dialog AiAssist components (surgical-equipment branch)
web/packages/miniapps/device-management/ — admin UI for devices + rounds (nutrition branch)
web/packages/miniapps/central-order-inspector/ — order tracking with vision-AI overlays (yolo-pharmacy + nutrition branches both touch this — merge conflict candidate)

7.3 Medical-Kit Packages (the AiAssist family)

Package	Component	From branch
`medical-kit/blood-bank-verify/`	BloodBankAiAssist	yolo-pharmacy
`medical-kit/lab-specimen-qa/`	SpecimenQaAiAssist	yolo-pharmacy
`medical-kit/pharmacy-verify/`	PharmacyAiAssist + OPD/IPD variants + Scanner viewport	yolo-pharmacy
`medical-kit/sterilization-qa/`	SterilizationQaAiAssist	yolo-pharmacy
`medical-kit/wound-assess/`	WoundAssessAiAssist	yolo-pharmacy
`medical-kit/wristband-id/`	WristbandIdAiAssist	yolo-pharmacy
`periops-kit/.../CountAiAssist/`	(on main; baseline for the family)	main

Action items:

Establish a reusable medical-kit/ai-assist-shell/ package containing the canonical 3-pane layout (image / fields / actions), MediaCapture component, and result-rendering utilities. Currently every AiAssist re-implements its shell. Estimated ~600 LOC dedup.

7.4 Sandbox

Every capability has a web/sandbox/targets/{Capability}AiAssistTarget.tsx for pnpm dev standalone testing. The yolo-pharmacy branch adds 6 targets; nutrition branch adds 2.

8. Multi-Region & Tenancy Considerations

Device tokens are tenant-scoped — devices.tenant_id already in schema.
Round profiles are ward-scoped with '*' wildcard for default — needs tenant_id for true multi-tenant.
LLM cost — Per-region cost ceiling. Recommend a vision_inference_budget policy in services/llm/ use-cases; vision honors it.
Data residency — Critical for Japan (APPI) and Philippines (Data Privacy Act). Already addressed: PHI never sent to LLM, raw frames stay in-hospital IPFS. Re-affirm in this doc.

9. Identified Gaps & Integration Risks

Before merging the three branches, these must be resolved. Severity: 🔴 blocker · 🟡 should-fix · 🟢 nice-to-have.

9.1 🔴 Adapter sprawl (3 overlapping LLM abstractions)

Problem: llmAdapter.ts, llmClient.ts, medicationInferenceAdapter.ts all wrap LLM provider calls with different shapes. Merging the three branches as-is would ship all four.

Fix: Pre-merge consolidation pass — collapse into the structure in §3.1 (llmGateway.ts, yoloAdapter.ts, ocrAdapter.ts, composed/*). Each mixin updated to use the consolidated layer.

Estimated work: ~400 LOC churn across services/vision; touches every mixin.

9.2 🔴 Orchestrator handler coverage (1 of 10 events handled)

Problem: Only manifest.medication.safety_verdict has a handler. Other events emit into hospital_events but no downstream side-effect runs. Critical-severity verdicts from other capabilities (blood bank mismatch, pharmacy dispense mismatch) silently fail to alert.

Fix: Implement 9 handler stubs (§6.2). Reuse the handleMedicationSafetyVerdict shape — most write into encounter_journey_cache.active_alerts with capability-specific severity rules.

Estimated work: ~1500 LOC orchestrator handlers + 9 case arms in index.ts + tests.

9.3 🟡 LLM service duplication with `services/llm/`

Problem: Vision’s _shared/llmClient.ts duplicates provider abstraction work happening in services/llm/. Two configs, two API key envs, two fallback chains.

Fix: Vision’s llmGateway.ts calls services/llm/ via Moleculer when VISION_LLM_VIA_GATEWAY=true. Direct provider call as fallback. Single config source of truth.

Estimated work: ~150 LOC + Moleculer action contract in services/llm/.

9.4 🟡 Frame storage is inconsistent

Problem: Some mixins persist raw base64 in the DB (debug); some upload to IPFS; some upload nowhere and lose the frame. No uniform contract.

Fix: All mixins MUST upload raw frame to filestore (IPFS) before persisting scan. Only store frame_uri (IPFS CID). The medication_safety_verdicts.frame_uri column already exists; promote pattern to all tables.

Estimated work: ~200 LOC + audit existing migrations for missing columns.

9.5 🟡 No FHIR projection for observation-type verdicts

Problem: Wound, vitals OCR, nutrition intake are clinical observations. They live in _scans tables but don’t appear as FHIR Observation resources. FHIR subscribers miss them.

Fix: §6.8 — call recordObservation() (FHIR write API) from the relevant mixin handlers; audit table becomes secondary.

Estimated work: ~300 LOC + FHIR transformer updates.

9.6 🟡 RUDS doesn’t see vision actions

Problem: user_action_events is platform-wide threat-detection feed; vision skips it. Rogue device-token use or out-of-pattern scanning would not trip RUDS rules.

Fix: §6.6 — vision_scan action class, insert into user_action_events from every mixin. Add 2-3 baseline detection rules (device token outside ward, ignored critical verdict).

Estimated work: ~150 LOC + 3 detection-rule seeds.

9.7 🟢 Central-order-inspector miniapp duplication

Problem: Both yolo-pharmacy and nutrition branches add web/packages/miniapps/central-order-inspector/ with overlapping OrderTrackingViews.tsx. Merging both will conflict.

Fix: Already a separate concern (order-inspector is not vision-AI proper). Merge once first, then vision-AI branches rebase.

9.8 🟢 Sandbox vite.config drift

Problem: All three branches modify web/sandbox/vite.config.ts to add their targets. Three conflicting edits.

Fix: Merge sequentially, resolve trivially. Add to merge runbook.

9.9 🟢 Test coverage

Problem: Zero new tests across the three branches for the new mixins / adapters / handlers. The single existing test in services/vision (surgicalCount) was the template; none of the new capabilities followed it.

Fix: Add unit tests per mixin (stub adapter path) + 1 integration test per handler. Reference: the /tmp/jest-run scratch-jest workflow in fhir-ipd-handoff-2026-05-09.md.

Estimated work: ~800 LOC tests.

9.10 🟢 Bilingual prompts

Problem: LLM prompts hardcoded English. JP/PH deploys would mis-localize output.

Fix: §6.9 — _shared/prompts/{capability}/{locale}.ts, default en, override via tenant.

Estimated work: ~200 LOC + 4 locales × ~7 capabilities = ~28 prompt translations (LLM-assisted).

10. Implementation Plan (Recommended Phases)

Phase 0 — Pre-merge consolidation (1 day, blocker)

Resolve §9.1 — collapse the 4 adapters into the §3.1 structure
Resolve §9.7 — merge central-order-inspector independently
Define final ManifestEventType enumeration for all 10 capabilities (§6.1)

Phase 1 — Cherry-pick the foundation (1 day)

Cherry-pick from surgical-equipment-counter-ai-LU0GW:
- _shared/llmGateway.ts (consolidated)
- _shared/ocrAdapter.ts
- _shared/composed/medicationOcrLlm.ts
- medicationSafety/ mixin (most complex; proves the pattern)
- handleMedicationSafetyVerdict.ts (the only handler so far)
Cherry-pick from nutrition-ai-meal-analysis-SLbto:
- nutritionAnalysis/ mixin (the clean reference impl)
- device-vision/ module in public-api
- All 4 migrations (devices, device_rounds, ward_round_profiles, nutrition)
Cherry-pick from surgical-ai-yolo-pharmacy-XB3Sx:
- Remaining 6 capability mixins (each is self-contained on the consolidated _shared/)
- Their migrations

Phase 2 — Orchestrator handler coverage (3 days)

Implement 9 missing handlers (§9.2)
Add case arms in encounter-orchestrator/index.ts
Wire AcknowledgementRequest creation for severity=critical (§6.4)

Phase 3 — Observation projection (2 days)

§9.5 — wound, vitals, nutrition → FHIR Observation via recordObservation()
CDS engine picks up automatically (§6.5)

Phase 4 — Platform integration polish (2 days)

§9.6 — RUDS feed
§9.3 — services/llm/ integration
§9.4 — uniform IPFS frame storage
§9.10 — bilingual prompts
§6.9 — market-pack vision seeds for JP/PH
§6.10 — scheduled rounds cron entry

Phase 5 — Tests + frontend polish (3 days)

§9.9 — unit + integration tests
§7.3 — medical-kit/ai-assist-shell/ dedup
Sandbox target consolidation

Total estimated effort: ~12 working days. Realistic given the branches are 95% done; the remaining 5% is the integration glue.

11. Fit Assessment: Does It Integrate?

Verdict: Yes, with the §9 gaps closed.

Things that fit cleanly:

✅ Service shape (Moleculer service with mixins) is identical to existing services (administration, clinical, medication)
✅ Event emission pattern matches manifest.rx.* family
✅ Audit table convention (*_scans, *_verifications, *_verdicts) matches *_dispenses, *_results etc.
✅ Policy-gate integration is the same shape as cashier/discharge gates (no new mechanism needed)
✅ AcknowledgementRequest is universal — vision verdicts plug in as a new template, no code change in the inbox
✅ Device-vision sub-domain uses public-api correctly (token auth, not user JWT)
✅ Market-pack pattern accommodates per-region LASA dictionaries and prompts
✅ Read-model pipeline (Supabase + hospital_events + orchestrator) is the same as every other domain

Things that need new contracts (one-time platform additions):

⚠️ A new user_action_events action class vision_scan with device-principal support (§6.6)
⚠️ A new Media resource in the FHIR transformer module for photo attachments (§6.8)
⚠️ A new use-case vision_* family in services/llm/ config (§6.7)

Things that are NOT a fit:

❌ The four-adapter sprawl as it currently stands across the three branches — must be consolidated before merge (§9.1)

12. File Inventory (post-consolidation target shape)

services/vision/src/api/vision/
  visionService.ts                                 ← composes all 10 mixins
  modules/
    _shared/
      llmGateway.ts                                ← single LLM client (chat + vision)
      yoloAdapter.ts                               ← (was inferenceAdapter.ts)
      ocrAdapter.ts                                ← promoted from deviceReader
      supabaseClient.ts                            ← already on main
      composed/
        medicationOcrLlm.ts                        ← (was medicationInferenceAdapter)
        nutritionLlm.ts                            ← (was llmAdapter.ts)
      prompts/
        medicationSafety/{en,th,ja,fil}.ts
        nutritionAnalysis/{en,th,ja,fil}.ts
        ... per capability
    surgicalCount/        (on main)
    nutritionAnalysis/
    medicationVerify/
    medicationSafety/
      medicationSafety.controller.mixin.ts
      patientContextFetcher.ts
      lasaDictionary.ts                            ← seed-driven, market-pack overridable
    medicationPlanner/
    deviceReader/
    bloodBankVerify/
    pharmacyVerify/
    specimenQa/
    sterilizationQa/
    woundAssess/
    wristbandId/

services/public-api/src/api/publicapi/modules/
  device-vision/
    device-vision.module.ts
    device-vision.controller.ts
    device-auth.guard.ts
    supabase.client.ts
    dto/{analyze-frame,device-heartbeat,round}.dto.ts

infrastructure/medbase/
  functions/
    encounter-orchestrator/index.ts                ← +9 case arms
    inpatient-handlers/
      handleMedicationSafetyVerdict.ts             ← already exists
      handleVisionScanVerified.ts                  ← NEW (medication scan)
      handleSurgicalCountScanned.ts                ← NEW
      handleNutritionAnalyzed.ts                   ← NEW
      handleBloodBagVerified.ts                    ← NEW
      handlePharmacyDispenseVerified.ts            ← NEW
      handleSpecimenQa.ts                          ← NEW (chains into lab pipeline)
      handleSterilizationQa.ts                     ← NEW
      handleWoundAssessed.ts                       ← NEW (writes FHIR Observation)
      handleWristbandVerified.ts                   ← NEW
    _shared/event-contract.ts                      ← +10 ManifestEventType entries
  migrations/
    20260524a_nutrition_meal_analysis.sql
    20260524b_devices.sql
    20260524c_device_rounds.sql
    20260524d_ward_round_profiles.sql
    20260524e_device_screen_readings.sql
    20260524f_sterilization_qa_scans.sql
    20260524g_wound_assess_scans.sql
    20260524h_medication_scan_verifications.sql
    20260524i_medication_safety_verdicts.sql
    20260524j_prescription_patterns.sql
    20260524k_blood_bag_verifications.sql
    20260524l_pharmacy_dispense_verifications.sql
    20260524m_specimen_label_qa.sql
    20260524n_wristband_verifications.sql
    20260524o_vision_policy_gates_seed.sql

web/
  src/services/
    vision-ai.service.ts                           ← consolidated client (was on main, expanded)
    nutrition-ai.service.ts
    blood-bank-vision-ai.service.ts
    pharmacy-vision-ai.service.ts
    specimen-vision-ai.service.ts
    sterilization-vision-ai.service.ts
    wound-vision-ai.service.ts
    wristband-vision-ai.service.ts
  packages/medical-kit/src/
    ai-assist-shell/                               ← NEW shared shell
      index.ts
      MediaCapture.tsx
      InferenceResultPanel.tsx
      AiAssistDialog.tsx
    blood-bank-verify/BloodBankAiAssist.tsx
    lab-specimen-qa/SpecimenQaAiAssist.tsx
    pharmacy-verify/{IpdPharmacyAiAssist,OpdPharmacyAiAssist,PharmacyScannerViewport}.tsx
    sterilization-qa/SterilizationQaAiAssist.tsx
    wound-assess/WoundAssessAiAssist.tsx
    wristband-id/WristbandIdAiAssist.tsx
  packages/miniapps/
    nutrition-intake-ai/
    device-management/
    e-mar/components/{MedicationAiPlanner,MedicationAiVerify,MedicationSafetyAgent,DeviceScreenReader}.tsx
  sandbox/targets/{Nutrition,DeviceManagement,BloodBank,Pharmacy,Specimen,Sterilization,Wound,Wristband,MedicationAiPlanner,MedicationSafetyAgent}AiAssistTarget.tsx

infrastructure/market-packs/
  medos-japan/seed-vision-ai.sql                   ← LASA-ja, profile-ja prompts
  medos-philippines/seed-vision-ai.sql             ← LASA-fil
  medos-thailand/seed-vision-ai.sql                ← (existing seeds promoted)

13. References

docs/architecture/vision-ai-complete-summary.md — nutrition-branch high-level overview (superseded by this doc)
docs/architecture/vision-ai-module-catalog.md — 9-module spec for builders (capability details)
docs/architecture/vision-ai-robotics-integration.md — robot/IoT device design (still authoritative for §5.2)
docs/architecture/medication-safety-agent.md — premium add-on deep-dive (data model + prompts)
docs/architecture/medication-ai-planner.md — LLM medication planning
docs/architecture/device-screen-reader.md — vitals monitor OCR
docs/architecture/encounter-orchestrator-triggers.md — master read-model layer (§6.2 anchors here)
docs/architecture/policy-gates.md + policy-gates-coverage.md — gate engine (§6.3)
docs/architecture/acknowledgement-system.md — universal ack (§6.4)
docs/architecture/cds-vital-signs-rules.md — CDS engine (§6.5)
docs/architecture/rogue-user-detection-system.md — RUDS feed (§6.6)
docs/architecture/fhir-transformer-module.md — FHIR write API (§6.8)
docs/architecture/cron-jobs-registry.md — cron registry (§6.10)
docs/architecture/lab-data-pipeline.md — specimen handler precedent
docs/module-loop/BACKEND-CONTRACTS.md — extensible shared shapes (EntityContract, status enum, event payload, audit-log table)
web/CLAUDE.md — frontend conventions
web/AGENTS.md — workflow-sensitive change rules

1. TL;DR

2. Domain Definition (Bounded Context)

2.1 Ubiquitous Language

2.2 Bounded Context Boundary

2.3 What is NOT in scope

3. Reusable Primitives (The Layer Cake)

3.1 The Two-Tier Inference Abstraction (target shape)

3.2 Mixin Contract

3.3 Frontend Service Contract

3.4 AiAssist Component Contract

4. Capability Catalog

4.1 The “verdict” sub-family

4.2 The “device-only” sub-family

5. End-to-End Data Flow

5.1 Human-initiated flow (e.g. nurse uses MedicationSafetyAgent in e-MAR)

5.2 Device-initiated flow (e.g. robo-dog round)

6. Backend Integration Map

6.1 NATS / Moleculer

6.2 Encounter Orchestrator (Deno edge function)

6.3 Policy Gates

6.4 AcknowledgementRequest

6.5 CDS Engine

6.6 RUDS (Rogue User Detection)

6.7 LLM Service (services/llm/)

6.8 FHIR & HL7v2

6.9 Market Packs (Multi-Region)

6.10 Cron Jobs Registry

7. Frontend Integration Map

7.1 Patient Profile

7.2 Miniapps

7.3 Medical-Kit Packages (the AiAssist family)

7.4 Sandbox

8. Multi-Region & Tenancy Considerations

9. Identified Gaps & Integration Risks

9.1 🔴 Adapter sprawl (3 overlapping LLM abstractions)

9.2 🔴 Orchestrator handler coverage (1 of 10 events handled)

9.3 🟡 LLM service duplication with services/llm/

9.4 🟡 Frame storage is inconsistent

9.5 🟡 No FHIR projection for observation-type verdicts

9.6 🟡 RUDS doesn’t see vision actions

9.7 🟢 Central-order-inspector miniapp duplication

9.8 🟢 Sandbox vite.config drift

9.9 🟢 Test coverage

9.10 🟢 Bilingual prompts

10. Implementation Plan (Recommended Phases)

Phase 0 — Pre-merge consolidation (1 day, blocker)

Phase 1 — Cherry-pick the foundation (1 day)

Phase 2 — Orchestrator handler coverage (3 days)

Phase 3 — Observation projection (2 days)

Phase 4 — Platform integration polish (2 days)

Phase 5 — Tests + frontend polish (3 days)

11. Fit Assessment: Does It Integrate?

12. File Inventory (post-consolidation target shape)

13. References

6.7 LLM Service (`services/llm/`)

9.3 🟡 LLM service duplication with `services/llm/`