Patient 360 - 3D Command Surface

3D patient command surface: body model centre, diagnoses/imaging/labs/vitals/orders as draggable cards, three modes, voice order, care launchers.

11 min read diagramsUpdated 2026-06-03docs/architecture/patient-360.md

One patient, one screen. A 3D body model sits in the centre; everything about the patient — diagnoses, imaging, labs, live vitals, orders, journey, alerts — floats around it as draggable cards. Clinicians collaborate live (cursors, presence, per-finding threads, room chat), dictate orders by voice, annotate the body, and launch the real nurse / IPD-medication modules — all without leaving the patient.

Built as a per-patient miniapp; renders in the clinic workspace via DynamicContentRenderer and in isolation via the lightweight sandbox.

1. Concept

Patient 360 reframes the patient chart as a spatial command centre:

The body is the anchor. Findings, orders, devices, imaging and vitals are points on the body (anatomically anchored, SNOMED-coded) — not rows in a table. The same value recorded once links everywhere at its timestamp.
Everything else orbits it. Read-only summary cards (who/where/what) on the edges; the body stays clear in the middle.
It’s multiplayer. Multiple clinicians share the room — you see each other’s cursors, who’s viewing which finding, and can discuss any point or the whole patient in real time.
It’s a launchpad, not a silo. Heavy clinical work (e-MAR, IPD med planner, nursing assessment, nurse-acknowledge) opens the real modules — Patient 360 summarises and routes, it does not re-implement.

Three modes (top-bar toggle): Summary (read-only overview, default) · Annotate (the marker studio) · Conference (collaboration-forward: roster + chat).

2. How to open it

Context	Identifier
Clinic workspace (per-patient)	`DynamicCoreApp.PATIENT_360` = `modules.Patient360` (legacy `modules.PatientBodySummary3D` still resolves)
Standalone sandbox	`http://localhost:5179/?target=Patient360` (aliases: `PatientBodySummary3D`, `BodyModelAnnotationStudio`)
Rendered by	`DynamicContentRenderer` → `PatientBodySummary3DSandboxTarget` (the sandbox target imported into the engine)

The same component runs in both. With a real patientId/encounterId (clinic) it reads live data; with none (sandbox) it falls back to deterministic mock data so the surface always renders.

3. Architecture

                        ┌─────────────────────────── Patient 360 surface ───────────────────────────┐
                        │  BodyModelAnnotationStudioTarget.tsx  (modes · panels · action bar)        │
                        │                                                                            │
   3D BODY  ────────────┤  BodyModelViewer (medical-kit/body-model-3d)                               │
   (R3F + drei)         │   • MUI-icon pins (token→icon)  • popup card: X + per-dx chat               │
                        │   • inline thread BELOW the card (renderMarkerThread)                       │
                        │   • camera: lock / auto-rotate / resetSignal                                │
                        │                                                                            │
   DATA  ───────────────┤  usePatientBodyData  →  patient · location · dx · imaging · labs ·          │
   (live sources)       │     encounter · orders · bloodRequests · journey · alerts · nextActions      │
                        │       ├── encounter_journey_cache    (header, journey, working_dx, alerts)   │
                        │       ├── ipd_admissions_dashboard   (WHICH WARD / BED · LOS · payor)        │
                        │       ├── department_queues          (order system — orders + imaging)       │
                        │       ├── observations (biomarker)   (labs, via useObservations)             │
                        │       └── v2/medication/bloodRequests (blood bank — REST, guarded)           │
                        │  useObservationSync / useLiveVitals →  live vitals (biomarker/vital basket)   │
                        │                                                                            │
   COLLAB  ─────────────┤  usePresence (body-collab)  →  cursors · presence · comments · chat         │
   (realtime)           │     transport = BroadcastChannel (sandbox) | Supabase realtime (prod)       │
                        │                                                                            │
   VOICE  ──────────────┤  SpeechToText  →  runVoiceOrder (LLM + REST catalog) →  ProposeOrderPayload │
                        │                                                                            │
   LAUNCH  ─────────────┤  Care launchers  →  onLaunchModule(key)  →  openModal(modalId)              │
   (summarize+launch)   │     IpdMedicationHub · MarSystem · EKardex · FocusListEnhanced · …           │
                        └────────────────────────────────────────────────────────────────────────────┘

4. Component / file map

Layer	File	Role
Surface	`web/sandbox/targets/BodyModelAnnotationStudioTarget.tsx`	The Patient 360 component — modes, all panels, action bar, wiring. Default export takes `{ patientId, encounterId, patientData, patientContext, currentUser, onLaunchModule, noWrapper }`.
3D viewer	`web/packages/medical-kit/src/body-model-3d/BodyModelViewer.tsx`	R3F/drei body + markers. Props added for P360: `icon` token→MUI icon on pins, `onDismissMarker`, `onOpenComments`, `commentCounts`, `openThreadId`, `renderMarkerThread`, `autoRotate`, `lockCamera`, `resetSignal`.
3D coords/store	`…/body-model-3d/bodyRegionCoordinates.ts`, `bodyAnnotationStore.ts`	SNOMED region coordinates; shared zustand marker store (`useBodyAnnotation`) + `registerMarkerSource`.
Data layer	`web/src/services/patient-body-data/usePatientBodyData.ts`	Reads `encounter_journey_cache` + `ipd_admissions_dashboard` (ward/bed) + `department_queues` (orders) + biomarker `observations` (labs) + `v2/medication/bloodRequests` (blood, REST via `callAPI`, guarded); returns card-shaped data (`BodyLocation`, `BodyBlood`, …) with per-section mock fallback. `deriveOwnedMarkers` lives in the target.
Vitals	`web/src/services/observation-sync/useObservationSync.ts`, `web/src/services/observation-core/useObservations.ts`	Offline-first vital/biomarker read+write basket; `useLiveVitals` (in the target) overlays real obs onto a demo pulse.
Journey shape	`web/src/services/clinical-dashboard/types.ts` (`EncounterJourneyCache`)	The read-model row shape the data layer maps from.
Collaboration	`web/src/services/body-collab/{usePresence,transports,types,index}.ts`	Presence (cursors/hover/active), comments-per-annotation, room chat. `BroadcastChannelTransport` (sandbox/cross-tab) + `SupabaseTransport` (prod).
Voice order	`web/src/services/ai/voice-order/{runner,client,rest-catalog,types}.ts`	`runVoiceOrder({transcript, llm, catalog})` → `ProposeOrderPayload`. `OpenAICompatibleLLM` (Ollama) + `RestVoiceOrderCatalog`.
STT capture	`web/src/common/components/shared-engine/speech-to-text/SpeechToText.tsx`	Web-Speech component (`onTranscriptComplete`). Redux-free.
Registration	`web/src/setup/dynamic/DynamicCoreApp.ts`	`PATIENT_360` / `PATIENT_BODY_SUMMARY_3D` enum values.
	`web/src/common/components/shared-engine/dynamic-core/engine/DynamicContentRenderer.tsx`	`case`s render the target with patient props + `onLaunchModule={(key)=>dispatch(openModal({data:{modalId:key}}))}`.
	`web/sandbox/registry.ts`	Sandbox keys `Patient360` / `PatientBodySummary3D` / `BodyModelAnnotationStudio`.

5. The four layers in detail

5.1 3D body viewer (`BodyModelViewer`)

Pins = MUI icons, not emoji. Each marker carries an icon token (diagnosis, warning, critical, vital, pain, msk, device, imaging, order, surgical, …); the viewer maps the token → MUI icon via MARKER_ICON_MAP. A region with >1 marker shows a count badge.
Popup card (drei `` at the projected marker position) has an ✕ (onDismissMarker) and a chat button per diagnosis (onOpenComments, badged by commentCounts).
Inline thread pops up below the card — when openThreadId matches a diagnosis, the card renders renderMarkerThread(id) beneath the dx list (light-themed InlineMarkerThread): live per-thread presence (“N here” + avatars), relative time (“just now”), comment input.
Camera: lockCamera (freeze rotate+zoom), autoRotate (turntable), resetSignal (bump → CameraReset eases back to full-body overview). Controls float above the head (top-centre).
Occlusion (far-side markers hide) + free-placement (exact mesh hit point) are pre-existing.

5.2 Data layer (`usePatientBodyData`)

Reads the whole patient, live:
- encounter_journey_cache — patient header, journey lane (current_physical_location + pending_tickets), working_dx, active_alerts, admission_context (Supabase, realtime).
- ipd_admissions_dashboard — which ward / bed the patient is in, plus LOS, payor, code status, isolation, fall risk (Supabase, realtime; falls back to ipd_admissions). Drives the new location section + the patient-header ward/bed + encounter dept/LOS/AN enrichment.
- department_queues — the order system: active orders grouped by a normalized dept type (normalizeDept folds LABORATORY/pharmacy_screening/… → one canonical lowercase type), imaging where the type is imaging, with priority + richer name resolution.
- biomarker observations — labs (via useObservations). Vitals ride useLiveVitals → useObservationSync in the surface.
- v2/medication/bloodRequests — blood bank requests (component · units · group · status · crossmatch · emergency · need-by). Read over REST with callAPI directly (NOT the bloodRequest.service, which imports platform-api-schema — a backend workspace pkg the lightweight sandbox’s fs.allow:[WEB] would reject). Wrapped so a missing backend (sandbox) silently yields [] → mock, never throws.
Per-section mock fallback so the sandbox renders with no patient; isLive flips the “Demo data” → “● Live read-model data” badge (and each card carries its own live/demo footer).
Returns shapes 1:1 with the card props (BodyPatient, BodyLocation, BodyDx, BodyImaging, BodyLab, BodyEncounter, BodyOrder, BodyBlood, BodyJourney, BodyAlert, BodyNextAction).
Owned/external markers (OR counts, imaging/X-ray, blood/surgical orders) are derived from body.imaging + body.orders (deriveOwnedMarkers) and upserted into the marker store (idempotent, selection-preserving). They are read-only — close-only in the detail panel (“edit in the source system; here you can only close”).

5.3 Collaboration (`usePresence` / body-collab)

One hook drives cursors (fractional x/y → rendered as avatar pills), hover/active awareness (peers viewing a marker show on the row + thread), comments per annotation, and room chat.
Transport-abstracted: BroadcastChannel in the sandbox / same-machine multi-tab demo; Supabase realtime (presence + broadcast) in production (hasBackend = !!patientId).
Reliability: late-joiner sync-request/reply handshake + proactive backfill on peer-join + bounded re-sync, so no message is lost in the connect cold-race. Avatars are initials.

5.4 Vitals (`useObservationSync` → `useLiveVitals`)

Real channel: useObservationSync({ source: 'vital-signs-summary' }) reads the central observations basket (offline-first). useLiveVitals overlays real values onto a demo pulse per LOINC, so vitals animate in the sandbox and light up live with a real patient. Clicking a vital flies the camera to its region; each vital also pins on the body.

6. Cross-cutting features

Feature	Where
Modes Summary / Annotate / Conference	top-bar segmented toggle; panels gated by `mode`
Action bar (search→zoom, mode, view, camera, voice, layout menu, presence)	draggable, anchored at the bottom by default
Layout menu Auto-arrange / Expand all / Collapse all / Close all / Hide-Show	`PanelControlContext { seq, cmd }`; `FloatingPanel` uses controlled position so “Auto-arrange” restores tidy defaults
Voice order	mic button → `VoiceOrderPanel` (SpeechToText → `runVoiceOrder` → reviewable proposal → “Place on body + route to Order System”)
Care launchers Nursing + IPD Medications	`NurseActionsPanel` (`CARE_GROUPS`) → `onLaunchModule(key)` → `openModal(modalId)`
Per-point & room chat	inline marker thread + Team Chat panel

Care launcher keys (open the real modules; from the IPD-med push): IpdNursingAssessment, FocusListEnhanced, MainNurseNote, PathologyRequestNurseAck (Nurse Acknowledge), OpdProcedureRecording, ActivityEngagementTracking, NandaCarePlan, MultidisciplinaryRounding · IpdMedicationHub (IPD Med Planner: order→ack→pharmacy→rounds→e-MAR), MarSystem (e-MAR), EKardex, MedicationReconciliation, IpdContinueOneDay.

7. Dual-context contract (sandbox vs clinic)

The surface is Redux-free so it runs in the lightweight sandbox. It degrades gracefully:

Capability	Sandbox (no patient / stale env)	Clinic (`DynamicContentRenderer`, real patient)
Summary cards	mock fallback (“Demo data” badge)	live `encounter_journey_cache` / `ipd_admissions_dashboard` / `department_queues` / `observations`
Ward / bed (which ward the patient is in)	mock ward+bed	live `ipd_admissions_dashboard` (realtime)
Blood bank requests	mock requests	live `v2/medication/bloodRequests` (REST, guarded)
Vitals	demo pulse	real biomarker/vital basket, overlaid
Presence/chat	BroadcastChannel (cross-tab)	Supabase realtime (cross-device)
Voice order parse	captures transcript; “needs LLM+backend” note	`runVoiceOrder` → real proposal
Care launchers	“opens in clinic app” note	`openModal` opens the real module
Identity	per-tab demo doctor	logged-in `currentUser`

8. Invariants / safety

Read models are read-only from the frontend — Patient 360 never writes encounter_journey_cache / ipd_admissions_dashboard / department_queues; the blood read is a plain GET v2/medication/bloodRequests; vitals go through the write-through → orchestrator path; orders route to the Order System.
Owned/external markers are close-only — the source system owns them.
Always renders — every data section has a mock fallback; the surface never blanks on a missing patient or stale env.
No new heavy infra — reuses observation-sync, body-collab, voice-order, body-model-3d, and existing miniapps via summarize-and-launch.
Backward-compatible — BodyModelViewer changes are additive (markers without icon get the count); legacy PatientBodySummary3D module name still resolves.

9. Extension points

Add a summary card → add a field to usePatientBodyData (real read + mock fallback) + a light *Card component + a `` in the mode fragment.
Add a marker source → registerMarkerSource({id,label,color}); set source on the marker; it auto-becomes close-only with its glyph.
Add a care launcher → append to CARE_GROUPS with a real DynamicCoreApp module key.
Wire a real read (replace a mock) → map the read model in usePatientBodyData; the card already consumes body.*.
Promote presence to Supabase for a tenant → it’s automatic when patientId is present (hasBackend → SupabaseTransport).

10. Verification

Sandbox: http://localhost:5179/?target=Patient360 (cold first load ~15–30s — large graph; warm after).
Playwright (no auth): load the target reload-tolerant (wait for “Team Chat”); WebGL renders in headed Chromium so pins/cards are inspectable. For multiplayer, open two pages in one browser.newContext() (shared origin → shared BroadcastChannel). Known gotcha: a concurrent session’s Vite error overlay can be broadcast to the page — document.querySelector('vite-error-overlay')?.remove() before interacting.
Compile: node_modules/.bin/esbuild <file> --bundle=false --jsx=automatic --loader:.tsx=tsx.

11. Deployment status

Piece	Status
Surface + 3D + modes + panels	✅ shipped (sandbox + DCR)
Data layer (journey cache / ward+bed via ipd_admissions_dashboard / queues / biomarkers / blood requests)	✅ wired, mock-fallback; lights up per-patient in DCR
Collaboration (presence/cursors/comments/chat)	✅ shipped; BroadcastChannel now, Supabase auto when patient present
Voice order	✅ capture + pipeline reused; parse needs Ollama + order backend reachable
Care launchers	✅ wired via `onLaunchModule`→`openModal` in DCR
Blood bank requests (`v2/medication/bloodRequests`, REST via `callAPI`)	✅ wired, guarded + mock-fallback; live per-patient when backend reachable
Backend reads beyond Supabase (full ICD list, imaging impressions in Mongo `imagingRequests`)	⏳ optional REST follow-up

docs/architecture/ekardex-from-journey-cache.md — the summarize-and-launch pattern + journey-cache JSONB extensions Patient 360 builds on.
docs/architecture/lab-data-pipeline.md — how observations / department_queues get populated (labs).
docs/architecture/widget-rail-surface-system.md — sibling patient-profile surface pattern.
docs/architecture/unified-clinical-assistant.md — the VoiceOrder / runner-engine substrate the voice order reuses.
web/src/services/observation-sync/README.md — the observation sync channel.

Note: the surface file is still named BodyModelAnnotationStudioTarget.tsx (renaming it would churn the registry + DCR import paths); the user-facing name + identifiers are “Patient 360”.