Ambient Clinical Scribe

Continuous-listen, draft-until-signed consultation scribe reusing existing AI infra; audio local-only, fail-open, per-tenant opt-in.

11 min read diagramsUpdated 2026-05-30docs/architecture/ambient-clinical-scribe.md

Date: 2026-05-29 Author: Claude (design session) Scope: A continuous-listen, draft-until-signed ambient scribe that captures the patient-clinician conversation, extracts structured clinical data, and pre-fills the consultation form — reusing the existing AI service infrastructure without gating any clinical write. Status: Design only — not yet built. Depends on existing: SpeechToText.tsx, shared/runner-engine.ts, ClinicalNoteAIService, SmartDiagnosis catalog.

1. Problem statement

The clinic currently has the AI pieces but they are siloed and point-in-time rather than ambient:

What exists	Gap
`SpeechToText.tsx` — per-form Web Speech API burst	No continuous session across a consultation
`ClinicalNoteAIService` — structures a given transcript	Never receives a full consultation transcript
`NurseNoteAIService` — polish/rewrite per note field	Not connected to live audio
`VoiceOrder` loop — proposes orders from voice commands	Triggered manually (FAB press), not ambient
`SmartDiagnosis` runner — differential from context	Pre-fill hook `useSmartDiagnosisPrefill` is siloed (AI runs, results don’t reach the form)
`shared/runner-engine.ts` (`runAgentLoop`)	Not called by any production runner (planned P0 refactor target)
`ASSISTANT_REGISTRY`	Written, not consulted by any UI

The ambient scribe closes these gaps in one pipeline: listen → transcribe → extract (multi-domain, parallel) → review draft → apply to form, all in one continuous session per consultation.

2. What “ambient” means here

Classic voice-order: nurse/doctor speaks a command → AI interprets → proposes.

Ambient scribe: doctor listens to patient, speaks notes aloud, has a back-and-forth → AI listens to the whole session → extracts all structured fields at once → presents a single review draft.

Three modes:

Mode	Who activates	When
Ambient	Doctor starts session at consultation open; listens passively	Full OPD consult capture
Dictation	Doctor holds mic button and dictates	Post-encounter note completion
Snippet	Nurse presses for one field (chief complaint, vitals narrated)	Existing per-form STT, unchanged

The design document covers Ambient + Dictation. Snippet mode is already working (SpeechToText.tsx per form).

3. Architecture: the three-phase pipeline

PHASE 1 — CAPTURE
  AmbientScribeSession (persistent object, per consultation)
    ├── ContinuousSTTStream  ← Web Speech API, SpeechRecognition.continuous = true
    │     interim results discarded; only final results appended to transcript buffer
    │     audio stream is NEVER sent anywhere; only the text transcript leaves the device
    └── TranscriptBuffer     utterances[] with speaker_hint (doctor | patient | unknown)
                             (speaker_hint from push-to-talk or diarisation heuristic)
                             ↓ debounce 30 s of silence OR manual "Done" press
PHASE 2 — PROCESS  (parallel, off main thread)
  runAmbientScribeExtraction(transcript, context)
    ├── ChiefComplaintExtractor     → chiefComplaint: string
    ├── HPIExtractor                → hpi: { onset, duration, location, quality, severity, modifiers }
    ├── VitalsNarratedExtractor     → vitals: { ... } (e.g. "BP 140 over 90")
    ├── SmartDiagnosisCatalog       → differentials: DiagnosisSuggestion[]
    │     reuses existing src/services/ai/smart-diagnosis/diagnosis-catalog.ts
    ├── VoiceOrderCatalog           → orders: ProposedOrder[]
    │     reuses existing src/services/ai/voice-order/rest-catalog.ts
    ├── AllergyMentionExtractor     → allergies: string[] (flagged mentions only)
    └── SOAPStructurer              → soapNote: { S, O, A, P }
          reuses ClinicalNoteAIService.processTranscript()
                       ↓
  ScribeDraft { session_id, encounter_id, extraction, confidence_map, requires_review[] }
PHASE 3 — REVIEW + APPLY
  ScribeDraftReviewDialog (non-blocking slide-in, not a blocking modal)
    ├── Side-by-side: transcript excerpt ↔ extracted field (with highlight)
    ├── Every field has Confirm / Edit / Reject
    ├── "Apply all confirmed" → dispatches to existing form state (Redux / form context)
    └── Signed-off draft → written to scribe_sessions log (audit)
          ↑ No clinical write happens until the doctor confirms in Phase 3

4. Reuse map — what changes vs. what is new

Component	Reuse / New	Notes
`SpeechToText.tsx`	Reuse — extend with `continuous = true` mode	Add `mode: ‘continuous’
`shared/runner-engine.ts` (`runAgentLoop`)	Reuse — this is the ambient scribe domain’s runtime	Registers as `'scribe'` in `ASSISTANT_REGISTRY`
`ClinicalNoteAIService.processTranscript()`	Reuse — SOAP structurer step	No changes needed
`smart-diagnosis/diagnosis-catalog.ts`	Reuse — plug in as DiagnosisCatalog tool	No changes needed
`voice-order/rest-catalog.ts`	Reuse — plug in as OrderCatalog tool	No changes needed
`NurseNoteAIService.extractStructuredNote()`	Reuse — HPI / chief-complaint extraction	No changes
`ASSISTANT_REGISTRY`	Extend — add `scribeDomain` entry	Unblocks the registry finally being used
`ScribeDraftReviewDialog`	New	Non-blocking slide-in; mirrors VoiceOrder’s review dialog pattern
`AmbientScribeSessionContext`	New	React context; session lifecycle (idle → listening → processing → reviewing)
`useAmbientScribe()`	New	Hook consumed by the consultation page
`AmbientScribeFAB`	New	Start/stop/status pill (not a blocking modal)
`scribe_sessions` Supabase table	New	Audit log of each session + accepted draft
`ScribeDraftSlice` (Redux)	New	Phase-2 output; consumed by `ScribeDraftReviewDialog`

5. Prompt contract (what goes to the LLM)

5.1 What is NOT in the prompt

No patient names, MRN, encounter ID, date of birth
No free-text medication names spelled out by patients (those go through the med catalog only)
No raw audio (the audio never leaves the browser — only the text transcript is processed)
No PHI beyond what the clinical note would contain anyway (same standing as ClinicalNoteAIService)

The existing redactForAudit / redactObject backstop from services/llm/.../_shared/redaction.ts runs over the transcript before it leaves for any remote LLM — same stance as the e-Kardex AI doc.

5.2 System prompt structure

You are a clinical documentation assistant.
You receive a raw consultation transcript (doctor-patient dialogue) and
structured encounter context.

Your output is a JSON object matching ScribeExtraction (see schema).
Extract ONLY what is explicitly stated. Do NOT infer, guess, or hallucinate.
If a field is not mentioned, return null for that field.

Encounter context (de-identified):
  role: {role}
  encounter_class: {encounter_class}
  chief_complaint_hint: {chief_complaint_hint_if_any}   ← from triage, not PHI

Transcript:
{transcript}

Return JSON only. No prose.

5.3 `ScribeExtraction` schema

interface ScribeExtraction {
  chief_complaint:   string | null;
  hpi: {
    onset:      string | null;
    duration:   string | null;
    location:   string | null;
    quality:    string | null;
    severity:   string | null;   // e.g. "7/10"
    modifiers:  string | null;
  } | null;
  vitals_narrated: {
    bp?:        string;   // "140/90"
    hr?:        string;
    rr?:        string;
    temp?:      string;
    spo2?:      string;
    weight?:    string;
  } | null;
  allergies_mentioned: string[] | null;   // drug/food names extracted only
  soap_note: {
    S: string;   // Subjective
    O: string;   // Objective
    A: string;   // Assessment
    P: string;   // Plan
  } | null;
  // Differentials and orders are resolved via catalog tools (Phase 2 parallel),
  // not from the LLM directly — catalog grounding, not hallucination.
}

6. Wiring into the consultation page

The consultation page is DynamicContentRenderer case 'ConsultationNote' → renders the encounter’s main form. The scribe hooks onto three existing chokepoints:

[AmbientScribeFAB] → useAmbientScribe()
    │
    ├── sets ContinuousSTTStream → appends to TranscriptBuffer
    │
    ├── on "Done" → dispatches runScribeExtraction(transcript, context)
    │       → ScribeDraftSlice.phase = 'processing'
    │
    └── ScribeDraftReviewDialog (portaled beside the form, not over it)
            on confirm per-field → dispatches to the form's Redux slice
              (same as useSmartDiagnosisPrefill was supposed to do — this finally closes that loop)
            on "Apply all" → batch dispatch
            on "Discard" → slice reset, audit log entry with outcome='discarded'

The ScribeDraftReviewDialog dispatches to the same Redux actions that the existing form components dispatch on manual user input. No special write path. The scribe is just a way to populate the same form that the doctor would fill out manually.

7. The smart-diagnosis prefill fix (closes the existing gap)

From the inventory: useSmartDiagnosisPrefill.ts is siloed — the SmartDiagnosis AI runs and produces differentials, but nothing reads them to pre-fill the diagnosis form.

The scribe’s Phase-3 Review Dialog reuses useSmartDiagnosisPrefill as its delivery mechanism for the differentials field. This closes the existing gap as a side-effect: once the ScribeDraftReviewDialog exists and dispatches to SmartDiagnosis Redux state, useSmartDiagnosisPrefill finally has a consumer.

8. Audio safety contract

Audio → browser MediaStream → WebSpeechRecognition API → text transcript
  ↑ this boundary is LOCAL; audio bytes NEVER leave the browser
  
text transcript → [OPTIONAL] remote LLM (Ollama local or configured AI backend)
  ↑ only the TEXT crosses the network; no audio; same PHI stance as ClinicalNoteAIService

scribe_sessions table → stores: session_id, encounter_id (FK), accepted_draft JSONB,
  outcome (applied|discarded), created_at
  ↑ NO transcript stored in Supabase (same "no PHI in prompt" rule = no transcript at rest)
  
Transcript lifecycle:
  1. Captured in React state (AmbientScribeSessionContext) — tab memory only
  2. Sent to LLM for extraction (text only, de-identified per §5.1)
  3. Extraction results stored in ScribeDraftSlice
  4. On apply OR discard → transcript cleared from memory
  5. Audit row written (draft summary only, no raw transcript)

9. `scribe_sessions` table (Supabase)

CREATE TABLE IF NOT EXISTS scribe_sessions (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  user_id         UUID,
  encounter_id    TEXT NOT NULL,     -- de-identify at export boundary (ai-training-corpus.md)
  tenant_id       UUID,
  session_mode    TEXT NOT NULL,     -- 'ambient' | 'dictation' | 'snippet'
  duration_s      INT,               -- seconds of active listening
  word_count      INT,               -- transcript word count (no raw text stored)
  extraction_json JSONB,             -- the ScribeExtraction object (PHI — service_role only)
  outcome         TEXT NOT NULL,     -- 'applied' | 'discarded' | 'partial'
  fields_applied  TEXT[],            -- which fields the doctor confirmed
  fields_rejected TEXT[],            -- which fields the doctor rejected
  llm_model       TEXT,              -- which model was used
  llm_latency_ms  INT
);
ALTER TABLE scribe_sessions ENABLE ROW LEVEL SECURITY;
CREATE POLICY scribe_service ON scribe_sessions
  FOR ALL TO service_role USING (true) WITH CHECK (true);
-- No authenticated-role reads — this is PHI; served via RPC only.

10. ASSISTANT_REGISTRY entry

// src/services/ai/registry.ts  — add to ASSISTANT_REGISTRY

import { scribeDomain } from './ambient-scribe';

export const ASSISTANT_REGISTRY = {
  // ... existing entries ...
  scribe: scribeDomain,
} as const;

// src/services/ai/ambient-scribe/index.ts
export const scribeDomain = {
  name: 'ambient-scribe',
  description: 'Continuous consultation listener → draft extractor',
  runner: runAmbientScribeExtraction,
  tools: [chiefComplaintTool, hpiTool, vitalsNarratedTool, allergyTool, soapTool],
  // catalog tools (diagnosis + order) injected at runtime from the per-encounter context
  requiresLLM: true,
  fallback: 'local-regex',   // ClinicalNoteAIService regex path
};

11. Safety invariants

Audio never leaves the browser — the browser Web Speech API is local; only the resulting text transcript is processed further.
Draft-until-signed — no clinical data (diagnosis, orders, note text) is written to any store until the doctor explicitly confirms each field in the Review Dialog. The scribe is a pre-fill mechanism, not an auto-writer.
Fail-open — if the LLM is not configured (no Ollama, no VITE_LOCAL_LLM_URL), the scribe falls back to ClinicalNoteAIService’s local-regex extraction. The FAB still shows; it just runs the lightweight regex path. No error surfaced to doctor.
No PHI in Supabase transcript storage — raw transcript stays in browser memory only; cleared after apply/discard. Only the extracted structured draft + outcome metadata persists.
Scribe never gates anything — it can suggest, never block. Even if extraction fails, the doctor fills the form normally.
Per-tenant feature flag — scribe.enabled defaults false in the tenant config. Opt-in per deployment, consistent with the e-Kardex AI stance.
Structured extraction only from catalog — differentials and orders are resolved via the existing catalog adapters (SmartDiagnosis catalog → ICD-10/SNOMED, VoiceOrder catalog → /v2/medication/*), not free-form LLM generation. Hallucinated drug names cannot appear because the LLM never names drugs; it only identifies that the doctor mentioned “starting a blood pressure medication” and the catalog resolves the actual medication entity.
Named clinical owner required — same governance posture as the training corpus and e-Kardex AI docs.

12. Rollout

Phase	Deliverable	Risk
P0	`SpeechToText.tsx` continuous mode + `AmbientScribeSessionContext` + `TranscriptBuffer`	Low — browser API only, no LLM
P0b	Wire `useSmartDiagnosisPrefill` into existing `diagnosis-ai` Review Dialog (closes existing gap, zero new code)	Low — one missing import
P1	`runAmbientScribeExtraction` — calls `ClinicalNoteAIService` (always works) + catalog tools; `ScribeDraftSlice`; `ScribeDraftReviewDialog`	Medium — new UI surface
P2	`AmbientScribeFAB` mounted on OPD consultation page; `scribe_sessions` Supabase table + audit	Medium — needs LLM configured to be useful
P3	Full Ollama / LLM integration for HPI/SOAP extraction; register `scribeDomain` in `ASSISTANT_REGISTRY`; `runAgentLoop` replaces inline runner	Low — `runner-engine.ts` already written
P4	Training corpus integration — sessionized consultation transcripts (de-identified per `ai-training-corpus.md`) feed `ml_session_trajectories` for future scribe model fine-tuning	Requires T1 of training corpus

P0/P0b are safe, additive, and require no LLM. P0b specifically closes an existing bug (SmartDiagnosis results don’t reach the form) with minimal code.

docs/architecture/navigation-next-best-action.md — navigation recommender sharing the same 3-tier signal model and gate-ladder pattern
docs/architecture/ai-training-corpus.md — the training corpus that eventually receives scribe sessions (P4); PHI/de-id posture applies identically
docs/architecture/ekardex-from-journey-cache.md — AI safety stance (recommender-only, no auto-writes, no PHI in prompt, per-tenant flag, named clinical owner)
docs/architecture/rogue-user-detection-system.md — user_action_events audit pattern mirrored by scribe_sessions
docs/architecture/unified-clinical-assistant.md — the broader unified assistant design (the ambient scribe is the “front door” described there; this doc is the detailed spec)
src/services/ai/shared/runner-engine.ts — the runAgentLoop the scribe domain uses
src/services/ai/registry.ts — the ASSISTANT_REGISTRY the scribe domain registers into
src/services/ai/clinical-note-ai.service.ts — SOAP structurer reused as the local-regex fallback
src/services/ai/smart-diagnosis/diagnosis-catalog.ts — the differential catalog plugged in as a Phase-2 tool
src/services/ai/voice-order/rest-catalog.ts — the order catalog plugged in as a Phase-2 tool
src/common/components/shared-engine/speech-to-text/SpeechToText.tsx — extended with continuous mode in P0