Device Screen Reader

Premium add-on. Feature-flagged behind medos_premium_device_reader. Replaces $50-200k HL7/serial device integration fees with a $30 webcam + YOLO screen detection + OCR + LLM parsing pipeline.

Problem Statement

Medical device integration (infusion pumps, anesthesia machines, patient monitors, ventilators, syringe drivers, dialysis machines) requires:

• Per-device vendor licenses ($5-20k each)
• Protocol adapters (HL7v2, serial RS-232, proprietary)
• Vendor certification and annual maintenance
• Dedicated integration engine hardware

Most hospitals (especially in SE Asia) can’t afford this. Nurses manually transcribe device readings every 15-60 minutes — error-prone and slow.

The fix: point a camera at the device display → AI reads the screen → structured observations flow into the EMR automatically.

Supported Device Types

Device Type	What We Read	Brands Supported (LLM-flexible)
Infusion Pump	Rate (ml/hr), VTBI, volume infused, drug name, concentration, mode, channel, alarms	Baxter Sigma, B.Braun Infusomat, Fresenius Agilia, BD Alaris, Terumo TE-LM
Syringe Driver	Rate (ml/hr), volume remaining, drug, concentration	B.Braun Perfusor, Fresenius Injectomat, Terumo TE-SS
Anesthesia Machine	TV, RR, ETCO2, FiO2, MAC, agent %, airway pressure (peak/mean/PEEP), fresh gas flow, I:E ratio	GE Aisys, Dräger Perseus/Primus, Mindray A7/A9, Penlon Prima
Patient Monitor	HR, SpO2, NIBP (sys/dia/mean), temp, RR, IBP, CVP	Philips IntelliVue, GE CARESCAPE, Mindray BeneVision, Nihon Kohden
Ventilator	Mode, TV, RR, PEEP, FiO2, PIP, Pplat, minute volume, compliance, I:E	Hamilton G5, Dräger Evita, Medtronic PB980, Maquet Servo-u
Dialysis Machine	UF rate, UF volume, blood flow rate, dialysate flow, TMP, time remaining	Fresenius 5008, Nikkiso DBB-07, Baxter AK 98
Pulse Oximeter	SpO2, pulse rate, PI, pleth waveform amplitude	Masimo Radical, Nellcor, Nonin
Glucometer	Blood glucose (mg/dL or mmol/L), timestamp	Accu-Chek, OneTouch, FreeStyle

The LLM-based parsing means we don’t need per-brand adapters — the LLM understands display layouts from its training data. New brands work out of the box; the user just tells the system what device type it is.

Architecture

Camera (USB/IP/phone)
       │ frame every N seconds
       ▼
┌──────────────────────────────────────────────────────────┐
│  Stage 1 — Screen Detection (YOLO)                       │
│                                                          │
│  Input:  raw camera frame (640×480+)                     │
│  Output: cropped screen region + device_type hint        │
│                                                          │
│  The YOLO model detects rectangular screen regions in    │
│  the frame. Multiple screens OK (split-screen monitors). │
│  Stub: returns the full frame as a single screen region. │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  Stage 2 — OCR (PaddleOCR / Tesseract / Cloud Vision)   │
│                                                          │
│  Input:  cropped screen region                           │
│  Output: raw text blocks with positions                  │
│                                                          │
│  Extracts all visible text from the device display.      │
│  Position info helps the LLM understand spatial layout.  │
│  Stub: returns realistic OCR text for the device type.   │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  Stage 3 — LLM Structured Parsing                        │
│                                                          │
│  Input:  raw OCR text + device_type                      │
│  Output: DeviceReading (structured vital signs / params) │
│                                                          │
│  The LLM knows medical device display conventions:       │
│    "Rate 125▼ ml/hr VTBI 450 ml" → infusionRate: 125    │
│    "HR 72 SpO2 98% BP 120/80(93)" → hr:72, spo2:98, ... │
│                                                          │
│  Also detects alarm states from OCR (flashing text,      │
│  "ALARM", "OCCLUSION", "AIR IN LINE", "LOW BATTERY").   │
│                                                          │
│  Confidence score per extracted value based on OCR       │
│  clarity + LLM parsing certainty.                        │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  Stage 4 — Validation + Write                            │
│                                                          │
│  1. Range check — HR 72 is plausible, HR 720 is not     │
│  2. Delta check — if HR was 72 last read and is now 170,│
│     flag as suspicious (possible OCR misread)            │
│  3. Write to Supabase `device_screen_readings` (audit)   │
│  4. Write to `observations` (same table the vitals       │
│     dashboard, CDS engine, and e-MAR already consume)   │
│  5. Emit `manifest.device.observation` (existing event   │
│     type — the encounter orchestrator already handles it)│
│  6. If alarm detected → emit `manifest.clinical.alert`   │
│                                                          │
│  Source field: 'vision-ocr' (distinguishes from          │
│  'device-hl7' or 'manual-entry')                        │
└──────────────────────────────────────────────────────────┘

Data Model

Supabase: device_screen_readings (audit trail)

create table device_screen_readings (
  id                  uuid primary key default gen_random_uuid(),
  reading_uid         text unique not null,
  device_type         text not null,        -- infusion_pump, anesthesia, monitor, ventilator, dialysis, syringe_driver, pulse_oximeter, glucometer
  device_brand        text,                 -- e.g. 'baxter_sigma', 'ge_carescape' (optional hint)
  device_location     text,                 -- e.g. 'OR-3 left pump', 'Bed 12A monitor'
  encounter_id        text,
  patient_id          text,
  bed_id              text,

  -- Camera frame
  frame_uri           text,                 -- IPFS / S3 link to captured frame
  frame_width         int,
  frame_height        int,
  screen_bbox         jsonb,                -- {x, y, width, height} of detected screen region

  -- OCR output
  ocr_raw_text        text,                 -- raw extracted text
  ocr_confidence      numeric(4,3),         -- overall OCR confidence
  ocr_engine          text,                 -- 'paddle', 'tesseract', 'cloud_vision', 'stub'

  -- LLM parsed output
  parsed_values       jsonb not null,       -- structured readings (see DeviceReading below)
  llm_model           text,
  llm_latency_ms      int,
  parse_confidence     numeric(4,3),

  -- Validation
  validation_status   text default 'pending' check (validation_status in ('pending','accepted','rejected','suspicious')),
  validation_flags    jsonb default '[]',   -- [{field, issue, detail}]
  validated_by        text,
  validated_at        timestamptz,

  -- Alarm detection
  alarm_detected      boolean default false,
  alarm_type          text,                 -- 'occlusion', 'air_in_line', 'low_battery', 'high_pressure', etc.
  alarm_message       text,

  tenant_id           uuid,
  created_at          timestamptz default now(),
  updated_at          timestamptz default now()
);

DeviceReading (parsed_values shape per device type)

// Infusion pump
interface InfusionPumpReading {
  type: 'infusion_pump';
  rate?: number;           // ml/hr
  rateUnit?: string;       // 'ml/hr', 'mcg/kg/min', etc.
  vtbi?: number;           // ml remaining
  volumeInfused?: number;  // ml
  drugName?: string;
  concentration?: string;  // e.g. '1mg/ml'
  mode?: string;           // 'rate', 'dose', 'taper', 'bolus'
  channel?: string;        // 'A', 'B', '1', '2'
  batteryPercent?: number;
  alarm?: string;
}

// Anesthesia machine
interface AnesthesiaReading {
  type: 'anesthesia';
  tidalVolume?: number;       // ml
  respiratoryRate?: number;   // breaths/min
  etco2?: number;             // mmHg
  fio2?: number;              // %
  macValue?: number;          // MAC
  agentName?: string;         // sevoflurane, desflurane, isoflurane
  agentPercent?: number;      // inspired/expired %
  peakPressure?: number;      // cmH2O
  meanPressure?: number;
  peep?: number;              // cmH2O
  freshGasFlow?: number;      // L/min
  ieRatio?: string;           // '1:2'
  minuteVolume?: number;      // L/min
  compliance?: number;        // ml/cmH2O
}

// Patient monitor
interface PatientMonitorReading {
  type: 'patient_monitor';
  hr?: number;              // bpm
  spo2?: number;            // %
  nibpSystolic?: number;    // mmHg
  nibpDiastolic?: number;
  nibpMean?: number;
  temperature?: number;     // °C
  respiratoryRate?: number;
  ibpSystolic?: number;     // arterial line
  ibpDiastolic?: number;
  ibpMean?: number;
  cvp?: number;             // cmH2O
  etco2?: number;
}

// Ventilator
interface VentilatorReading {
  type: 'ventilator';
  mode?: string;             // SIMV, AC, PS, CPAP, PRVC, etc.
  tidalVolume?: number;
  respiratoryRate?: number;
  setRR?: number;
  peep?: number;
  fio2?: number;
  pip?: number;              // peak inspiratory pressure
  pplat?: number;            // plateau pressure
  minuteVolume?: number;
  compliance?: number;
  ieRatio?: string;
  inspTime?: number;         // seconds
}

// Dialysis
interface DialysisReading {
  type: 'dialysis';
  ufRate?: number;           // ml/hr
  ufVolume?: number;         // ml removed
  ufGoal?: number;           // ml target
  bloodFlowRate?: number;    // ml/min
  dialysateFlowRate?: number;
  tmp?: number;              // transmembrane pressure
  timeRemaining?: string;    // 'HH:MM'
  arterialPressure?: number;
  venousPressure?: number;
}

// Syringe driver
interface SyringeDriverReading {
  type: 'syringe_driver';
  rate?: number;
  rateUnit?: string;
  volumeRemaining?: number;
  drugName?: string;
  concentration?: string;
  alarm?: string;
}

// Pulse oximeter
interface PulseOximeterReading {
  type: 'pulse_oximeter';
  spo2?: number;
  pulseRate?: number;
  perfusionIndex?: number;
}

// Glucometer
interface GlucometerReading {
  type: 'glucometer';
  glucose?: number;
  glucoseUnit?: string;     // 'mg/dL' or 'mmol/L'
}

type DeviceReading =
  | InfusionPumpReading
  | AnesthesiaReading
  | PatientMonitorReading
  | VentilatorReading
  | DialysisReading
  | SyringeDriverReading
  | PulseOximeterReading
  | GlucometerReading;

LLM Prompt Template

You are a medical device display reader. You receive raw OCR text extracted
from a {{device_type}} screen. Parse it into structured JSON.

## Device type: {{device_type}}
## Device brand hint: {{device_brand}} (may be empty — infer from display layout)
## Device location: {{device_location}}

## Raw OCR text (may contain noise, partial characters, misreads):
{{ocr_raw_text}}

## Previous reading (for delta validation — may be null):
{{previous_reading | json}}

## Instructions:
1. Extract every numeric reading visible on the display
2. Map each to the correct clinical parameter (e.g. "72" next to a heart
   icon or "HR" label → hr: 72)
3. Include units when visible
4. Detect alarm states (text like "ALARM", "OCCLUSION", "AIR IN LINE",
   "LOW BATTERY", flashing indicators described as "***" or "!!!")
5. Assign a confidence (0.0–1.0) to each extracted value based on OCR
   clarity (garbled text → low confidence)
6. If a value changed dramatically from previous_reading (e.g. HR 72→720),
   flag it as suspicious (possible OCR misread of "72" as "720")

Return JSON:
{
  "type": "{{device_type}}",
  // ...all extracted fields for this device type...
  "confidence": 0.0-1.0,    // overall parse confidence
  "alarm": "OCCLUSION" | null,
  "alarmSeverity": "critical" | "warning" | null,
  "suspicious": [            // fields that look like OCR misreads
    { "field": "hr", "value": 720, "previousValue": 72, "reason": "10x jump" }
  ],
  "rawFieldMap": {           // for audit: which OCR text mapped to which field
    "hr": "HR 72",
    "spo2": "SpO2 98%"
  }
}

Validation Rules (per device type)

Parameter	Plausible Range	Suspicious Delta
HR	20–250 bpm	>50 bpm change
SpO2	50–100 %	>15% drop
BP systolic	40–300 mmHg	>60 mmHg change
BP diastolic	20–200 mmHg	>40 mmHg change
Temperature	30–42 °C	>2°C change
RR	4–60 breaths/min	>20 change
Infusion rate	0.1–2000 ml/hr	>100% change
ETCO2	10–80 mmHg	>20 change
FiO2	21–100 %	—
Tidal volume	50–2000 ml	>50% change
PEEP	0–30 cmH2O	>10 change

Values outside plausible range → validation_status: 'rejected' Values with suspicious delta → validation_status: 'suspicious'

Implementation Plan

Phase 1 — Backend

File	Purpose
modules/deviceReader/deviceReader.controller.mixin.ts	vision.deviceReader.read — chains YOLO → OCR → LLM → validate → persist
modules/deviceReader/ocrAdapter.ts	OCR layer (stub → PaddleOCR → Cloud Vision)
modules/deviceReader/deviceParserPrompt.ts	Per-device-type LLM prompt builder
modules/deviceReader/validationRules.ts	Plausible ranges + delta checks
modules/deviceReader/deviceTypes.ts	TypeScript interfaces for all 8 device reading types

Phase 2 — Database

File	Purpose
migrations/20260524e_device_screen_readings.sql	Table + indexes + RLS

Phase 3 — Frontend

File	Purpose
web/packages/miniapps/e-mar/components/DeviceScreenReader.tsx	Camera panel + device type selector + live reading display + validation status
web/src/services/vision-ai.service.ts	readDeviceScreen() client function

Phase 4 — Observation integration

Write accepted readings to the existing observations table with source: 'vision-ocr'. The vitals dashboard, CDS engine, NEWS2/MEWS scoring, and e-MAR timeline already consume observations — zero wiring needed on the read side.

Configuration

Env var	Default	Purpose
VISION_DEVICE_READER_ENABLED	false	Kill switch
VISION_OCR_ENGINE	stub	stub, paddle, tesseract, cloud_vision
VISION_DEVICE_READ_INTERVAL_SEC	30	Auto-read frequency in continuous mode

Feature Flags

• Backend: VISION_DEVICE_READER_ENABLED env var
• Frontend: localStorage.getItem('medos_premium_device_reader') === 'true'

Cost Model

Component	Cost per read
YOLO screen detection	~$0 (on-device)
OCR (PaddleOCR local)	~$0 (on-prem)
OCR (Google Cloud Vision)	~$0.0015
LLM (gpt-4o-mini)	~$0.001
Total per read	~$0.001–0.0025

At 4 reads/hr × 20 beds × 24hr = 1,920 reads/day → ~$2-5/day with cloud, $0 with Ollama + PaddleOCR on-prem.

vs Traditional Integration

	Camera + AI	HL7 Device Integration
Setup cost	$30 webcam + software	$50-200k per facility
Per-device cost	$0	$5-20k license/device
New device support	Instant (LLM adapts)	Months (protocol dev)
Latency	2-5s per read	Real-time
Alarm relay	OCR-based (2-5s delay)	Native (instant)
Accuracy	95-99% (depends on screen clarity)	100% (digital)
Best for	Charting, trending, CDS	Critical alarms, closed-loop