medOS ultra

Vision AI Module Catalog

Catalog of vision-AI modules under the services/vision microservice.

11 min read diagramsUpdated 2026-05-25docs/architecture/vision-ai-module-catalog.md

Parent service: services/vision/ (Moleculer microservice) Pattern: Each module follows the surgical-count / nutrition-analysis blueprint: adapter (stub + real backend) → controller mixin → Supabase persistence → NATS broadcast → hospital_events → frontend service with fallback → miniapp + dialog wrapper.

Reference implementations:

  • YOLO object detection: modules/surgicalCount/ + vision-ai.service.ts
  • LLM vision analysis: modules/nutritionAnalysis/ + nutrition-ai.service.ts

1. Medication Tray Verification

Problem

Medication administration errors are among the top causes of preventable harm. Nurses prepare trays with multiple medications for multiple patients. A single swap — wrong drug, wrong dose, wrong patient — can be catastrophic. Today the check is manual (read label → compare to MAR → administer).

Concept

Nurse photographs the medication tray before administration. The LLM identifies each visible medication (pill shape/color, blister pack text, vial labels) and cross-references against the patient’s active MedicationRequest list from the e-MAR. Returns a pass/fail result with per-item confidence.

Key Output

{
  "verificationStatus": "pass | fail | partial",
  "identifiedMedications": [
    {
      "name": "Metformin 500mg",
      "nameLocal": "เมตฟอร์มิน 500 มก.",
      "matched": true,
      "matchedOrderId": "rx-12345",
      "confidence": 0.92
    }
  ],
  "missingFromTray": ["Amlodipine 5mg"],
  "extraOnTray": [],
  "patientId": "...",
  "encounterId": "...",
  "verifiedAt": "ISO timestamp"
}

Architecture

Nurse captures tray photo
  → POST /api/v2/vision/medication/verify
  → LLM adapter identifies medications from image
  → Reconcile vs active MedicationRequest list (fetched from medication service)
  → Persist to `medication_tray_verifications` table
  → Emit MEDICATION_TRAY_VERIFIED on NATS
  → hospital_events insert
  → If FAIL → trigger AcknowledgementRequest to charge nurse + pharmacist

Integration Points

  • e-MAR administration dialog — “Verify Tray” button before confirming administration
  • IPD medication cart workflow — scan before each med pass round
  • Pharmacy dispensing — verify filled cart against orders before sending to ward
  • Policy gate: medication.administration.tray_verified — block administration until AI verification passes

Config

Env Var Default Purpose
VISION_MED_VERIFY_ENABLED false Kill switch
VISION_MED_VERIFY_CONFIDENCE 0.75 Min confidence to count as “identified”

Files to Create

services/vision/src/api/vision/modules/medicationVerify/
  medicationVerify.controller.mixin.ts
services/vision/src/api/vision/modules/_shared/
  medVerifyAdapter.ts
web/src/services/medication-verify-ai.service.ts
web/packages/miniapps/medication-tray-verify/
  MedicationTrayVerify.tsx
  MedicationTrayVerifyDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_medication_tray_verifications.sql

2. Blood Bag / Transfusion Label Verification

Problem

Transfusion of the wrong blood product is a never-event. Current verification is two-nurse manual check of the bag label against the blood bank order. This is error-prone under time pressure (trauma, OR).

Concept

Nurse photographs the blood bag label. OCR extracts: blood type, unit number, component type, expiry date, crossmatch result. System cross-references against the active BloodProductDispense order. Returns match/mismatch with specific field-level comparison.

Key Output

{
  "matchStatus": "match | mismatch | partial",
  "extractedFields": {
    "unitNumber": "BB-2026-00451",
    "bloodType": "A Rh+",
    "component": "Packed RBC",
    "expiryDate": "2026-06-15",
    "crossmatchResult": "Compatible"
  },
  "orderComparison": {
    "unitNumber": { "expected": "BB-2026-00451", "match": true },
    "bloodType": { "expected": "A Rh+", "match": true },
    "expiry": { "expired": false }
  },
  "patientBloodType": "A Rh+",
  "allergenFlags": []
}

Integration Points

  • Blood bank dispense dialog — mandatory scan before release
  • Bedside transfusion initiation — second verification at patient
  • Blood bank module (@miniapps/blood-bank-*) — existing blood intake/dispense flows
  • Policy gate: blood_bank.transfusion.bag_verified

Files to Create

services/vision/src/api/vision/modules/bloodBagVerify/
  bloodBagVerify.controller.mixin.ts
  bloodBagOcrAdapter.ts
web/src/services/blood-bag-verify-ai.service.ts
web/packages/miniapps/blood-bag-verify/
  BloodBagVerify.tsx
  BloodBagVerifyDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_blood_bag_verifications.sql

3. Specimen Label QA

Problem

Mislabeled specimens cause wrong results on wrong patients. Pre-analytical errors account for 60-70% of all lab errors. Current check: lab tech visually compares tube label to requisition form.

Concept

Lab tech or phlebotomist photographs the labeled specimen tube(s). OCR reads patient HN, name, test codes, collection time, and barcode. System cross-references against the active LabRequest / LabSpecimen records. Flags mismatches before the specimen enters the analyzer.

Key Output

{
  "verificationStatus": "verified | mismatch | unreadable",
  "specimens": [
    {
      "tubeType": "EDTA (purple top)",
      "extractedHN": "HN-12345",
      "extractedName": "นายสมชาย",
      "extractedTests": ["CBC", "BUN"],
      "matchedLabRequestId": "lr-789",
      "labelsMatch": true,
      "barcodeValue": "SPE-2026-0891"
    }
  ],
  "missingSpecimens": [],
  "extraSpecimens": []
}

Integration Points

  • Lab specimen collection workflow — scan after labeling, before transport
  • Lab receiving desk — second scan on arrival at lab
  • Lab results entry — optional re-verification before reporting
  • Existing lab data pipeline (handleLabSpecimenPipeline) — inject verification event

Files to Create

services/vision/src/api/vision/modules/specimenLabelQa/
  specimenLabelQa.controller.mixin.ts
  specimenOcrAdapter.ts
web/src/services/specimen-label-ai.service.ts
web/packages/miniapps/specimen-label-verify/
  SpecimenLabelVerify.tsx
  SpecimenLabelVerifyDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_specimen_label_verifications.sql

4. Order Request Card OCR

Problem

Many Thai hospitals still use handwritten or pre-printed order request cards (ใบสั่งยา, ใบสั่งตรวจ). Clerks manually transcribe these into the digital system — slow, error-prone, and a bottleneck during peak hours.

Concept

Staff photographs the order card. LLM reads handwriting + printed text, extracts structured order data: medication name, dose, route, frequency, quantity, prescriber signature, patient HN. Pre-fills the digital order form for clerk review and confirmation.

Key Output

{
  "cardType": "medication | lab | imaging | nutrition",
  "extractedOrders": [
    {
      "itemName": "Amoxicillin 500mg",
      "dose": "500mg",
      "route": "PO",
      "frequency": "TID",
      "quantity": 21,
      "duration": "7 days",
      "confidence": 0.88
    }
  ],
  "patientHN": "HN-67890",
  "prescriberName": "นพ.สมศักดิ์",
  "orderDate": "2026-05-24",
  "handwritingQuality": "legible | partial | illegible",
  "flaggedItems": ["Dosage unusually high for Amoxicillin — please verify"]
}

Integration Points

  • Order entry dialog — “Scan Order Card” button pre-fills form fields
  • Central order manager — batch scan multiple order cards
  • Pharmacy receiving — verify printed prescription against digital order
  • Nutrition request — scan dietary order card

Files to Create

services/vision/src/api/vision/modules/orderCardOcr/
  orderCardOcr.controller.mixin.ts
  orderCardAdapter.ts
web/src/services/order-card-ocr-ai.service.ts
web/packages/miniapps/order-card-scanner/
  OrderCardScanner.tsx
  OrderCardScannerDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_order_card_scans.sql

5. Vital Signs Monitor OCR

Problem

Nurses manually read bedside monitors and type values into the observation form — BP, HR, SpO2, RR, temperature. This is done dozens of times per shift across all patients. Manual transcription is slow and error-prone (digit transposition is common: 120/80 → 102/80).

Concept

Nurse photographs the bedside monitor screen. LLM reads the displayed values and returns structured vital signs. Values auto-populate the observation form. Nurse reviews and confirms with one tap.

Key Output

{
  "vitals": {
    "systolicBP": 120,
    "diastolicBP": 78,
    "heartRate": 72,
    "spO2": 98,
    "respiratoryRate": 16,
    "temperature": 36.8,
    "etCO2": null
  },
  "monitorBrand": "Philips IntelliVue",
  "readConfidence": 0.95,
  "alarmActive": false,
  "waveformDetected": ["ECG Lead II", "SpO2 pleth"],
  "timestamp": "2026-05-24T14:30:00Z"
}

Integration Points

  • Graphic sheet / vital signs form — “Scan Monitor” button auto-fills all fields
  • Nursing observation workflow — replaces manual entry
  • CDS engine integration — auto-fire CDS rules on captured vitals (NEWS2, MEWS)
  • Anesthesia record — continuous capture during procedures

Files to Create

services/vision/src/api/vision/modules/vitalSignsOcr/
  vitalSignsOcr.controller.mixin.ts
  monitorOcrAdapter.ts
web/src/services/vitals-monitor-ai.service.ts
web/packages/miniapps/vitals-monitor-scan/
  VitalsMonitorScan.tsx
  VitalsMonitorScanDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_vitals_monitor_scans.sql

6. Drug Vial / Ampoule OCR

Problem

Medication errors from look-alike/sound-alike (LASA) drugs are a persistent safety issue. Nurses and pharmacists visually verify vial labels under time pressure. Small text, similar packaging, and poor lighting increase risk.

Concept

Staff photographs the medication vial or ampoule. OCR extracts: drug name, concentration, lot number, expiry date, manufacturer. System checks against the hospital drug master and flags LASA alerts, expired stock, or recalled lots.

Key Output

{
  "drugName": "Adrenaline (Epinephrine)",
  "concentration": "1:1000 (1mg/mL)",
  "volume": "1mL",
  "lotNumber": "L2026-0451",
  "expiryDate": "2027-03-15",
  "manufacturer": "GPO",
  "expired": false,
  "recallAlert": false,
  "lasaWarning": ["Looks similar to: Atropine 1mg/mL — verify before use"],
  "matchedDrugMasterId": "drug-12345",
  "barcodeValue": "8858861..."
}

Integration Points

  • Medication preparation workflow — scan before drawing up
  • Pharmacy dispensing — verify stock during pick
  • Inventory management — scan during stock count to verify expiry
  • Drug master setup — bulk-scan new stock for registration

Files to Create

services/vision/src/api/vision/modules/drugVialOcr/
  drugVialOcr.controller.mixin.ts
  drugVialAdapter.ts
web/src/services/drug-vial-ai.service.ts
web/packages/miniapps/drug-vial-scanner/
  DrugVialScanner.tsx
  DrugVialScannerDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_drug_vial_scans.sql

7. Wound Assessment

Problem

Wound documentation is subjective (“healing well”, “looks infected”) and inconsistent across shifts. Measurement is manual (ruler against wound). Tracking healing progression over time requires comparing narrative notes, which is unreliable.

Concept

Nurse photographs the wound with a reference scale marker (ruler or standard color/size card). LLM estimates wound dimensions, describes tissue type (granulation, slough, necrotic, epithelial), drainage characteristics, and surrounding skin condition. Each photo links to the encounter and builds a visual timeline for healing progression comparison.

Key Output

{
  "woundType": "pressure_ulcer | surgical | diabetic | traumatic | burn",
  "stage": "Stage III",
  "dimensions": {
    "lengthCm": 4.2,
    "widthCm": 3.1,
    "depthCm": 0.8,
    "areaCm2": 13.02
  },
  "tissueComposition": {
    "granulation": 60,
    "slough": 25,
    "necrotic": 10,
    "epithelial": 5
  },
  "drainage": { "amount": "moderate", "type": "serosanguinous" },
  "surroundingSkin": "erythema within 2cm margin",
  "infectionIndicators": ["periwound warmth", "increased drainage"],
  "healingTrend": "stable",
  "notes": "Wound bed predominantly granulating with moderate slough...",
  "notesLocal": "แผลมีเนื้อเยื่อสร้างใหม่เป็นส่วนใหญ่..."
}

Integration Points

  • Wound care module / nursing note — “Photograph Wound” button
  • Patient profile wound tracking tab — visual timeline with before/after
  • Wound care consultation request — attach AI assessment
  • Surgical follow-up — post-op wound monitoring

Files to Create

services/vision/src/api/vision/modules/woundAssessment/
  woundAssessment.controller.mixin.ts
  woundAnalysisAdapter.ts
web/src/services/wound-assessment-ai.service.ts
web/packages/miniapps/wound-assessment/
  WoundAssessment.tsx
  WoundAssessmentDialog.tsx
  WoundTimeline.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_wound_assessments.sql

8. Patient Wristband Verification

Problem

Patient identification errors occur when staff skip the manual wristband check or when wristbands are damaged/smudged. The Joint Commission lists patient identification as the #1 National Patient Safety Goal.

Concept

Staff scans the patient wristband (barcode, QR code, or printed text) using the device camera. System extracts patient HN, name, date of birth, and allergies. Cross-references against the current encounter and flags mismatches. Works alongside the existing QR self-service pattern.

Key Output

{
  "extractedHN": "HN-12345",
  "extractedName": "นายสมชาย ใจดี",
  "extractedDOB": "1985-03-15",
  "extractedAllergies": ["Penicillin"],
  "barcodeValue": "HN12345-ENC67890",
  "matchStatus": "match | mismatch",
  "matchDetails": {
    "hn": true,
    "name": true,
    "dob": true,
    "encounter": true
  },
  "wristbandCondition": "good | faded | damaged"
}

Integration Points

  • Pre-procedure verification — bedside scan before any intervention
  • Medication administration — scan wristband as part of 5 Rights check
  • Blood transfusion — identity verification step
  • Specimen collection — verify patient before drawing
  • Ties into existing patient-qr-self-service-pattern.md

Files to Create

services/vision/src/api/vision/modules/wristbandVerify/
  wristbandVerify.controller.mixin.ts
  wristbandScanAdapter.ts
web/src/services/wristband-verify-ai.service.ts
web/packages/miniapps/wristband-verify/
  WristbandVerify.tsx
  WristbandVerifyDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_wristband_verifications.sql

9. Inventory Shelf Scan

Problem

Manual inventory counting is time-consuming (monthly cycle counts take days), inaccurate (miscounts are common), and disrupts operations. Low-stock situations are discovered too late, causing stockouts.

Concept

Staff photographs supply shelves or storage bins. LLM identifies visible items, estimates quantities, and compares against expected inventory levels. Flags low-stock items for reorder and discrepancies for investigation.

Key Output

{
  "shelfId": "shelf-pharmacy-A3",
  "location": "Main Pharmacy, Shelf A3",
  "scannedItems": [
    {
      "itemName": "Normal Saline 0.9% 1000mL",
      "estimatedCount": 12,
      "expectedCount": 20,
      "status": "low",
      "reorderSuggested": true
    },
    {
      "itemName": "IV Set (macro drip)",
      "estimatedCount": 45,
      "expectedCount": 50,
      "status": "adequate",
      "reorderSuggested": false
    }
  ],
  "discrepancies": 3,
  "totalItemsScanned": 8,
  "coveragePct": 85,
  "scanConfidence": 0.78
}

Integration Points

  • Inventory module (@miniapps/inventory-*) — “Scan Shelf” button on count page
  • Ward supply cabinet — nurse quick-scan for reorder request
  • Pharmacy stock management — daily visual count augmentation
  • Central supply — receiving verification

Files to Create

services/vision/src/api/vision/modules/inventoryScan/
  inventoryScan.controller.mixin.ts
  inventoryScanAdapter.ts
web/src/services/inventory-scan-ai.service.ts
web/packages/miniapps/inventory-shelf-scan/
  InventoryShelfScan.tsx
  InventoryShelfScanDialog.tsx
  index.ts
infrastructure/medbase/migrations/
  YYYYMMDD_inventory_shelf_scans.sql

Shared Infrastructure

All 9 modules above share:

Component Location Purpose
Vision service services/vision/visionService.ts Host — add each module as a mixin
LLM adapter modules/_shared/llmAdapter.ts Shared Anthropic/OpenAI calling pattern
Supabase client modules/_shared/supabaseClient.ts Shared persistence layer
Config ever.config/IConfig.ts Shared env var pattern
NATS events Broker broadcast Per-module event type
hospital_events Supabase insert Encounter orchestrator integration
IPFS upload nutrition-ai.service.ts pattern Photo persistence for clinical record
Dialog wrapper *Dialog.tsx pattern Reusable MUI Dialog for embedding
DynamicCoreApp enum DynamicCoreApp.ts Module registration
DynamicContentRenderer Switch case Patient profile rendering

Priority Order for Implementation

Priority Module Rationale
P0 Medication Tray Verification Direct patient safety — prevents wrong drug/dose
P0 Blood Bag Verification Never-event prevention — wrong blood kills
P1 Specimen Label QA Pre-analytical error is 60-70% of lab errors
P1 Order Card OCR Operational efficiency — saves hours of transcription
P1 Vital Signs Monitor OCR High-frequency task — every nurse, every shift
P2 Drug Vial OCR LASA prevention + expiry tracking
P2 Wound Assessment Clinical documentation quality + healing tracking
P2 Wristband Verification Identity safety — ties into existing QR pattern
P3 Inventory Shelf Scan Operational — less clinical urgency

Agent Assignment Pattern

Each module is independent and can be built by a separate agent in parallel. The agent prompt should reference:

  1. This document for the module spec
  2. services/vision/src/api/vision/modules/nutritionAnalysis/ as the reference implementation
  3. web/src/services/nutrition-ai.service.ts as the frontend service pattern
  4. web/packages/miniapps/nutrition-intake-ai/ as the miniapp + dialog pattern
  5. infrastructure/medbase/migrations/20260524a_nutrition_meal_analysis.sql as the migration pattern
Ask Anything