Vision AI Integration

medOS includes a built-in computer-vision microservice that uses YOLO-family object detection models to verify physical objects in clinical workflows. It scans items via camera, reconciles detections against expected lists, and gates clinical actions through the policy engine.

Modules

Module	What it scans	Clinical value
Surgical Count	Instruments on the surgical table	Prevents retained foreign objects (AORN compliance)
Pharmacy Verify	Medication tray/cart (OPD + IPD)	Catches mispicks before dispensing
Specimen QA	Lab tubes by cap color + label	Prevents wrong-patient/wrong-tube errors
Blood Bank Verify	Blood product bags + ABO labels	Prevents fatal transfusion reactions
Wristband ID	Patient wristband QR/barcode + text	Right-patient verification at bedside
Wound Assessment	Wound photos with ruler reference	Auto-measure + classify tissue for longitudinal tracking
Sterilization QA	CSSD indicator tape + seal + label	Catches sterilization failures before case start

How It Works

sequenceDiagram
    participant Device as Camera Device
    participant API as Vision Service
    participant Model as Inference Backend
    participant DB as Supabase
    participant UI as Frontend

    Device->>API: POST /vision/{module}/scan (frame + expected items)
    API->>Model: Detect objects in frame
    Model-->>API: detections[] with bboxes + confidence
    API->>API: Reconcile (missing / extra / mismatch)
    API->>DB: Persist scan result (fail-soft)
    API->>API: Broadcast NATS event + hospital_events
    API-->>Device: ScanResult (detected, missing, status)
    UI->>DB: Realtime subscription
    DB-->>UI: Scan result appears in dashboard

Inference Backends

The backend is pluggable — one config change deploys all modules against a real model:

Backend	Config	Use case
Stub	inferenceBackend: 'stub'	Demo & development. No model needed — produces realistic simulated detections.
ONNX	inferenceBackend: 'onnx'	On-premise / edge. Runs YOLOv8 via onnxruntime-node locally.
Remote	inferenceBackend: 'remote'	Cloud GPU. POSTs frames to NVIDIA Triton or custom YOLO server.

Frontend Components

Each module ships a standalone React component that handles camera access, scanning, bbox overlay, results display, and pharmacist/clinician sign-off:

import { OpdPharmacyAiAssist } from '@medical-kit/pharmacy-verify';
import { SpecimenQaAiAssist } from '@medical-kit/lab-specimen-qa';
import { BloodBankAiAssist } from '@medical-kit/blood-bank-verify';
import { WristbandIdAiAssist } from '@medical-kit/wristband-id';
import { WoundAssessAiAssist } from '@medical-kit/wound-assess';
import { SterilizationQaAiAssist } from '@medical-kit/sterilization-qa';

All components fall back to local simulation when the backend is unreachable — the clinical workflow never stalls on connectivity.

Policy Gate Integration

Scan results feed into the configurable policy_gates engine (admin-managed at /admin/policy-gates). Example gates:

• Cannot close surgical case unless AI count is verified
• Cannot dispense medication unless pharmacy pick is verified
• Cannot transfuse unless blood type match is confirmed
• Cannot use sterile pack if indicator tape shows failure

Robo-Dog Patrol

The vision platform supports autonomous robotic patrol — a quadruped robot (Spot, Unitree Go2) walks the hospital on a schedule and performs scans at each waypoint.

How it connects

The robot is just another scan source. It hits the same POST /vision/{module}/scan endpoints as a human-held tablet. The backend doesn’t distinguish the caller.

Robo-dog (Jetson + Camera + Nav)
  → Walks to waypoint (pharmacy shelf, blood bank fridge, CSSD room)
  → Captures frame
  → Runs local YOLO (or sends to cloud)
  → POSTs result to medOS vision service
  → Dashboard shows: "Robot scan at Pharmacy Station B — all clear ✓"

Patrol modes

Mode	Description
Scheduled	Cron-based (e.g., every 4 hours audit pharmacy shelves)
On-demand	Nurse/pharmacist taps “Robot scan” button in dashboard
Event-triggered	Auto-dispatched when a stock alert fires or a discrepancy needs re-check

Planned waypoint types

• Pharmacy controlled substance cabinets (night audit)
• Blood bank fridges (inventory + expiry check)
• CSSD sterilization area (pack indicator spot checks)
• Ward corridors (wristband confirmation rounds)
• Wound care rooms (follow-up photo documentation)

Safety

• Speed-limited to 1.0 m/s (0.5 m/s near beds)
• LiDAR + depth camera obstacle avoidance
• Night patrol priority (22:00–06:00)
• UV-C sterilizable shell
• No facial recognition — wristband text/QR only
• Frames deleted after inference (never stored on robot)

See docs/architecture/robo-dog-patrol-integration.md for full technical design.

REST API Reference

All modules follow the same pattern:

POST /api/v2/vision/{module}/scan        — run AI scan
POST /api/v2/vision/{module}/:scanUid/verify — clinician sign-off

Module	Path prefix
Surgical Count	/vision/surgical-count
Pharmacy Verify	/vision/pharmacy-verify
Specimen QA	/vision/specimen-qa
Blood Bank	/vision/blood-bank-verify
Wristband ID	/vision/wristband-id
Wound Assess	/vision/wound-assess
Sterilization QA	/vision/sterilization-qa

Database Tables

Each module persists scan results to its own Supabase table for audit + analytics:

• surgical_count_scan_results
• pharmacy_verify_scan_results
• specimen_qa_scan_results
• blood_bank_verify_scan_results
• wristband_id_scan_results
• wound_assess_scan_results
• sterilization_qa_scan_results

All tables have RLS (authenticated read, service_role write) and updated_at triggers.

Sandbox Testing

Each module has a sandbox target for isolated development:

http://localhost:5179/?target=SurgicalTableLayout
http://localhost:5179/?target=PharmacyAiAssist
http://localhost:5179/?target=SpecimenQaAiAssist
http://localhost:5179/?target=BloodBankAiAssist
http://localhost:5179/?target=WristbandIdAiAssist
http://localhost:5179/?target=WoundAssessAiAssist
http://localhost:5179/?target=SterilizationQaAiAssist