Vision AI (YOLO) Platform

Overview

The services/vision/ microservice provides a unified computer-vision inference platform for clinical workflows. It uses YOLO-family object detection models to scan physical objects (instruments, medications, specimens, blood bags, wristbands, wounds, sterilization packs) via camera capture, reconcile detections against expected item lists, and gate clinical actions via the policy engine.

All 7 modules share the same pluggable architecture — swapping a stub simulator for a real ONNX or remote model is a one-file change per module.

Module Registry

#	Module	REST Prefix	Label Space	Phases	NATS Event	Table
1	Surgical Count	/vision/surgical-count	22 AORN instruments	initial, first_procedure, closing, final	OR_COUNT_SCANNED	surgical_count_scan_results
2	Pharmacy Verify	/vision/pharmacy-verify	22 medication forms	picking, verification, final_check	PHARMACY_PICK_SCANNED	pharmacy_verify_scan_results
3	Specimen QA	/vision/specimen-qa	14 tube types (by color)	pre_collection, post_collection, pre_transport	LAB_SPECIMEN_SCANNED	specimen_qa_scan_results
4	Blood Bank Verify	/vision/blood-bank-verify	18 (bags + ABO labels)	crossmatch_check, pre_transfusion, bedside_verify	BLOOD_BANK_UNIT_SCANNED	blood_bank_verify_scan_results
5	Wristband ID	/vision/wristband-id	12 (bands + QR + text)	medication_admin, procedure, specimen_collection, transfusion, general_verify	PATIENT_ID_VERIFIED	wristband_id_scan_results
6	Wound Assessment	/vision/wound-assess	15 tissue features	initial_assessment, follow_up, post_debridement, discharge	WOUND_ASSESSED	wound_assess_scan_results
7	Sterilization QA	/vision/sterilization-qa	15 pack features	pre_case_check, opening_verification, spot_check	STERILIZATION_PACK_SCANNED	sterilization_qa_scan_results

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│  SCAN SOURCE (any of)                                               │
│  ┌─────────┐  ┌────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │ Tablet  │  │ Fixed Cam  │  │ Robo-Dog     │  │ Phone PWA    │  │
│  │ (human) │  │ (ceiling)  │  │ (patrol)     │  │ (bedside)    │  │
│  └────┬────┘  └─────┬──────┘  └──────┬───────┘  └──────┬───────┘  │
└───────┼──────────────┼────────────────┼─────────────────┼──────────┘
        │              │                │                  │
        ▼              ▼                ▼                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│  POST /api/v2/vision/{module}/scan                                  │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │  Vision Service (Moleculer)                                 │    │
│  │  ┌──────────────────────────────────────────────────────┐   │    │
│  │  │  Inference Adapter (per module)                      │   │    │
│  │  │  ┌──────────┐  ┌──────────┐  ┌───────────────────┐  │   │    │
│  │  │  │  stub    │  │  onnx    │  │  remote (Triton)  │  │   │    │
│  │  │  │ (demo)   │  │ (edge)   │  │  (GPU cluster)    │  │   │    │
│  │  │  └──────────┘  └──────────┘  └───────────────────┘  │   │    │
│  │  └──────────────────────────────────────────────────────┘   │    │
│  │                         │                                    │    │
│  │                    detections[]                               │    │
│  │                         │                                    │    │
│  │  ┌──────────────────────▼───────────────────────────────┐   │    │
│  │  │  Reconciler (per module)                             │   │    │
│  │  │  expected vs detected → missing / extra / mismatch   │   │    │
│  │  └──────────────────────┬───────────────────────────────┘   │    │
│  └─────────────────────────┼───────────────────────────────────┘    │
│                             │                                        │
│         ┌──────────────────┼──────────────────────┐                 │
│         ▼                  ▼                      ▼                 │
│  ┌────────────┐  ┌─────────────────┐  ┌──────────────────┐         │
│  │  Supabase  │  │  hospital_events│  │  NATS broadcast  │         │
│  │  (persist) │  │  (orchestrator) │  │  (realtime)      │         │
│  └────────────┘  └─────────────────┘  └──────────────────┘         │
└─────────────────────────────────────────────────────────────────────┘
        │                    │                      │
        ▼                    ▼                      ▼
┌──────────────┐  ┌───────────────────┐  ┌─────────────────────┐
│  Frontend    │  │  Encounter        │  │  Department         │
│  (realtime)  │  │  Orchestrator     │  │  Dashboards         │
│  Scan Result │  │  (side-effects)   │  │  (live counters)    │
└──────────────┘  └───────────────────┘  └─────────────────────┘

Inference Backend Selection

Configured via services/vision/src/ever.config/:

Backend	config.inferenceBackend	When to use
stub	'stub'	Demo, development, integration testing. No model required.
onnx	'onnx'	On-premise / edge deployment. Requires onnxruntime-node + model weights at config.modelPath.
remote	'remote'	Cloud / GPU cluster. POSTs frame to config.remoteInferenceUrl (e.g., NVIDIA Triton, custom YOLO microservice).

All modules share the same backend config — a single env var switch deploys all 7 modules against a real model server.

Frontend Components

Each module ships a standalone React component:

Component	Package	Import
CountAiAssist	@periops-kit/operating-room	Surgical count
OpdPharmacyAiAssist	@medical-kit/pharmacy-verify	OPD single-Rx
IpdPharmacyAiAssist	@medical-kit/pharmacy-verify	IPD cart/cycle
SpecimenQaAiAssist	@medical-kit/lab-specimen-qa	Lab specimen tubes
BloodBankAiAssist	@medical-kit/blood-bank-verify	Blood product bags
WristbandIdAiAssist	@medical-kit/wristband-id	Patient wristband
WoundAssessAiAssist	@medical-kit/wound-assess	Wound photos
SterilizationQaAiAssist	@medical-kit/sterilization-qa	CSSD packs

All components share:

• Camera access with fallback to simulation
• YOLO bounding box overlay on live video
• Results panel (missing/extra/mismatch chips)
• Pharmacist/clinician sign-off
• Policy gate integration
• Live continuous-scan mode

Shared internals (usePharmacyScannerCore, PharmacyScannerViewport) are exported for custom compositions.

Data Flow

1. Capture — device camera or uploaded frame → base64 JPEG
2. Scan — POST /api/v2/vision/{module}/scan with frame + expected items
3. Infer — selected backend produces Detection[] with bboxes + confidence
4. Reconcile — module-specific logic (missing, quantity mismatch, type mismatch, label error, etc.)
5. Persist — upsert to module’s Supabase table (fail-soft, never blocks clinical workflow)
6. Broadcast — NATS event for in-cluster subscribers + hospital_events for orchestrator
7. Verify — clinician sign-off via POST /api/v2/vision/{module}/:scanUid/verify
8. Gate — policy engine evaluates scan results before allowing the next clinical action

Policy Gate Integration

Each module connects to the configurable policy_gates engine:

Module	Gate Trigger	Blocks
Surgical Count	close_surgical_case	Case closure without verified count
Pharmacy Verify	dispense_medication	Dispensing with unverified pick
Specimen QA	transport_specimens	Specimen transport with label errors
Blood Bank	transfuse_blood_product	Transfusion with type mismatch
Wristband ID	administer_medication	Med admin without patient ID confirm
Sterilization	open_sterile_pack	Using pack with failed indicator

File Map

services/vision/
├── src/api/vision/
│   ├── visionService.ts                     ← entry point, mounts all mixins
│   └── modules/
│       ├── _shared/
│       │   ├── inferenceAdapter.ts          ← surgical (original)
│       │   └── supabaseClient.ts
│       ├── surgicalCount/
│       ├── pharmacyVerify/
│       ├── specimenQa/
│       ├── bloodBankVerify/
│       ├── wristbandId/
│       ├── woundAssess/
│       └── sterilizationQa/
├── src/ever.config/
│   ├── IConfig.ts                           ← shared config interface
│   ├── ever-dev.config.ts
│   └── ever-prod.config.ts
└── moleculer.config.ts

infrastructure/medbase/migrations/
├── 20260523a_surgical_count_scans.sql
├── 20260524a_pharmacy_verify_scans.sql
├── 20260524b_specimen_qa_scans.sql
├── 20260524c_blood_bank_verify_scans.sql
├── 20260524d_wristband_id_scans.sql
├── 20260524e_wound_assess_scans.sql
└── 20260524f_sterilization_qa_scans.sql

web/src/services/
├── vision-ai.service.ts                     ← surgical
├── pharmacy-vision-ai.service.ts
├── specimen-vision-ai.service.ts
├── blood-bank-vision-ai.service.ts
├── wristband-vision-ai.service.ts
├── wound-vision-ai.service.ts
└── sterilization-vision-ai.service.ts

web/packages/medical-kit/src/
├── pharmacy-verify/                         ← OPD + IPD + shared hook/viewport
├── lab-specimen-qa/
├── blood-bank-verify/
├── wristband-id/
├── wound-assess/
└── sterilization-qa/

Adding a New Vision Module

1. Create services/vision/src/api/vision/modules/{name}/ with inference adapter + controller mixin
2. Add mixin to visionService.ts mixins array
3. Create migration in infrastructure/medbase/migrations/
4. Create frontend service in web/src/services/{name}-vision-ai.service.ts
5. Create React component in web/packages/medical-kit/src/{name}/
6. Register sandbox target
7. (Optional) Add policy gate seed in migrations

Deployment Notes

• Demo/dev: All modules run with inferenceBackend: 'stub' — no GPU, no model weights, realistic simulation
• Production: Set inferenceBackend: 'remote' + remoteInferenceUrl pointing to your YOLO inference server (per-module routing handled by the domain field in the POST body)
• On-premise: Set inferenceBackend: 'onnx' + ship model weights to config.modelPath
• Robo-dog patrol: See docs/architecture/robo-dog-patrol-integration.md