Vision AI Robotics & IoT

Robotics & IoT device integration design concept for the vision-AI service.

11 min read diagramsUpdated 2026-05-25docs/architecture/vision-ai-robotics-integration.md

Status: Design concept — not yet implemented Depends on: docs/architecture/vision-ai-module-catalog.md (9 vision modules) Parent service: services/vision/ + services/public-api/

The Idea

A robot dog (Boston Dynamics Spot, Unitree Go2, Xiaomi CyberDog, etc.) or any camera-equipped IoT device accompanies nurses on ward rounds. It carries a camera and automatically:

Scans patient wristbands for identity verification
Photographs meal trays for nutrition intake analysis
Reads bedside monitor displays for vital signs capture
Verifies medication trays before administration
Photographs wounds for progression tracking
Counts supplies on shelves during rounds

The robot is a mobile camera + compute platform — the intelligence lives in the medOS vision service. The robot just captures and transmits frames.

Architecture

                                    ┌─────────────────────────────────┐
                                    │        medOS Backend            │
                                    │                                 │
┌──────────────┐    HTTPS/WSS       │  ┌───────────────────────────┐  │
│  Robot Dog   │───────────────────▶│  │  public-api (port 8082)   │  │
│  (Spot/Go2)  │  Device API Token  │  │  /device/v1/vision/*      │  │
│              │◀───────────────────│  │  WebSocket /device/stream  │  │
│  • Camera    │    JSON results    │  └───────────┬───────────────┘  │
│  • LiDAR     │                    │              │ Moleculer call   │
│  • Speaker   │                    │  ┌───────────▼───────────────┐  │
│  • Display   │                    │  │  vision service           │  │
│  • GPS/UWB   │                    │  │  (existing modules)       │  │
└──────┬───────┘                    │  │  • surgicalCount          │  │
       │                            │  │  • nutritionAnalysis      │  │
       │  WiFi / 5G                 │  │  • medicationVerify       │  │
       │                            │  │  • vitalSignsOcr          │  │
       │                            │  │  • wristbandVerify        │  │
       │                            │  │  • woundAssessment        │  │
┌──────▼───────┐                    │  │  • specimenLabelQa        │  │
│  Nurse App   │                    │  │  • inventoryScan          │  │
│  (tablet/    │◀──Supabase RT──────│  └───────────┬───────────────┘  │
│   phone)     │                    │              │                  │
│              │                    │  ┌───────────▼───────────────┐  │
│  • Live feed │                    │  │  Supabase + NATS          │  │
│  • Results   │                    │  │  (persistence + events)   │  │
│  • Override  │                    │  └───────────────────────────┘  │
└──────────────┘                    └─────────────────────────────────┘

What the Robot Does vs. What medOS Does

Responsibility	Robot	medOS
Camera capture	Frames at 1-5 FPS	—
Location awareness	UWB/BLE beacon → knows which bed	Receives `bedId` / `wardId` in API call
Object detection (fast)	On-device YOLO (Jetson/NPU) for real-time framing	—
Clinical analysis	—	LLM vision via `llmAdapter` / `inferenceAdapter`
Results display	Show on robot screen + speak alerts	Push via WebSocket + Supabase realtime
Clinical decision	—	CDS rules, policy gates, acknowledgements
Persistence	—	Supabase tables, IPFS photo storage
Manual override	—	Nurse confirms/corrects via tablet app

Key principle: The robot is a dumb camera with legs. All clinical intelligence stays in medOS. The robot never makes clinical decisions.

Device API Design

Authentication

Devices authenticate via API tokens (not JWT user sessions). Each registered device gets a long-lived token stored in a devices table.

CREATE TABLE devices (
  device_id       TEXT PRIMARY KEY,
  device_name     TEXT NOT NULL,           -- 'Spot-Ward3A'
  device_type     TEXT NOT NULL,           -- 'robot_dog' | 'wall_camera' | 'cart_camera'
  api_token_hash  TEXT NOT NULL,           -- bcrypt hash of the token
  facility_id     TEXT,
  default_ward_id TEXT,
  capabilities    JSONB DEFAULT '[]',      -- ['camera','lidar','speaker','display']
  status          TEXT DEFAULT 'active',   -- 'active' | 'inactive' | 'maintenance'
  last_seen_at    TIMESTAMPTZ,
  registered_at   TIMESTAMPTZ DEFAULT now(),
  registered_by   TEXT
);

Token is sent as Authorization: Bearer <device-api-token> — the public-api service validates against the devices table (not the user auth system).

REST Endpoints (single-frame analysis)

All endpoints live under public-api since they’re device-facing (no user session required).

POST /device/v1/vision/analyze
  Body: {
    "moduleType": "nutrition" | "medication" | "vitals" | "wristband" | "wound" | "specimen" | "inventory" | "surgical",
    "imageBase64": "...",
    "context": {
      "wardId": "ward-3a",
      "bedId": "bed-301",
      "patientId": "...",           // optional — robot may not know
      "encounterId": "...",         // optional
      "locationBeaconId": "ble-301" // robot's UWB/BLE position
    },
    "deviceMetadata": {
      "deviceId": "spot-ward3a",
      "captureTimestamp": "ISO",
      "cameraIndex": 0,             // robot may have multiple cameras
      "frameWidth": 1920,
      "frameHeight": 1080
    }
  }
  Response: {
    "analysisUid": "...",
    "moduleType": "nutrition",
    "result": { ... },              // module-specific result shape
    "modelVersion": "...",
    "inferenceMs": 1200,
    "persisted": true
  }

GET /device/v1/vision/modules
  → List available vision modules + their capabilities
  → Robot uses this to decide which module to invoke based on context

POST /device/v1/vision/stream/start
  → Initiate a WebSocket session for continuous streaming
  Body: { "sessionId": "...", "moduleType": "vitals", "fps": 2, "context": {...} }

POST /device/v1/heartbeat
  → Device health check + location update
  Body: { "deviceId": "...", "batteryPct": 85, "locationBeaconId": "ble-301", "status": "patrolling" }

WebSocket Streaming (continuous analysis)

For real-time scenarios (vital signs monitor reading, surgical instrument counting during a procedure), the robot opens a WebSocket connection and streams frames continuously.

WSS /device/v1/vision/stream

→ Client sends:
  { "type": "frame", "imageBase64": "...", "seq": 42, "timestamp": "ISO" }

← Server sends:
  { "type": "result", "seq": 42, "moduleType": "vitals", "result": {...}, "inferenceMs": 800 }
  { "type": "alert", "severity": "warning", "message": "SpO2 dropping: 91%", "cdsRuleId": "..." }
  { "type": "ack_required", "message": "Medication mismatch detected", "ackId": "..." }

→ Client can also send:
  { "type": "context_update", "bedId": "bed-302" }   // robot moved to next bed
  { "type": "module_switch", "moduleType": "wound" }  // nurse says "scan wound now"

Frame throttling: The server processes at most N frames/sec per device (configurable). Excess frames are dropped with a { "type": "throttled" } response. This prevents a malfunctioning robot from overwhelming the LLM API.

Robot Round Workflow

Typical Nurse + Robot Round

1. Nurse starts round on tablet → POST /device/v1/round/start
   Robot follows nurse (autonomous navigation or remote control)

2. Arrive at Bed 301
   Robot detects BLE beacon → context auto-switches to bed-301
   ┌─────────────────────────────────────────────────────┐
   │  Auto-sequence (configurable per ward):             │
   │  a. Scan wristband → verify patient identity        │
   │  b. Photograph monitor → capture vital signs         │
   │  c. Photograph meal tray → nutrition intake analysis │
   │  d. Photograph wound → wound assessment (if flagged) │
   │  e. Verify medication tray → if med pass scheduled   │
   └─────────────────────────────────────────────────────┘
   Results stream to nurse's tablet in real-time

3. Nurse reviews results on tablet
   - Confirms or corrects AI readings
   - Signs off on medication verification
   - Adds notes to wound assessment

4. Move to Bed 302 → context auto-switches
   Robot repeats sequence

5. End of round → POST /device/v1/round/end
   Summary report generated: all patients scanned, flags raised

Round Configuration

Each ward can configure its default scan sequence:

{
  "wardId": "ward-3a",
  "roundProfile": "morning-med-pass",
  "sequence": [
    { "module": "wristband", "required": true },
    { "module": "vitals", "required": true },
    { "module": "medication", "required": true, "condition": "med_pass_scheduled" },
    { "module": "nutrition", "required": false, "condition": "meal_delivered" },
    { "module": "wound", "required": false, "condition": "wound_care_flagged" }
  ],
  "autoAdvance": true,
  "speakResults": true,
  "alertThreshold": "warning"
}

Location Awareness

The robot needs to know which patient/bed it’s near. Options:

Method	Hardware	Accuracy	Cost
BLE beacons per bed	BLE tag on each bed frame	~1-2m	$5-10/beacon
UWB anchors	4+ UWB anchors per ward	~10-30cm	$50-100/anchor
QR codes on bed rails	Printed QR with bed ID	Exact (requires scan)	~$0
Visual room number OCR	Robot camera reads room/bed signs	Variable	$0
WiFi fingerprinting	Existing WiFi APs	~3-5m	$0

Recommended: BLE beacons per bed (cheap, passive, reliable) + QR fallback on bed rail for explicit confirmation. The robot’s onboard BLE scanner auto-detects proximity; the QR is a manual override when BLE is ambiguous (adjacent beds).

On-Device vs. Cloud Processing

Task	Where	Why
Frame capture + encoding	On-device	Must be real-time
Object detection (YOLO)	On-device (Jetson/NPU)	Low latency for framing, navigation
Clinical analysis (LLM)	Cloud (vision service)	Requires medical knowledge, too heavy for edge
Text-to-speech alerts	On-device	Low latency for spoken alerts
Display rendering	On-device	Show results on robot screen
Location sensing	On-device	BLE/UWB scanning
Persistence + events	Cloud (Supabase/NATS)	Clinical record, audit trail

Edge Inference Option

For hospitals with strict data-locality requirements or poor connectivity, the vision service’s remote inference backend already supports pointing to a local YOLO server. A GPU box in the server room can run:

Robot (WiFi) → Local YOLO server (RTX 4090) → Vision service (results only)

The VISION_REMOTE_URL config already exists for this — no code changes needed for the YOLO path. For LLM vision (nutrition, wound, vitals OCR), a local Ollama/vLLM instance running a vision model (LLaVA, Qwen-VL) can be configured via VISION_LLM_BACKEND=openai + VISION_LLM_API_KEY pointing to the local server’s OpenAI-compatible endpoint.

Supported Robot Platforms

Platform	Camera	Compute	Navigation	Est. Cost
Boston Dynamics Spot	5 cameras + optional payload cam	Jetson AGX (payload)	Autonomous + map	$75K+
Unitree Go2 Pro	Front stereo + payload cam	Jetson Orin Nano	Autonomous	$3-8K
Unitree B2	Multi-camera array	Jetson AGX	Autonomous (industrial)	$15-30K
Xiaomi CyberDog 2	Depth camera + RGB	Qualcomm 8-series	Semi-autonomous	$3K
AgileX Limo	Configurable payload	Jetson Nano/Orin	ROS2 autonomous	$2-5K
Custom cart	USB webcam + tablet	Tablet/RPi	Manual (nurse pushes)	$500-1K

Cheapest MVP: A tablet mounted on a medication cart with a phone camera — literally a phone on a stick. It hits the same device API endpoints. Graduate to a robot dog when the workflow is proven.

SDK / Client Library

Provide a lightweight Python SDK for robot developers:

from medos_vision import MedOSVisionClient

client = MedOSVisionClient(
    base_url="https://medos.hospital.local/device/v1",
    api_token="dev-token-spot-3a",
    device_id="spot-ward3a"
)

# Single frame analysis
result = client.analyze(
    module="nutrition",
    image=frame_bytes,
    context={"ward_id": "ward-3a", "bed_id": "bed-301"}
)
print(result.analysis.percentage_eaten)  # 72

# Streaming session
async with client.stream(module="vitals", fps=2) as session:
    async for frame in camera.frames():
        result = await session.send_frame(frame)
        if result.alerts:
            robot.speak(result.alerts[0].message)

# Round management
round_id = client.start_round(ward_id="ward-3a", profile="morning-med-pass")
# ... robot does its thing ...
summary = client.end_round(round_id)

Also provide a ROS2 node (medos_vision_ros2) for direct integration with robot navigation stacks — publishes results on /medos/vision/result topic and subscribes to /camera/image_raw for frame capture.

Files to Create

Backend (device API gateway)

services/public-api/src/api/publicapi/modules/device-vision/
  device-vision.module.ts          -- NestJS module registration
  device-vision.controller.ts      -- REST endpoints (/device/v1/vision/*)
  device-vision.gateway.ts         -- WebSocket gateway for streaming
  device-auth.guard.ts             -- API token validation against devices table
  round-manager.service.ts         -- Round start/end, sequence orchestration
  frame-throttle.interceptor.ts    -- Rate limiting per device
  dto/
    analyze-frame.dto.ts
    start-stream.dto.ts
    device-heartbeat.dto.ts
    start-round.dto.ts

Infrastructure

infrastructure/medbase/migrations/
  YYYYMMDD_devices_table.sql           -- devices + device_api_tokens
  YYYYMMDD_device_rounds.sql           -- round sessions + round_scans join
  YYYYMMDD_ward_round_profiles.sql     -- configurable scan sequences per ward

infrastructure/market-packs/*/
  seed-ward-round-profiles.sql         -- default round configs per region

SDK (future, separate repo or subdir)

sdk/
  python/
    medos_vision/
      client.py
      streaming.py
      types.py
    setup.py
  ros2/
    medos_vision_ros2/
      medos_vision_node.py
      package.xml
      setup.py

Frontend (device management admin)

web/packages/miniapps/device-management/
  DeviceManagement.tsx          -- CRUD for registered devices
  DeviceStatusDashboard.tsx     -- Live status of all devices
  RoundProfileEditor.tsx        -- Configure ward scan sequences
  DeviceRoundHistory.tsx        -- View past round summaries
  index.ts

Implementation Phases

Phase	Scope	Effort
Phase 1: Device API	`devices` table + token auth + single-frame `/analyze` endpoint that proxies to existing vision modules	2-3 days
Phase 2: Cart MVP	Tablet-on-cart app that calls device API — proves the workflow without a robot	1-2 days
Phase 3: WebSocket streaming	Continuous frame analysis for vital signs + surgical count	3-4 days
Phase 4: Round management	Start/end round, auto-sequence, summary report	2-3 days
Phase 5: Location awareness	BLE beacon integration, auto-context switching	3-4 days
Phase 6: Robot integration	Unitree Go2 or Spot SDK integration, ROS2 node	1-2 weeks
Phase 7: Python SDK	Packaged client library for third-party robot developers	3-4 days

Phase 1+2 is the critical path — once the device API exists and a tablet-on-cart proves the workflow, robot integration is just a different client calling the same endpoints.

Safety Rules

Robot never makes clinical decisions — it captures and transmits; medOS decides
Robot never blocks a nurse — if API is unreachable, nurse continues manually
Robot never touches the patient — camera-only interaction
All results require nurse confirmation — robot findings are suggestions, not orders
Robot has a physical kill switch — hardware e-stop accessible to any staff
Robot operates in supervised mode only — a nurse is always present during rounds
PHI stays in-hospital — no patient data sent to robot vendor cloud
Device tokens are ward-scoped — a robot registered for Ward 3A cannot access Ward 4B data