Robo-Dog Patrol Integration

Overview

A robotic quadruped (Boston Dynamics Spot, Unitree Go2, Xiaomi CyberDog, etc.) acts as a mobile autonomous scan station that patrols the hospital on a schedule, performing vision AI scans at each waypoint. The dog uses the same POST /api/v2/vision/{module}/scan endpoints as human-operated tablet cameras — the backend doesn’t distinguish the source.

Architecture

┌─────────────────────────────────────────────────────────────┐
│  ROBO-DOG (Edge Compute: Jetson Orin / Raspberry Pi 5)     │
│                                                             │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │  Camera     │  │  Nav Stack   │  │  Patrol           │  │
│  │  (RGB +     │  │  (ROS2 Nav2  │  │  Scheduler        │  │
│  │   depth)    │  │   + SLAM)    │  │  (cron waypoints) │  │
│  └──────┬──────┘  └──────┬───────┘  └────────┬─────────┘  │
│         │                 │                    │            │
│         ▼                 ▼                    ▼            │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  medOS Vision Client (Python / ROS2 Node)           │   │
│  │                                                     │   │
│  │  1. Navigate to waypoint                            │   │
│  │  2. Capture frame (+ optional local YOLO)           │   │
│  │  3. POST to /api/v2/vision/{module}/scan            │   │
│  │  4. Log result, move to next waypoint               │   │
│  │  5. If mismatch → alert + stay for re-scan          │   │
│  └─────────────────────────────────────────────────────┘   │
│                              │                              │
└──────────────────────────────┼──────────────────────────────┘
                               │  WiFi / 5G
                               ▼
┌──────────────────────────────────────────────────────────────┐
│  medOS Backend (services/vision/)                            │
│  POST /api/v2/vision/{module}/scan                           │
│  → inference → reconcile → persist → broadcast               │
└──────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────┐
│  Ward Dashboard / Pharmacy Dashboard / CSSD Dashboard        │
│  Realtime: "Robot patrol scan at Station B — 2 missing items"│
└──────────────────────────────────────────────────────────────┘

Integration Modes

Mode A: Dog as Dumb Camera (Cloud Inference)

Dog captures JPEG frames and sends them to the backend. The stub or remote backend runs inference server-side.

# dog_client.py (runs on Jetson)
import requests, base64, cv2

frame = cv2.imread(capture())  # or cv2.VideoCapture
_, buf = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 80])
b64 = base64.b64encode(buf).decode()

resp = requests.post(
    f"{MEDOS_API}/v2/vision/specimen-qa/scan",
    json={
        "phase": "pre_transport",
        "imageBase64": b64,
        "frameWidth": frame.shape[1],
        "frameHeight": frame.shape[0],
        "expectedItems": waypoint["expected_items"],
        "scanSource": "robot_patrol",
    },
    headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
)
result = resp.json()

Pros: Simple, no model on dog, always up-to-date model.
Cons: Needs reliable WiFi, 200-500ms latency per scan.

Mode B: Dog as Edge Inferencer (Local YOLO)

Dog runs YOLOv8/v11 locally on Jetson Orin, sends finished detections (not raw frames) to the backend for persistence + reconciliation.

from ultralytics import YOLO

model = YOLO("medos-pharmacy-v1.pt")  # shipped to dog
results = model(frame, conf=0.5)

detections = [
    {"id": map_class(r.cls), "name": r.name, "confidence": r.conf, "bbox": r.xyxy}
    for r in results[0].boxes
]

# POST pre-computed detections (backend skips inference, just reconciles)
resp = requests.post(
    f"{MEDOS_API}/v2/vision/pharmacy-verify/scan",
    json={
        "phase": "verification",
        "imageBase64": "",  # empty = edge-inferred
        "precomputedDetections": detections,
        "expectedItems": waypoint["expected_items"],
        "scanSource": "robot_edge",
    },
    headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
)

Pros: <50ms latency, works offline (queues results), no bandwidth for frames.
Cons: Need to ship model weights, version management, more compute on dog.

Mode C: Hybrid (Edge + Cloud Verify)

Dog runs fast local YOLO for immediate go/no-go. If mismatch detected, uploads the frame to cloud for high-accuracy re-inference with a larger model.

Patrol Schedule System

Database: robot_patrol_schedules

create table robot_patrol_schedules (
  id uuid primary key default gen_random_uuid(),
  robot_id text not null,
  name text not null,
  schedule_cron text not null,           -- e.g. '0 6,12,18,22 * * *'
  waypoints jsonb not null default '[]', -- ordered array of waypoint configs
  status text default 'active',
  organization_id uuid,
  created_at timestamptz default now()
);

Waypoint Config Shape

interface PatrolWaypoint {
  id: string;
  label: string;                    // "Pharmacy Shelf A", "Ward 3A Bed 12"
  nav_goal: { x: number; y: number; theta: number };  // ROS2 nav goal
  vision_module: 'pharmacy-verify' | 'specimen-qa' | 'blood-bank-verify'
    | 'wristband-id' | 'wound-assess' | 'sterilization-qa' | 'surgical-count';
  phase: string;                    // module-specific phase
  expected_items_source: 'static' | 'supabase_query' | 'api_call';
  expected_items_config: any;       // depends on source type
  dwell_time_sec: number;           // how long to observe
  retry_on_mismatch: boolean;       // re-scan if discrepancy
  max_retries: number;
}

Example Patrol: Night Pharmacy Shelf Audit

{
  "robot_id": "spot-001",
  "name": "Night Pharmacy Shelf Audit",
  "schedule_cron": "0 2 * * *",
  "waypoints": [
    {
      "id": "wp-1",
      "label": "Controlled Substance Cabinet",
      "nav_goal": { "x": 12.5, "y": 8.3, "theta": 1.57 },
      "vision_module": "pharmacy-verify",
      "phase": "verification",
      "expected_items_source": "supabase_query",
      "expected_items_config": {
        "table": "pharmacy_inventory",
        "filter": { "cabinet_id": "controlled-A", "expected_present": true }
      },
      "dwell_time_sec": 10,
      "retry_on_mismatch": true,
      "max_retries": 2
    },
    {
      "id": "wp-2",
      "label": "Fridge - Blood Bank",
      "nav_goal": { "x": 15.0, "y": 12.1, "theta": 0 },
      "vision_module": "blood-bank-verify",
      "phase": "crossmatch_check",
      "expected_items_source": "api_call",
      "expected_items_config": {
        "endpoint": "/api/v2/blood-bank/fridge/expected-units",
        "params": { "fridge_id": "bb-fridge-1" }
      },
      "dwell_time_sec": 15,
      "retry_on_mismatch": true,
      "max_retries": 1
    }
  ]
}

Backend Changes Required

1. Add scan_source field to all vision tables

-- Add to each *_scan_results table:
alter table pharmacy_verify_scan_results
  add column scan_source text default 'human_camera'
  check (scan_source in ('human_camera','fixed_camera','robot_patrol','robot_edge'));

2. Accept precomputedDetections in scan actions

When the dog sends pre-computed detections (Mode B), the backend skips inference and jumps straight to reconciliation:

// In each controller mixin:
if (p.precomputedDetections && p.precomputedDetections.length > 0) {
  detections = p.precomputedDetections;
  modelVersion = 'edge-' + (p.edgeModelVersion || 'unknown');
  inferenceMs = p.edgeInferenceMs || 0;
} else {
  ({ detections, modelVersion, inferenceMs } = await detectItems(raw, ...));
}

3. Robot patrol REST endpoints

POST /api/v2/vision/robot/patrol/start    — trigger a named patrol
POST /api/v2/vision/robot/patrol/stop     — abort current patrol
GET  /api/v2/vision/robot/patrol/status   — current waypoint + progress
GET  /api/v2/vision/robot/patrol/history  — past patrol runs with results

4. MQTT/ROS2 bridge

Add to the messaging service:

// On-demand scan command (ward nurse presses "Robot scan bed 12")
broker.emit('ROBOT_SCAN_REQUEST', {
  robotId: 'spot-001',
  waypoint: { ... },
  priority: 'urgent',
  requestedBy: userId,
});

The dog subscribes via MQTT and receives the nav command.

Hardware Requirements

Component	Minimum	Recommended
Robot platform	Unitree Go2 ($1,600)	Boston Dynamics Spot ($75K)
Compute	Raspberry Pi 5 (Mode A only)	Jetson Orin Nano (Mode B/C)
Camera	1080p RGB	4K RGB + depth (Intel RealSense D455)
Network	WiFi 5 (802.11ac)	WiFi 6E or 5G module
Battery	2hr patrol autonomy	4hr with auto-dock charging
Storage	32GB (frame cache)	256GB NVMe (local model + replay buffer)

Safety & Compliance

1. Speed limit — max 1.0 m/s in patient areas, 0.5 m/s near beds
2. Obstacle avoidance — LiDAR + depth camera, emergency stop on contact
3. Operating hours — night patrols (22:00–06:00) avoid patient/staff congestion
4. Infection control — UV-C sterilizable shell, no fabric surfaces, wipe-down protocol
5. Patient consent — no facial recognition, wristband scan only reads text/QR
6. Data handling — frames deleted after inference (unless flagged for audit), never stored on dog long-term
7. Failsafe — if WiFi drops, dog returns to charging dock, queued results sync on reconnect

Phased Rollout

Phase	Scope	Duration
1 — Proof of Concept	1 dog, 3 waypoints (pharmacy shelf, CSSD room, blood bank fridge), Mode A, night only	4 weeks
2 — Edge Inference	Add Jetson, run local YOLO, Mode B, validate accuracy vs cloud	4 weeks
3 — Ward Expansion	Add wristband + specimen scanning waypoints in 1 ward, on-demand scan button in dashboard	6 weeks
4 — Multi-Robot Fleet	2-3 dogs covering different floors, patrol scheduler, fleet management dashboard	8 weeks
5 — Full Autonomy	Auto-dock charging, self-scheduling based on department queue load, wound assessment rounds	Ongoing

Cost Estimate (Phase 1)

Item	Cost
Unitree Go2 Pro	$2,800
Jetson Orin Nano 8GB	$500
Intel RealSense D455	$350
Charging dock (custom)	$200
5G module	$150
Integration development	80 hrs
Total (hardware)	~$4,000