Status (2026-05-29): Tier 1 LIVE — registry-based restart cron (every 5 min, backend-auto-repair.sh) + Supabase observability (migration 20260529b) + super-admin dashboard at /super-admin/auto-repair are shipped and verified. Tier 2 (autonomous AI repair) is DESIGNED below — not yet built. Original 2026-05-15 concept notes retained for context.

Problem Statement

The PH demo backend runs 13 services natively on a single EC2 instance. Services crash for various reasons:

Crash type	Example	Frequency
Code bug in DTO/schema	@Optional() doesn’t exist in fastest-validator-decorators	On bad deploy
NestJS DI wiring error	WebNotificationGateway not exported from its module	On bad deploy
Stale dist files	moleculer-runner picks up NestJS class files as Moleculer schemas	On rebuild
Env not loaded	NestJS services need env sourced before node main.js	On manual restart
Mongoose-5 patch lost	yarn install wipes node_modules, removing the asPromise() patch	On dependency change
Memory/OOM	Gateway log grows to 1G+; long-running services leak	Periodic

When any service crashes, the entire system degrades — other services can’t call it via NATS, API calls 404/500, and the frontend breaks for the affected domain.

Architecture

┌─────────────────────────────────────────────────────┐
│  EC2 (3.0.73.73)                                    │
│                                                     │
│  ┌──────────────┐  ┌──────────────┐                 │
│  │ health-check │  │ auto-repair  │                 │
│  │   (cron)     │──│   (script)   │                 │
│  └──────────────┘  └──────┬───────┘                 │
│         │                 │                          │
│         ▼                 ▼                          │
│  ┌──────────────────────────────────────────┐       │
│  │  13 Services                              │       │
│  │  11 × moleculer-runner (clinical, etc.)   │       │
│  │   2 × node main.js (filestore, messaging) │       │
│  └──────────────────────────────────────────┘       │
│         │                                            │
│         ▼                                            │
│  /tmp/medos-*.log                                    │
│  /tmp/medos-health.json  (health snapshot)           │
│  /tmp/medos-repair.log   (repair audit trail)        │
└─────────────────────────────────────────────────────┘

Service Registry

Two different startup patterns exist on the EC2:

Service	Start method	Command
gateway, auth, clinical, diagnostic, financial, medication, foundation, administration, global-sequence, eform, interoperability	moleculer-runner	moleculer-runner --envfile ... --config ... './dist/src/**/*Service.js'
filestore	node main.js (NestJS)	source envfile && node dist/src/main.js (port 8083)
messaging	node main.js (NestJS)	source envfile && node dist/src/main.js (port 8084)

Critical: filestore and messaging MUST have the envfile sourced into the shell environment before running node main.js — moleculer-runner --envfile handles this automatically, but NestJS bootstrap does not.

Layer 1: Health Check Script

infrastructure/scripts/backend-health-check.sh — fast, non-destructive probe.

What it checks

1. Process existence — pgrep -af moleculer-runner + ss -tlnp for ports 8083/8084
2. Log freshness — last-modified time of /tmp/medos-{service}.log (stale = crashed or hung)
3. Startup success — grep 'started successfully' in each log
4. Error rate — count of ERROR lines in last N minutes
5. Log size — flag gateway log > 500MB for truncation

Output

{
  "timestamp": "2026-05-15T06:52:00Z",
  "services": {
    "gateway":       { "pid": 3226930, "status": "ok", "upSince": "2026-05-14T10:00:00Z", "errors1h": 2 },
    "clinical":      { "pid": 3623261, "status": "ok", "upSince": "2026-05-15T06:49:22Z", "errors1h": 0 },
    "filestore":     { "pid": 3623324, "status": "ok", "port": 8083, "upSince": "2026-05-15T06:49:27Z" },
    "messaging":     { "pid": 3625822, "status": "ok", "port": 8084, "upSince": "2026-05-15T06:52:33Z" },
    "financial":     { "pid": null, "status": "DEAD", "lastError": "Optional is not a function" }
  },
  "recommendations": [
    { "service": "financial", "action": "restart", "reason": "process not found" },
    { "service": "gateway", "action": "truncate_log", "reason": "log 1.2G" }
  ]
}

Usage

# From Claude Code
ssh ph-demo "bash /opt/medos/medOS-ultra/infrastructure/scripts/backend-health-check.sh"

# Quick one-liner (always works)
ssh ph-demo "pgrep -af moleculer-runner | wc -l"  # expect 11
ssh ph-demo "ss -tlnp | grep -E '8083|8084'"      # expect 2 lines

Layer 2: Auto-Repair Script

infrastructure/scripts/backend-auto-repair.sh — restarts dead services automatically.

Repair actions

Condition	Action
moleculer-runner service missing	rsync src from ~/medOS-ultra → /opt, tsc -p, relaunch with moleculer-runner
filestore (port 8083) not listening	source envfile && node dist/src/main.js
messaging (port 8084) not listening	source envfile && node dist/src/main.js
mongoose-5 patch missing	Re-apply sed patch to moleculer-db-adapter-mongoose/src/index.js
gateway log > 500MB	truncate -s 0 /tmp/medos-gateway.log

Safety rules

1. Never rebuild if source hasn’t changed — compare git log -1 --format=%H between ~/medOS-ultra and /opt/medos/medOS-ultra
2. Never restart a healthy service — only act on services confirmed dead
3. Max 2 repair attempts per service per hour — prevent restart loops
4. Log everything to /tmp/medos-repair.log with timestamps
5. Exit code 0 = all healthy, 1 = repairs attempted, 2 = repairs failed

Cron schedule (recommended)

# Check every 5 minutes, repair dead services
*/5 * * * * /opt/medos/medOS-ultra/infrastructure/scripts/backend-auto-repair.sh >> /tmp/medos-repair.log 2>&1

Layer 3: AI Auto-Repair (Claude Code Agent)

For crashes that can’t be fixed by simple restarts (code bugs, schema errors, DI wiring issues), a Claude Code session can diagnose and fix.

When to escalate from Layer 2 to Layer 3

• Service restarts but crashes again within 60 seconds (restart loop)
• Error log contains TypeError, Cannot find module, SyntaxError, or is not a function
• Build (tsc -p) fails with type errors

How it works

1. Claude Code runs ssh ph-demo "grep -i error /tmp/medos-{service}.log | tail -30" to capture the error
2. Reads the source file indicated in the stack trace
3. Applies a targeted fix (like the @Optional() → @String({ optional: true }) fix)
4. Commits, pushes, and triggers rebuild via GH Actions or manual SSH deploy

Prompt for Claude Code auto-repair session

Check backend service health on the PH demo EC2:
1. ssh ph-demo "pgrep -af moleculer-runner | wc -l" — expect 11
2. ssh ph-demo "ss -tlnp | grep -E '8083|8084'" — expect filestore + messaging
3. For any missing service, check /tmp/medos-{service}.log for errors
4. If the error is a code bug, fix it in the source, commit, push, and redeploy
5. If the error is a restart issue, follow the manual deploy runbook in CLAUDE.md

Lessons Learned (2026-05-15)

The @Optional() incident

Root cause: fastest-validator-decorators doesn’t export Optional, IsString, IsOptional, IsArray, or IsEnum. These names were likely hallucinated by an AI agent that generated the DTO files, assuming class-validator-style decorators.

Fix: Replace @Optional() with @String({ optional: true }) or @Any({ optional: true }). Replace @IsString() with @String(), @IsArray() with @Array().

Available decorators (from fastest-validator-decorators):

Schema, String, Number, Boolean, Date, Email, UUID, ObjectId,
Enum, Array, Any, Field, Custom, Nested, Url, Currency, Luhn, Mac,
Func, Equal, Instance, validate, validateOrReject, getSchema

The filestore/messaging startup pattern

Root cause: filestore and messaging use NestJS NestFactory.create() bootstrap via main.ts, not Moleculer runner. The auto-deploy workflow only knows about moleculer-runner, so it was starting them wrong.

Fix: Start with source envfile && node dist/src/main.js instead of moleculer-runner ... *Service.js.

The WebNotificationGateway DI error

Root cause: NotificationModule declared WebNotificationGateway as a provider but didn’t export it. AcknowledgementDispatcherModule imported NotificationModule and tried to inject WebNotificationGateway — NestJS requires explicit exports even for @Global() modules.

Fix: Added exports: [NotificationService, WebNotificationGateway] to NotificationModule.

Deploy Workflow Enhancement (TODO)

The GH Actions deploy-backend.yml needs to be updated to handle NestJS services differently:

# Proposed: detect service type and use correct start command
NESTJS_SERVICES="filestore messaging"
if echo "$NESTJS_SERVICES" | grep -qw "$svc"; then
  # NestJS bootstrap
  nohup bash -c "set -a && source $ENVFILE && set +a && cd $SVC_DIR && exec node dist/src/main.js" \
    > /tmp/medos-$svc.log 2>&1 &
else
  # Moleculer runner
  nohup moleculer-runner --envfile $ENVFILE --config ./dist/moleculer.config.js \
    './dist/src/**/*Service.js' > /tmp/medos-$svc.log 2>&1 &
fi

---

Tiered Autonomous Repair (Layer 3 — designed 2026-05-29)

Supersedes the manual “Layer 3: AI Auto-Repair (Claude Code Agent)” runbook above as the target design (the runbook still describes the eventual Tier‑3 human step). Status: designed, not yet built. Tier 1 (reflex restart) + observability are live.

Locked decisions (2026-05-29)

Axis	Decision
Autonomy	AI auto-acts on safe ops only (restart / rebuild / re-apply known patch). Any code change is PR-first — the agent opens a PR, a human merges. Nothing AI‑authored reaches main (which auto-deploys to the shared backend) without human review.
Brain	On-box Ollama (private; default mistral:7b), model set by OLLAMA_MODEL, behind a Brain interface so it can be upgraded to a larger local model or a hosted API later without touching the ladder. PR-first contains the weak-model risk: a poor patch is caught at human review.
Escalation	Dashboard timeline + multi-channel page via the existing AcknowledgementRequest system (app/email/SMS/push) with an escalation chain. See docs/architecture/acknowledgement-system.md.

The ladder

T0 detect ─▶ T1 reflex restart ─▶ T2 AI triage + bounded repair ─▶ T3 human page
 registry      (LIVE, autonomous)    (NEW: autonomous ops / PR for code)   (ack chain)
 says down     restart from dist     classify → act by policy → verify     dashboard + page
               ≤2×/hr/service        ≤2 cycles/service/day, time-boxed
   ▲                │ recovers            │ recovers           │ can't fix / needs code
   └── resolved ◀───┴─────────────────────┴────────────────────┘ (PR opened → awaits merge)

A service only climbs a rung when the rung below is exhausted. Every transition is recorded on the escalation row → the dashboard renders it as a timeline (“restarted ×2 → AI rebuilt → still failing → paged on-call”).

Tier‑2 failure taxonomy → action policy

Class (model output)	Action	Autonomy
transient (Mongo/NATS/dependency down)	back off + retry; mark “waiting on dependency”	no change made
stale_dist / missing_dist	rebuild (tsc) + restart	autonomous (safe ops)
lost_patch (e.g. mongoose-5)	re-apply whitelisted patch + restart	autonomous (safe ops)
code_bug (TypeError, bad decorator, DI, syntax)	draft patch → branch → open PR (never merge)	PR-first (human merges)
unknown / low confidence	no action → Tier 3	escalate

Runner

• Runs on the EC2 box (it must read logs and perform ops). Small worker (Node or bash+curl), invoked by cron every ~1 min or triggered when backend-auto-repair.sh writes a Tier‑2-eligible escalation.
• Loop: pick backend_ai_escalations where status='open' + Tier‑2-eligible + kill switch on → build a structured prompt → call the Brain → execute by policy → verify via the broker registry → update the row → on exhaustion create the Tier‑3 AcknowledgementRequest.
• Prompt contract (Ollama JSON mode, like services/gateway/src/ai/voiceOrderProxy.ts): input { service, log_excerpt (last ~50 lines, scrubbed), recent_repair_events } → output { class, confidence, action, patch_plan? }.
• Bounded: ≤2 Tier‑2 cycles/service/day, time-boxed per cycle, global concurrency 1.

Data model — extend backend_ai_escalations

Add (additive migration): tier INT, attempts INT, actions_log JSONB (append per action), classification TEXT, confidence NUMERIC, pr_url TEXT, resolution TEXT, updated_at TIMESTAMPTZ.

Guardrails (hard rules)

1. Kill switch AI_AUTOREPAIR_ENABLED (default OFF) ⇒ today’s behavior (restart + surface), no autonomy.
2. AI may run only a whitelist of ops: restart, rebuild (tsc), re-apply a whitelisted patch. Any other shell command is blocked.
3. Code is PR-only. The runner has no push rights to main; it pushes a branch + opens a PR. Enforce with a scoped token and branch protection on main.
4. Scope: only the crash-looping service’s directory. Must not touch infrastructure/scripts/*, settings, the runner itself, other services, or its own guardrails.
5. Audit: every action → backend_repair_events + the escalation actions_log.
6. Freeze-aware: an AI_AUTOREPAIR_FREEZE flag (or demo-freeze) ⇒ escalate, don’t act.
7. No PHI: only service logs / stack traces go to the model; a scrubber strips obvious tokens/emails/keys first.
8. Circuit breaker: if N PRs sit unmerged or M cycles fail, stop opening new work and page.

Brain abstraction (upgrade path)

interface Brain {
  classify(ctx: RepairContext): Promise<{
    class: FailureClass; confidence: number; action: ActionPlan; patchPlan?: PatchPlan;
  }>;
}
// OllamaBrain (now, OLLAMA_MODEL) → swap to a larger local model or a HostedBrain later.

Phased rollout

Phase	Scope	Risk
P1 — substrate	backend_ai_escalations columns + kill-switch config + dashboard tier-timeline display	none (no autonomy)
P2 — recommend-only	runner classifies + recommends; writes classification, takes no actions — validates Ollama quality on real failures	low (read-only)
P3 — ops autonomy	enable safe-ops execution (restart/rebuild/re-apply) behind the kill switch	medium
P4 — code PR-first	generate patch → branch → open PR (never merge)	medium (human-gated)
P5 — ack escalation	AcknowledgementRequest multi-channel + escalation chain	low
P6 — hardening	budgets, circuit breaker, freeze windows, PHI scrubber, audit review	—

Recommendation: ship P1–P2 and run recommend-only for ~1 week to measure the model’s classification accuracy on real crashes before enabling any autonomy (P3+). The 7b model’s triage quality is the gating risk; PR-first contains the code-quality risk.