medOS ultra

HIE Migration + Clinical-AI (Master)

Strategic envelope: HIE to AWS Thailand migration plus clinical-AI landing zone.

25 min read diagramsUpdated 2026-06-11docs/architecture/hie-migration-clinical-ai/00-MASTER.md

Status: Planning (decisions locked 2026-05-31). No code/infra changed yet — this is the strategic envelope. Scope: Migrate the hie-prod GCP project (EverMed multi-hospital HIE / patient-referral platform) off GCP, and stand up the hospital-mandated clinical-AI layer (LLM + ML + symptom-based pre-diagnosis) on the migrated data. Companion docs: ai-training-corpus.md (de-id medallion), unified-clinical-assistant.md + smart-diagnosis-unified-pipeline.md (AI surfaces), project_llm_platform memory (Ollama+RAG), @medical-kit/code-systems (terminology homogenizer).

1. Decisions locked (2026-05-31)

Decision Choice Rationale / caveat
Driver Leave GCP to save money + Thailand data residency Today’s data sits in GCP asia-southeast1 = Singapore, not Thailand.
Destination AWS Asia Pacific (Thailand), ap-southeast-7 (Bangkok, 3 AZs, GA) Reportedly prices below Singapore; puts data in-country.
App platform Keep k8s → EKS App tier is stateless (only monitoring PVCs), so this is mostly mechanical. ⚠️ Verify EKS is GA in ap-southeast-7 on the AWS Regional Services List before committing; fallback = self-managed k8s on EC2 same region.
MongoDB target DocumentDB (confirmed available in ap-southeast-7) if compat-spike passes; else lift-shift Mongo VMs → EC2 DocumentDB ≠ 100% Mongo. Spike must check: change streams, transactions, aggregation operators, mongoose-5 / moleculer-db adapter behavior.
AI consumers BOTH clinician-facing AND patient-facing ⇒ one engine, two surfaces, different output contracts (§6).
AI approach LLM-RAG grounding first; ML predictive models = phase 2 RAG on the referral corpus gets a “tuned on our data” pre-diagnosis fast, no model training.
Cutover Per-site, one hospital at a time; pilot prod-hie-center first Each site is independently isolated (own namespace + Mongo VM + buckets) ⇒ no big-bang.

2. Current state — hie-prod (GCP, verified live 2026-05-31)

App tier — GKE prod-hie-apps (regional asia-southeast1, 12× e2-medium, k8s 1.33). Single public entry: ingress-nginx LB 34.87.180.155 (host-routed, cert-manager TLS). Bastion jump-host ext 34.124.150.18.

One namespace per hospital site, three app archetypes:

  • Gen-1 “refer” (cbn, ccs, cti, ksn, pri, rayong, vajira): prod-refer-backend (3000/3030/3210) + -frontend + -sync-api (+ report-service, + sync-queue-worker on ccs).
  • Gen-2 “ever” (smpk only — fuller): prod-ever-his + prod-ever-refer-backend (3005) + -external (3004) + -sync-api + -sync-queue-worker.
  • Central: prod-hie-core (hie-sync-service + sync-adapter-file — the cross-site brain); prod-hie-center (hie-center, new 73d).
  • Teleconsult (prod-teleconsultation-ksn, Kalasin): kalasin-backend/frontend/sync-api.

Data tier — ALL external to the cluster (key migration win — only monitoring PVCs in-cluster: grafana 20Gi, loki 500Gi):

  • 9 MongoDB VMs, one per site (10.20.1.x, mostly e2-standard-2, look standalone — -main with no replica siblings). (3× restore-test-* Mongo VMs are TERMINATED.)
  • Brokers: NATS (vajira, ext 34.124.209.75); RabbitMQ (ccs, smpk).
  • GCS: per-site evermed-hie-*-prod-gcs + a backup-* bucket.
  • Cross-site exchange = Pub/Sub topics crossrefer + crossrefer-error (the inter-hospital referral bus).
  • Artifact Registry hie-app (Docker, ~19.5 GB, actively pushed).

Not used (APIs disabled): Cloud SQL, Firestore, Cloud Run, Redis, Cloud Asset. No Firebase in this project. BigQuery enabled but empty.

3. Target architecture (AWS Thailand ap-southeast-7)

GCP today AWS Thailand target
GKE prod-hie-apps EKS (preserve per-namespace tenant model)
Artifact Registry hie-app ECR
nginx-ingress LB 34.87.180.155 + cert-manager nginx-ingress on EKS behind an ALB/NLB + ACM/cert-manager; per-site DNS cutover
9 standalone Mongo VMs DocumentDB (per-site clusters or consolidated) or EC2 lift-shift — per §1 spike
NATS (vajira) + RabbitMQ (ccs, smpk) Amazon MQ (RabbitMQ) + self-hosted NATS on EC2 (or MSK if re-platforming)
GCS buckets S3 (in-region)
Pub/Sub crossrefer SNS + SQS (topic→queue per consumer) — preserve referral-bus semantics
Secret Manager AWS Secrets Manager
Managed Prometheus + Grafana/Loki AWS Managed Prometheus + Grafana, or self-host the existing Grafana/Loki on EKS

4. Migration plan (phased, per-site)

  • Phase 0 — Landing zone. AWS account + VPC in ap-southeast-7, EKS cluster, ECR, networking, Secrets Manager, base observability. Verify EKS + DocumentDB availability. Run the DocumentDB compat-spike on one site’s Mongo.
  • Phase 1 — Pilot prod-hie-center (newest, single deployment). Prove the full runbook end-to-end before touching busy sites.
  • Phase 2 — Roll remaining sites one at a time: cbn, ccs, cti, ksn, pri, rayong, vajira, then smpk last (it’s the fuller “ever” full-HIS stack), then teleconsultation-ksn.
  • Phase 3 — Central + decommission. Migrate hie-core sync + replace crossrefer (SNS/SQS); cut DNS; decommission GCP after a soak period.

Per-site runbook skeleton: push images → ECR · apply manifests → EKS namespace · move Mongo (mongodump/mongorestore, or AWS DMS for low-downtime) · port secrets/config · smoke test internally · flip A record (hostname unchanged; pre-stage TLS + pre-register AWS egress IP with MOPH first) · verify referral flow · keep GCP site warm for instant A-record rollback.

Principle — move the data once: the Mongo move (Phase 1/2) is also when the de-identified AI corpus is tapped (§5), so data isn’t extracted twice.

Unmeasured input (gates the cutover window): per-site Mongo data volume. Get it via the bastion:

gcloud compute ssh jump-host --zone=asia-southeast1-b --project=hie-prod
# then per Mongo VM (10.20.1.x): mongosh --quiet --eval 'db.stats(1024*1024*1024)'  # GB

5. Data foundation (the corpus that powers the AI)

Two dumps — keep them distinct (the compliance trap):

  1. Migration dump — full-PHI, 1:1, mongodump→restore, stays in-region. Operational.
  2. AI corpus — a de-identified derivative, never a raw copy of #1.

Medallion (per ai-training-corpus.md), landed during cutover:

  • Bronze = operational Mongo (referral records).
  • Silver = de-identified + terminology-normalized (chief-complaint / Dx free-text → ICD/SNOMED via @medical-kit/code-systems). Drop direct identifiers, pseudonymize, generalize quasi-IDs. + synthesize/translate the Thai clinical free-text → English so the corpus is globally usable (source HIS is Thai; coded fields carry through as-is, only the ~5 enumerable free-text fields need translation).
  • Gold = labeled corpus: symptom → provisional Dx → final Dx chains across 9 hospitals — the RAG grounding (and later ML training) signal.

Landing + compute: raw-PHI dumps → s3://ever-hie-prod-th; de-identified + English-synthesized corpus → s3://ever-hie-ai-corpus-th (both ap-southeast-7, locked owner-only + CMK + TLS-only). Training compute = the local laptop (300 GB headroom; P1 embeddings/RAG fit locally; heavy P2 ML training may still want a GPU box). De-identify before the English-synthesis/embedding step regardless of laptop safety — that’s PDPA, not disk hygiene.

Governance (non-negotiable): opt-in consent (default off), in-region S3 only, right-to-erasure crosswalk, named data owner per tenant, per-inference audit, Thai PDPA + DPIA.

6. Clinical AI — one engine, two surfaces

                         ┌──────────────────────────────────────┐
  patient symptoms ─────▶│ symptom-mapper → differential/risk    │
  (web/app intake)       │ engine, RAG-grounded on Gold corpus    │
                         └───────────────┬──────────────────────┘
                          ┌──────────────┴───────────────┐
              CLINICIAN surface                  PATIENT surface
        full ranked differentials +        TRIAGE + ROUTING (not Dx):
        ICD + reasoning → doctor            "see a doctor in 24h /
        confirms (draft-until-signed)       go to ER / call 1669"
                          │                            │
                          └──── confirmed Dx ───────────┘
                               = training label (the flywheel)

Reuse, don’t rebuild — substrate already ~70% in-repo (verified real code):

Capability Existing module Gap to close
Symptom intake web/src/services/ai/symptom-mapper/ (chief-complaint → grounded SNOMED symptoms; Ollama + heuristicMapNote offline) RAG-ground on Gold corpus
Pre-diagnosis (differentials) web/src/services/ai/smart-diagnosis/ (catalog-grounded via terminology-tier-config) Siloed (useSmartDiagnosisPrefill results don’t reach the form) + not yet in ASSISTANT_REGISTRY — wire it in
LLM runtime services/llm (Ollama+RAG) + shared/runner-engine.ts (agent loop) Add the referral-corpus RAG retriever
Unified surface web/src/services/ai/registry.ts (ASSISTANT_REGISTRY — only clinical-query live) Fold in the diagnosis domain (planned P0)

Clinician surface = full ranked differentials + ICD + reasoning → doctor confirms. Recommender, draft-until-signed. (Mostly built — finish wiring.)

Patient surface = triage + routing + safety-netting, NOT a diagnosis label. This is the HIE’s existing referral/routing function (good fit): symptom intake → urgency + which hospital/department. Output is “what to do,” never “you have X.”

Flywheel: patient symptoms pre-fill clinician intake → clinician confirms Dx → confirmation becomes a labeled training example → improves both surfaces.

AI phasing: P1 RAG-grounded clinician differentials → P2 patient triage surface (SaMD, §7) → P3 ML predictive models on the Gold corpus.

7. Patient-facing = Software as a Medical Device (SaMD)

The patient surface must be built to a device standard from day one:

  • Triage, not diagnosis. No treatment/medication advice to patients.
  • Red-flag / emergency escalation is the priority feature — reliably catch chest pain / stroke / sepsis → ER / 1669 (Thai EMS). Err toward over-escalation.
  • Fail-SAFE (opposite of the clinician fail-open): on uncertainty or outage, default to “seek care.”
  • Always a human escalation path. Clear “this is not a diagnosis” disclaimer.
  • Consent + PDPA, every interaction audited, versioned + clinically validated before release.
  • Regulatory (external action item, not Claude’s call): obtain a Thai FDA (อย.) SaMD classification. The triage-not-diagnosis scoping is what typically keeps it in a lower risk class. Build it to be classifiable/defensible; classification itself needs a regulatory/legal read.

8. Invariants (carry the repo house rules)

  1. Recommender-only — never autonomous. AI proposes; humans decide.
  2. Catalog-grounded — no hallucinated ICD/SNOMED codes.
  3. Clinician path = draft-until-signed; patient path = fail-safe (escalate on doubt).
  4. Per-tenant feature flag + role gate on every AI surface.
  5. Every inference audited (ai_* decision logs).
  6. No raw PHI in prompts — governed/de-identified corpus only; in-region.
  7. AI never gates/blocks a clinical or policy decision.

9. Open items / risks

  • [ ] Verify EKS GA in ap-southeast-7 (else self-managed k8s on EC2). (2026-05-31: eks list-clusters + docdb describe-db-clusters both respond in-region — encouraging, not GA-confirmed.)
  • [~] DocumentDB compat — pri (refer) GREEN + connection proven; smpk (full HIS) has BLOCKERS ($text / $where / $function, see §11) → HYBRID: DocumentDB for refer sites, Mongo-on-EC2 for smpk. smpk live getIndexes ✅ DONE (2026-06-11 — all 28 visit indexes btree, no text/geo/hashed/wildcard/partial; see §11) → EC2-Mongo lift-shift is index-clean. Still open: sync-layer change-stream vs oplog; full live probe board (refer).
  • [ ] Per-site Mongo data volume unmeasured — sets cutover window (§4 command).
  • [ ] crossrefer Pub/Sub → SNS/SQS semantic mapping (ordering, retries, DLQ).
  • [ ] Live-traffic coordination — sites carry production referral traffic; per-site cutover windows + rollback.
  • [ ] DNS / endpoint coupling (referral mesh) — breaks the “fully independent per-site” assumption. Each gateway reaches peers + central hub + MOPH by hostname (per-site *.hie.everapp.io sync URLs, host-routed ingress, external moph-refer.inet.co.th / ppkportal.moph.go.th), and cross-refer traffic likely advertises each site’s own callback URL → a migrated site’s URL must resolve + be reachable for peers still on GCP and for MOPH. Mitigation = migrate by flipping A records, keep *.hie.everapp.io hostnames IDENTICAL (org controls everapp.io DNS — it’s a record change, not a rename); lower TTLs a few days pre-cutover; pre-register the AWS NAT egress IP with MOPH/INET (external allowlist = the long-lead item); pre-stage TLS certs on AWS (ACM/cert-manager) before the flip; keep GCP warm for instant A-record rollback. Verify (Cloud Shell): (a) does the cross-refer message format embed sender callback URLs? (b) DNS authority for everapp.io? © does MOPH allowlist HIE by source IP?
    • FINDINGS 2026-05-31: (b) everapp.io is DUAL-DELEGATED — dig NS everapp.io returns 4× AWS Route 53 (awsdns-*) AND 4× Google Cloud DNS (ns-cloud-*.googledomains.com); SOA owner = Route 53 (ns-50.awsdns-06.com). ⇒ records may live in two zones that can drift; before any cutover, identify which zone actually serves *.hie.everapp.io (dig A <host> @<each-ns>) and consolidate into Route 53 — it aligns with the AWS move and gives a single A-flip control point. hie.everapp.io has no separate NS delegation (records sit in the parent zone). Secrets follow prod-hie-<site>-refer-backend / -refer-sync-api (GSM); no moph-named secret → MOPH URL is a value inside the refer-backend secret/env (read via kubectl exec … printenv). (a)/© still open: sync-service /usr/src/app grep for crossrefer (*.js) returned nothing → publisher is likely the refer-backend, not sync-service; cluster egress check failed (cmd bug) — re-run per below; Cloud Shell→moph-refer.inet.co.th = 403 (inconclusive alone; pod has no curl).
    • ROUND 2 (confirmed 2026-05-31):
      • (b) DNS — ⚠️ this first read was WRONG; see ✅ CORRECTION at the end of this item — cbn.hie.everapp.io → 35.247.189.47, served by Cloud DNS (@ns-cloud-b1 answers); the Route 53 NS returns nothing for it. So *.hie.everapp.io lives ONLY in Cloud DNS while the apex is dual-delegated to Route 53 too ⇒ resolvers that pick an AWS NS get NXDOMAIN ⇒ resolution is non-deterministic today (possible existing intermittent-referral-failure source). Also 35.247.189.47 ≠ frontend ingress 34.87.180.155 ⇒ sync endpoints are a SEPARATE LB; each site has ≥2 external IPs to flip. Fix = consolidate everapp.io (incl. hie.*) into Route 53, then flip A-records there (one control plane, aligns with the AWS move). REFINED 2026-05-31: the Route 53 everapp.io zone is NOT in account 523231704210 (it holds only evernetwork.io + ever.healthcare; standalone, not in an AWS Org) → the apex’s AWS half lives in a different Ever AWS account (locate via other CLI profiles). BUT the records we actually flip (*.hie.everapp.io) are in GCP Cloud DNS, which the user already controls → the per-site HIE cutover flips happen in Cloud DNS (change the A-record value GCP-IP → AWS-IP); the mystery Route 53 account is only needed for the apex dual-delegation cleanup (secondary hygiene, not on the cutover critical path). whois (2026-05-31): registrar = Key-Systems GmbH (privacy-proxied via whoisproxy.com), domain created 2017, updated 2026-01-31. The .io registry delegates everapp.io to the 4 Route 53 awsdns-* NS ONLY — the 4 Cloud DNS NS appear only inside the Route 53 zone’s apex NS RRset (a misconfig). That’s the root cause of the flakiness: resolution starts at Route 53, but hie.* records exist only in Cloud DNS, so any resolver that follows a Route 53 NS gets NXDOMAIN for *.hie.everapp.io. Local AWS has exactly one profile (523231704210 = evernetwork.io + ever.healthcare) → the everapp.io Route 53 zone is in a different Ever AWS account (likely the dev/devops account that fronts his-nonprod/devops-nonprod.everapp.io); add a CLI profile or use that account’s console to reach it. The per-site migration flip still happens in the user-controlled GCP Cloud DNS zone regardless.
      • ✅ CORRECTION 2026-05-31 (definitive — dig +trace from public resolvers): the DNS is properly configured, NOT flaky. Chain: everapp.io → Route 53 (4 awsdns; holds apex + Google-Workspace MX + infra subdomains). hie.everapp.io is cleanly DELEGATED to GCP Cloud DNS — a real NS → ns-cloud-b1..b4.googledomains.com record in the Route 53 zone, TTL 60. cbn.hie.everapp.io = 35.247.189.47 resolves identically from system / 1.1.1.1 / 8.8.8.8. The earlier “NXDOMAIN / latently flaky” was a dig +short artifact (a referral lives in the authority section, which +short NS hides). Migration impact: per-site cutover = change the A value in the user-controlled Cloud DNS zone (60 s TTL → fast flip + instant rollback); Route 53 / the other AWS account / Squarespace are NOT on the HIE critical path. Domain is managed via Squarespace (backend registrar Key-Systems), NS → Route 53. ⚠️ Never “switch to Squarespace nameservers” — it repoints everapp.io to an inert record set and breaks live MX + every subdomain (his-nonprod, devops-nonprod, hie.*). 35.247.189.47 (sync LB) ≠ 34.87.180.155 (frontend ingress) → each site has ≥2 public IPs to flip. Open: which GCP project owns the hie.everapp.io Cloud DNS managed-zone (likely hie-prod).
      • (a) sites do NOT HTTP-callback each other — refer-backend env has only the in-cluster sync-api ClusterIP (10.100.4.88) + MOPH; cross-site = Pub/Sub crossrefer + sync layer, so a migrated site needn’t advertise a new URL to peers (bus decouples them). *.hie.everapp.io = frontend/external-inbound endpoints (NOT the cross-site path). CONFIRMED 2026-05-31: no everapp.io URL in refer-backend / sync-api / hie-sync-service envs → inter-site referral is 100% Pub/Sub (hie-sync-service consumes SUBSCRIPTION_NAME=crossrefer-sub, errors → ERROR_TOPIC_NAME=crossrefer-error; sync-adapter-file handles file sync). ⇒ zero inter-site URL coupling. The ONLY shared/central piece is the crossrefer Pub/Sub bus itself — all sites share it, so it can’t migrate per-site: it stays on GCP Pub/Sub (accessed cross-cloud by migrated sites, +egress cost) until the final central cutover, OR gets bridged to SNS/SQS. This is THE exception to per-site independence (matches the §3 Phase-3 / §9 crossrefer items).
      • © MOPH = TOKEN auth, pointed at UAT — MOPH_REFER_CLIENT_ID/SECRET/TOKEN (OAuth+JWT, env:uat), MOPH_REFER_URL = https://moph-refer-uat.inet.co.th/backend/api/v1/service. ⇒ IP-allowlisting is NOT the gate → “pre-register AWS egress IP” drops from mandatory to “confirm w/ INET, likely unneeded.” No Cloud NAT router exists (ephemeral node egress, can’t be allowlisted anyway). Open Q: why do prod sites point at MOPH UAT? MOPH cutover = just carry the MOPH_REFER_* secrets.
  • [ ] SaMD/Thai-FDA classification + clinical validation plan for the patient surface.
  • [ ] PDPA DPIA + consent capture for the AI corpus.
  • [ ] Cost model: AWS Thailand vs current GCP spend (confirm the savings thesis with real numbers).

10. Next actions

  1. Phase 0 landing-zone design → WRITTEN: phase-0-landing-zone.md (VPC 10.50.0.0/16 + EKS + ECR + DocumentDB-refer + EC2-Mongo-smpk in ap-southeast-7; user sequencing = services→data→telemedicine). Spikes done (DocumentDB compat ✅, EKS API present). Blocking decisions D1–D4 in that doc (account, cross-cloud-transition, DocumentDB shape, crossrefer bus).
  2. Size the 9 Mongo VMs (cutover-window input).
  3. Write the per-site migration runbook (expand §4) and pilot it on prod-hie-center.
  4. Stand up the RAG retriever over the Gold referral corpus; wire smart-diagnosis into ASSISTANT_REGISTRY (clinician surface P1).

11. Compat-spike log

2026-05-31 — DocumentDB Stage 1 (schema audit, $0)

  • Schema verdict for pri (local Colima mongo:7 restore — 320k visits, 25 collections): GREEN. No DocumentDB-hostile constructs — zero text / 2dsphere / 2d / geoHaystack / hashed / wildcard / collation indexes, no capped collections. Only 3 partial indexes (hospitals.categories.code, mophreferlogs ×2) → DocumentDB doesn’t support partialFilterExpression; recreate as plain/sparse (perf-only, cosmetic).
  • DocumentDB gotchas confirmed (AWS functional-differences doc; engine now emulates 3.6 / 4.0 / 5.0 / 8.0 APIs):
    • retryWrites=false is MANDATORY in the connection string — DocumentDB rejects retryable writes (mongoose / 4.2+ drivers default to true). The #1 connection gotcha.
    • Transactions OK (multi-doc ACID since 4.0).
    • No implicit result ordering — app must use explicit sort(); $sort only preserved as the last stage ($group caveats).
    • No reverse $natural ({$natural:-1} errors); no correlated $lookup subqueries; one index build at a time; no admin/local db (recreate user roles after mongorestore).
    • Bonus: DocumentDB has vector search (vectorSearch inside $search) — a candidate RAG store for the AI corpus.
  • THE GATE (unresolved): change-stream vs oplog in the sync layer. DocumentDB has no oplog; the change-streams API works but is opt-in + time-bounded. If hie-sync-service / *-sync-queue-worker / prod-refer-sync-api tail the oplog, DocumentDB breaks them → fall back to Mongo-on-EC2. This is a code fact, not in pri data, and the HIE refer/sync source is NOT on this laptop (siblings ever-medos-pure=medos-clinic, medOs-master=his-vajira are the HIS frontend/clinic, not the refer stack). Resolve before any Stage-2 spend: grep the HIE refer-backend + sync-service source, OR pull the hie-app image from GCP Artifact Registry and grep compiled JS for oplog | changeStream | \.watch\( | retryWrites | moleculer-db-adapter | mongoose.
  • ⚠️ pri is the SMALLEST “refer” site (1.2 GB). Schema-green here does NOT cover smpk (gen-2 “ever” full-HIS, 82 GB) — audit it separately before committing DocumentDB platform-wide.
  • Stage 2 (live DocumentDB probe) deferred until the change-stream question is answered — provisioning real DocumentDB (VPC + bastion, no public endpoint, ~$/hr) just to confirm a code fact would be premature.

2026-05-31 — AI-corpus bucket created

  • s3://ever-hie-ai-corpus-th (ap-southeast-7) created + hardened to parity with the raw bucket: block-public-access (all 4), versioning on, owner-only Deny (aws:PrincipalArn ≠ :root), TLS-only Deny, SSE-KMS via existing CMK alias/ever-hie-prod-th, ACLs disabled (BucketOwnerEnforced). Holds the de-identified + English-synthesized corpus only — never raw PHI. ⚠️ Add the training principal’s ARN to the bucket policy before pointing non-root creds at it (same lock-out footgun as the raw bucket).

2026-05-31 — DocumentDB Stage 2 (live, partial → torn down)

  • Provisioned real DocumentDB 5.0 + bastion in ap-southeast-7 (tagged Project=hie-docdb-spike). Connection PROVEN: mongosh over TLS (RDS global CA bundle) with retryWrites=false&replicaSet=rs0 → ping ok — the mongoose-5-style handshake works against DocumentDB. (mongo-tools install needed gpgcheck=0; the repo’s -RC GPG key was wrong.)
  • mongorestore of pri (--noIndexRestore) was mid-flight when the run was interrupted (twice) → full probe board NOT captured. Stage-1 schema-green + this connection proof keep DocumentDB the front-runner; the aggregation/transaction/change-stream/partial-index board is the only thing still unrun.
  • Torn down to stop billing (instance → cluster → subnet group → SG → bastion → key pair). Re-provision ≈15 min if the live board is wanted. Scripts kept at /tmp/hie-spike-*.sh + /tmp/probe.js.

2026-05-31 — smpk (full HIS / gen-2 “ever”) app-layer proxy audit → DocumentDB BLOCKERS found

Grepped the ever-family backend on disk (medOS-ultra/services + ever-medos-pure) as a proxy for prod-ever-his. The full HIS uses three things DocumentDB does NOT support — none of which the refer stack had:

  • $text search (requires a text index; DocumentDB supports neither) — financial/revenueCollection.transform.ts, foundation/conference.repository.ts, foundation/leaveRequest.repository.ts, administration/anesthesiologistRequest.repository.ts.
  • $where (server-side JS predicate) — administration/patientTransfer.service.ts:91.
  • $function (aggregation server-side JS) — administration/encounter.service.ts:2053.
  • Clean: no capped collections, no geo, no change-streams in the app code (the smpk sync-worker CS/oplog usage is still unverified — its source isn’t in these repos). Drivers: mongoose ^5.13.8 + ^8.18.1, moleculer-db-adapter-mongoose ^0.9/0.10 — fine on DocumentDB only with retryWrites=false, but the $text/$where/$function blockers are version-independent.

VERDICT — hybrid Mongo strategy: DocumentDB for the refer sites (cbn/ccs/cti/ksn/pri/rayong/vajira — pri schema was GREEN), self-managed Mongo-on-EC2 for smpk (lift-shift — avoids a multi-week refactor of $text→$regex/external search, $where/$function→supported ops). This changes Phase 0: provision DocumentDB (refer) + an EC2 Mongo (smpk) in ap-southeast-7, not DocumentDB-only. Confirm physically on live smpk via the data-layer index audit (jump-host getIndexes, metadata only — no 82GB transfer) before locking it in.

2026-05-31 — DNS migration started (hie.everapp.io: GCP Cloud DNS → Route 53)

DNS is a first-class migration item (user’s call): external agents/integrators + the per-site sync-api endpoints resolve *.hie.everapp.io, and that zone is currently GCP Cloud DNS — a residual GCP dependency to shed. DNS-first sequencing is cleanest: move the zone before the app cutovers, so every per-site A-change lands in the AWS zone.

  • ✅ Target zone CREATED (inert): hie.everapp.io Route 53 hosted zone Z0183131WJ46LM6T5UFW in account 523231704210 (natural Ever DNS home — alongside ever.healthcare + evernetwork.io). Delegation NS: ns-1471.awsdns-55.org, ns-960.awsdns-56.net, ns-414.awsdns-51.com, ns-1782.awsdns-30.co.uk. Zero traffic impact until the parent everapp.io zone repoints hie.everapp.io’s NS here. (A masked first-create left a dup Z01817131XRZVD1S3X2TP — deleted.)
  • Steps: (1) ✅ target zone created; (2) extract current *.hie.everapp.io records from GCP Cloud DNS (gcloud dns record-sets list) — pending the project-locate; (3) replicate them into Z0183131WJ46LM6T5UFW (initially with the same GCP IPs, so cutover is then per-record); (4) one-time delegation flip in the parent everapp.io Route 53 zone (the other AWS account): hie.everapp.io NS → the 4 NS above (lower TTL first); (5) verify resolution unchanged via public resolvers; (6) per-site app cutover = change <site>.hie.everapp.io A → AWS IP in Z0183131WJ46LM6T5UFW; (7) retire the GCP Cloud DNS zone after soak.
  • Two inputs still needed: the current GCP record list (step 2) + access to the parent everapp.io Route 53 zone for the one-time NS flip (step 4). Note: the sub-delegation flip can ONLY be done in the parent zone (or by re-delegating all of everapp.io at Squarespace — bigger, avoid).

2026-05-31 — DECISION: Full consolidation of everapp.io → Route 53 (user’s call) + complete topology mapped

Full everapp.io topology (recon via dig against authoritative ns-50.awsdns-06.com):

everapp.io                     [Route 53, the orphaned/"mystery" AWS account — NOT 523231704210]
  ├─ MX  → Google Workspace (aspmx.l.google.com pri 1; alt1/alt2 pri 5; alt3/alt4 pri 10)
  ├─ TXT → SPF  "v=spf1 include:_spf.google.com ~all"
  ├─ TXT google._domainkey → DKIM (v=DKIM1; k=rsa; p=… 2048-bit)
  ├─ hie.everapp.io            → DELEGATED → GCP Cloud DNS (ns-cloud-b1..b4)   [HIE sites]
  ├─ his-nonprod.everapp.io    → DELEGATED → GCP Cloud DNS (ns-cloud-d1..d4)   [medOS sandboxes]
  └─ devops-nonprod.everapp.io → DELEGATED → GCP Cloud DNS (ns-cloud-d1..d4)   [devops infra]
  (no apex A / AAAA / DMARC / CAA; no other subdomains — his-prod/his-uat/sandbox/www all NXDOMAIN)

Two simplifications this revealed:

  1. The Squarespace NS switch bypasses the orphaned AWS account entirely — re-delegate everapp.io at the registrar (Squarespace → new Route 53 apex zone’s NS). The mystery account never needs creds; it just goes dark. So locating it is NO LONGER required.
  2. The apex is fully captured by recon (MX + SPF + DKIM only). The ONLY data still needed = the contents of the 3 GCP Cloud DNS zones (hie ns-cloud-b, his-nonprod + devops-nonprod ns-cloud-d — likely 1–2 GCP projects).

⚠️ Blast radius = whole domain: the NS flip repoints Google Workspace email + the medOS sandboxes (his-sandbox-*.his-nonprod / devops-nonprod) + the HIE hosts, all at once. So all 4 zones must be faithfully rebuilt in 523231704210 BEFORE the flip; verify email + every subdomain after; keep the old NS for instant rollback.

Consolidation runbook: (1) build everapp.io apex zone in 523231704210 = 5 MX + SPF TXT + DKIM TXT + 3 NS-delegations → to (2) the 3 rebuilt sub-zones (hie.everapp.io already exists = Z0183131WJ46LM6T5UFW; create his-nonprod.everapp.io + devops-nonprod.everapp.io), each populated from its GCP Cloud DNS export; (3) verify the new apex+subs resolve correctly by querying its NS directly (before any switch); (4) lower Squarespace TTL; (5) switch Squarespace nameservers → the new apex zone’s 4 NS; (6) verify email (send/receive) + every *.hie / *.his-nonprod / *.devops-nonprod host; (7) decommission GCP Cloud DNS zones + ignore the orphaned AWS apex zone after soak. Gate: the 3 GCP zone exports (find the project[s] via Console).

✅ STATUS 2026-05-31 — Stage-1 apex replica BUILT + verified (inert): everapp.io apex zone Z06290792NBY03ITQXHY in 523231704210 now holds MX (Google Workspace) + SPF TXT + DKIM + the 3 NS-delegations (hie→ns-cloud-b*, his-nonprod/devops-nonprod→ns-cloud-d*, mirroring live so the cutover is transparent). New apex NS (the Squarespace switch target): ns-384.awsdns-48.com, ns-827.awsdns-39.net, ns-1049.awsdns-03.org, ns-1705.awsdns-21.co.uk.

  • 🐛 DKIM bug found + fixed: live google._domainkey.everapp.io was mis-split into 3 separate TXT records (DKIM must be ONE record) → almost certainly failing verification today (hurts deliverability). Rebuilt as one correct record in the replica; key verified byte-identical to the live concatenation (410 chars, also matches Squarespace’s stored config). ⚠️ Do a final sanity-check of this key against the Google Workspace admin console before the switch.
  • This is staged: Stage-1 switch (Squarespace NS → the 4 NS above) takes everapp.io off the orphaned AWS account; subdomains keep resolving via GCP (delegations preserved) — zero subdomain disruption. Stage-2 then migrates hie/his-nonprod/devops-nonprod off GCP one at a time (needs each zone’s GCP export + repoint that one delegation in the apex). The Squarespace NS switch is the user’s action (I can’t touch Squarespace).
  • Pre-switch completeness VERIFIED (2026-05-31): no DS records on the 3 delegations, everapp.io is not DNSSEC-signed (no DNSKEY / no DS at .io), no other apex record types (CAA/SRV/NAPTR/SVCB/HTTPS/PTR all empty). ⇒ the replica is a complete faithful copy, so the switch is transparent to GCP — hie/his-nonprod/devops-nonprod keep resolving via the preserved GCP Cloud DNS delegations; GKE/apps/sandboxes/email untouched. (The hie Cloud DNS zone is internally DNSSEC-signed but unanchored — no parent DS — so no DNSSEC impact.) GCP services migrate off ONLY in Stage 2, deliberately, per-subdomain.

2026-06-11 — smpk (ever3 / gen-2 full-HIS) LIVE schema probe + de-id corpus adapter

Probed live smpk Mongo (10.20.1.4, db ever3) read-only via the jump-host (PHI-safe: structure + codes only, identifiers hidden), then built + ran the smpk corpus extractor. AI-corpus workstream (Silver de-id), not the infra cutover.

Collection reality (only these populated): visits 14,446,653 · patients 1,726,185 · histories 6,084,495 (audit bloat — SKIP) · hospitals 41,230 (national directory + this hospital’s billingRules/billingItemRules fee schedule) · billings 1,040 (aggregated monthly provider/payer rollups only) · users 367. EMPTY: bills, billitems, billingcategories, billingitemrules, invoices, the standalone referrals coll (referrals are EMBEDDED in visits), ipfs*, reportlinks, roles, syncusers.

⚠️ CORRECTS the 2026-06-10 note (“smpk visits have NO top-level diagnoses[]/drugs[]/billingItems[]”): they DO — the first findOne doc just had them empty and they sit AFTER field 26. An ICD-code hunt over 80 visits found diagnoses[].icd10 (28 hits: I10/B24/I64) + referrals[].icd. smpk’s visits is a SUPERSET of the refer shape, with extra embedded arrays.

Adapter signal map (extract_corpus_smpk.py):

  • Coded Dx = visits.diagnoses[] → icd10/icd10Name(EN)/icd10ThaiName/diagTypeName — SAME shape as refer.
  • Free-text Dx = visits.diagnosisOPDTexts[].diagText (+ top-level diagText).
  • Drugs = visits.drugs[] → tmtCode/genericName/drugNondugName/income (not refer’s medname/medtype_name).
  • Procedures = visits.operations[] → operationItemID/operationItemName (ICD-9-CM in the name).
  • Labs = visits.labResults[].labReportData[] → labItemsName/labOrderResult/labItemsUnit/labItemsNormalValueRef.
  • Referrals (HIE signal, NEW vs refer) = visits.referrals[] → icd/preDiagnosis/toHospCode/fromHospCode/department/referType/status.
  • FINANCIAL = embedded visits.billingItems[] (icode/qty/unitPrice/sumPrice/genericName/tmtCode/income/payment_status) — the refer extractor’s bills/billitems collections are EMPTY here; billings(1040) is monthly rollups, useless for per-visit cost.
  • Demographics = patients joined by patients.bid == visits.cid; patients.data.{age,DOB,sexName} + top-level chronics[].{clinicName,clinicSubType} (e.g. “DM Type II”) + data.drugAllergies[].genericname.

Two data-quality bugs found + fixed in the extractor:

  1. Join key — patients’ cid/nonThaiId/data.cid are ALL empty; the national id lives in bid (proven by a server-side countDocuments({bid: visit.cid})==1). First attempt keyed on cid → 0-entry patient map → null demographics. Fixed → 1,726,183-entry map joins.
  2. Corrupt out-of-range dates — some visits carry BSON dates with year 0 / year 5479 that crash pymongo’s DEFAULT datetime decoder mid-stream (InvalidBSON, fires INSIDE the cursor, OUTSIDE the per-row try → one bad doc kills the whole run; the first full run died at ~83k visits). Fixed with MongoClient(datetime_conversion="DATETIME_CLAMP") + a 1990–2026 ym() guard that nulls the clamped junk. Re-validated past the crash point (LIMIT=250000, 0 errors).

Index audit (closes the §9 “confirm on live smpk via getIndexes” open item): visits = 28 indexes, ALL standard btree (cid, hn, visitDateTime, hospCode+hospmain+pttype+visitDateTime, many referrals.*); patients 6 (cid/nonThaiId/hn/passport/bid); billings 1; hospitals 3. NO text / 2dsphere / hashed / wildcard / partial indexes → the smpk→EC2-Mongo lift-shift is index-clean (the $text/$where/$function DocumentDB blockers are in APP code, not indexes — confirmed 2026-05-31).

Corpus (extract_corpus_smpk.py, in docs/architecture/hie-migration-clinical-ai/): single streaming pass over 14.4M visits → de-id clinical + financial in ONE pass (the refer two-pass bills approach doesn’t apply). De-id parity with extract_corpus.py (HMAC-pseudonymized patient(cid/bid)+visit ids, age-band, ym, regex-scrubbed free text); staff-name fields (itemsNoteText/itemsNoteUser/doctorName/ptname) explicitly EXCLUDED. Toggles HIE_SMPK_LABS/HIE_SMPK_VITALS (default on), HIE_LIMIT for MVP chunks. Full in-place run on the jump-host (raw PHI never leaves GCP), ~4.3GB de-id extrapolated. [RESULT counts — pending run completion.] Remaining ever-stack sites for full corpus coverage: ccs + vajira.

Ask Anything