medOS ultra

ML Training Corpus

Separate medallion layer: Bronze to Silver (de-identified) to Gold immutable manifests; JSONL/Parquet/RAG outputs; opt-in consent governance.

8 min read diagramsUpdated 2026-05-30docs/architecture/ai-training-corpus.md

Date: 2026-05-29 Author: Claude (design session) Scope: A governed, de-identified, versioned data-dump layer that turns operational events into training/eval datasets — separate from the real-time serving path. Status: Design only — not built. Cross-cutting: serves the navigation recommender (Domain 6), the 5-domain AI engine, RUDS, and the future form-autofill tier.

1. Why a separate layer (serving ≠ training)

The operational serving path (navigation_patterns MV + RPC, the gold-layer MVs) is built for low-latency reads on recent data. A training corpus has the opposite requirements. Mixing them is a mistake on every axis:

Axis Serving path Training corpus
Horizon recent (e.g. 90d) full history, frozen
Mutability live, always-current immutable, versioned snapshots (reproducibility)
Granularity aggregated counts per-example, sessionized, labeled
Identifiers present (operational) de-identified at the boundary
Retention short (30d raw) long-horizon (survives raw retention)
Location in operational DB exportable to object storage (S3), in-region
Access hot path batch ETL, off-peak, governed
Consent operational necessity per-tenant opt-in

So: a dedicated medallion ML-corpus layer fed from the operational streams on a schedule, never in the request path.

2. Reuse vs. New

Concern Existing artifact Reuse or New
Medallion refresh pattern 013_gold_layer.sql (gold_refresh_log, fn_refresh_gold_layer(), gold-layer-refresh edge fn) Reuse pattern → ml_refresh_log, fn_refresh_ml_corpus(), ml-corpus-export edge fn
RAG corpus + embeddings llm_corpora / llm_embeddings (pgvector hnsw) / llm_use_cases / llm_search_embeddings Reuse (the RAG output path)
PII scrub services/llm/.../_shared/redaction.ts (redactForAudit/redactObject) Reuse as ONE layer (insufficient alone — see §6)
Scheduling cron_jobs registry (036_…) Reuse
Source: attention ui_interaction_events (navigation-next-best-action.md) Consume (bronze)
Source: actions hospital_events Consume (bronze)
Silver/gold ML tables — New (ml_session_trajectories, ml_training_examples)
Dataset version registry — New (ml_dataset_versions)
De-id at export boundary — New (structural; §6)
Consent + access audit — New (ml_export_consent, ml_access_log)

3. Medallion flow (training variant)

BRONZE (operational, short-lived, identified)
  ui_interaction_events   ── attention signals (client)
  hospital_events         ── action signals (Moleculer-authoritative)
            │  scheduled ETL (cron_jobs → fn_refresh_ml_corpus)   ❰ never hot path ❱
            ▼
SILVER (sessionized + joined + DE-IDENTIFIED)   ← the safety boundary
  ml_session_trajectories
    one row per de-identified session: ordered [(context, element, interaction)]
    identifiers dropped/HMAC-pseudonymized · quasi-identifiers generalized
    rare contexts (k<K) suppressed
            ▼
GOLD / TRAINING (labeled, split, frozen)
  ml_training_examples  (context + prefix[] → next_action label, split=train|val|test)
  ml_dataset_versions   (immutable manifest: version, range, schema_hash, deid_policy,
                         consent_scope, split_seed, storage_uri, owner, lineage)
            ▼
OUTPUTS (any of three; §7)
  ① JSONL/Parquet → S3 (classical seq-model / fine-tune)
  ② llm_corpora row → llm_embeddings (RAG "typical workflows")
  ③ held-out test split → offline eval (accuracy@k) before anything ships

4. Schemas (sketch)

-- ── SILVER: de-identified sessionized trajectories ──────────────────────────
CREATE TABLE ml_session_trajectories (
  traj_id         UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  snapshot_version TEXT NOT NULL,              -- FK → ml_dataset_versions
  subject_pseudonym TEXT,                      -- per-snapshot salted HMAC of user_id (NOT the real id)
  -- generalized context (quasi-identifiers coarsened)
  role            TEXT,
  dept_type       TEXT,                        -- NOT department_id
  encounter_class TEXT,
  encounter_subtype TEXT,                      -- suppressed if cohort < K
  shift           TEXT,
  dow             SMALLINT,                    -- day-of-week (absolute date dropped)
  -- the trajectory: ordered, identifier-free
  steps           JSONB NOT NULL,              -- [{element, interaction, tod_bucket, value}]
  step_count      INT  NOT NULL,
  source_signals  TEXT[] NOT NULL DEFAULT '{attention}'  -- {attention, action}
);

-- ── GOLD: labeled training examples (next-action) ───────────────────────────
CREATE TABLE ml_training_examples (
  example_id      UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  snapshot_version TEXT NOT NULL,
  context         JSONB NOT NULL,              -- role/dept_type/encounter_class/subtype/journey_state
  prefix          TEXT[] NOT NULL,             -- the k preceding elements
  next_action     TEXT NOT NULL,              -- the label
  split           TEXT NOT NULL,               -- train | val | test  (by subject_pseudonym, not row)
  weight          NUMERIC DEFAULT 1.0
);

-- ── REGISTRY: immutable dataset manifests (reproducibility) ─────────────────
CREATE TABLE ml_dataset_versions (
  snapshot_version TEXT PRIMARY KEY,           -- e.g. 'nav-2026-05-29-001'
  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  domain          TEXT NOT NULL,               -- 'navigation' | 'order-set' | 'coding' | ...
  date_range      TSTZRANGE NOT NULL,          -- source window
  row_count       BIGINT,
  schema_hash     TEXT,                        -- example shape hash → reproducibility
  deid_policy     TEXT NOT NULL,               -- version of the de-id rules applied
  consent_scope   TEXT NOT NULL,               -- which tenants opted in
  split_seed      INT  NOT NULL,
  storage_uri     TEXT,                        -- s3://…/nav-2026-05-29-001/  (in-region)
  owner           TEXT NOT NULL,               -- named data owner (required)
  lineage         JSONB NOT NULL DEFAULT '{}'::jsonb,
  frozen          BOOLEAN NOT NULL DEFAULT TRUE
);

-- ── GOVERNANCE: opt-in + access audit ───────────────────────────────────────
CREATE TABLE ml_export_consent (
  tenant_id       UUID PRIMARY KEY,
  enabled         BOOLEAN NOT NULL DEFAULT FALSE,  -- opt-IN (default off)
  scope           JSONB NOT NULL DEFAULT '{}'::jsonb,
  approved_by     TEXT, approved_at TIMESTAMPTZ
);
CREATE TABLE ml_access_log (                    -- every export + every read of ml_* logged
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  actor TEXT, action TEXT, snapshot_version TEXT, detail JSONB
);

All ml_* tables: RLS service_role-only (no authenticated read — these are not operational data).

5. The pipeline (no hot-path contention)

  1. cron_jobs row schedules fn_refresh_ml_corpus(domain, window) nightly/weekly, off-peak.
  2. fn_refresh_ml_corpus:
    • reads bronze (ui_interaction_events ⨄ hospital_events) for the window only for opt-in tenants,
    • sessionizes + joins attention⨄action into trajectories,
    • applies the §6 de-id pipeline,
    • writes ml_session_trajectories + ml_training_examples,
    • registers an immutable ml_dataset_versions row.
  3. ml-corpus-export edge fn (mirrors gold-layer-refresh: manual POST / cron / GET health):
    • serializes the snapshot to JSONL/Parquet in S3 (in-region prefix),
    • optionally registers an llm_corpora row + triggers embedding for the RAG path,
    • writes ml_access_log.

Everything is batch + service-role; the serving RPC and clinical writes never touch it.

6. De-identification at the export boundary (structural, not just regex)

redaction.ts scrubs obvious PII strings, but a training export that leaves the operational boundary needs structural de-id — navigation sequences + timestamps + dept + role are quasi-identifiers (a night-shift nurse in a rare unit is re-identifiable; this is the exact “cohort re-id risk” flagged in ekardex-from-journey-cache.md):

  1. Drop direct identifiers — encounter_id, patient_id, department_id never enter silver.
  2. Pseudonymize the subject — user_id → HMAC(user_id, per-snapshot salt). Within-snapshot sequences stay coherent; cross-snapshot linkage is broken. Salt held in a restricted vault (see §8 erasure).
  3. Generalize quasi-identifiers — absolute timestamp → tod_bucket + dow; department_id → dept_type; keep only coarse encounter_subtype/acuity buckets.
  4. k-anonymity suppression — any context cell with cohort count < K (e.g. K=20, reuse the recommender’s s_min) is generalized-up or dropped. No singleton trajectories.
  5. Belt-and-suspenders — run redactObject() over residual string/metadata fields. But the corpus is safe primarily by construction: the capture layer already forbids free-text/PHI (structured ids only — see navigation-next-best-action.md §4.2). That’s the real guarantee; redaction is the backstop.

The deid_policy version is stamped on every ml_dataset_versions row so a snapshot’s exact treatment is auditable and reproducible.

7. Output formats — one corpus, three training modes

Mode Output Path
Classical seq-model / ranker JSONL {context, prefix[], next_action, split} → S3 trains the P3+ next-action ranker offline
LLM fine-tune instruction/completion pairs (system+context → "next: X") → S3 fine-tune a small Ollama model
RAG corpus (cheapest, reuses your stack) register llm_corpora row → llm_embeddings → retrieved by llm_search_embeddings at serve time “typical workflows” reference for the LLM re-rank (Gate 2)

The RAG path is the recommended first target — it requires zero new model training, just embedding the de-identified trajectories into the corpus the serving LLM already retrieves from.

8. Governance, residency, erasure

  • Opt-in — ml_export_consent.enabled defaults false; a tenant must explicitly enable contributing to training corpora.
  • Named owner — required on every ml_dataset_versions row (the docs mandate a named data/clinical owner per tenant).
  • Full audit — every export and every ml_* read in ml_access_log (mirrors the per-inference audit stance of RUDS / e-Kardex AI).
  • In-region storage — S3 bucket per region, matching the BAA/DPA + regional-endpoint posture established across the AI docs.
  • Right-to-erasure — keep a restricted subject → pseudonym crosswalk in a vault so an erasure request can purge a subject’s silver/gold rows. (Alternative: declare fully-anonymized snapshots out of erasure scope — defensible only if truly non-re-identifiable; the crosswalk is the safer default.)

9. Safety invariants (extend the recommender’s)

  1. De-identified at the boundary — structural de-id + k-anon, not just regex.
  2. No PHI/free-text by construction — inherited from the capture layer.
  3. Opt-in only, per tenant; named owner; in-region; full access+export audit.
  4. Immutable, versioned snapshots — a model is reproducible from its snapshot_version.
  5. A better-trained model still NEVER gates or acts — training improves the ranker; the gate ladder and recommend-only contract are unchanged.
  6. Held-out eval before ship — every snapshot carries a test split; report accuracy@k offline before any model reaches serving.

10. Rollout

Phase Deliverable
T0 ml_* tables + ml_export_consent (default off) + RLS
T1 fn_refresh_ml_corpus (sessionize attention⨄action) + §6 de-id pipeline + first ml_dataset_versions snapshot
T2 ml-corpus-export edge fn → JSONL/Parquet to S3 + ml_access_log
T3 RAG path: register llm_corpora + embed → available to the Gate-2 LLM re-rank
T4 Held-out eval harness (accuracy@k) + fine-tune/seq-model export (optional)

T0–T1 are additive and safe (no tenant opted in → empty corpus). Nothing ships to a model until T3/T4 with eval.

  • docs/architecture/navigation-next-best-action.md — the capture layer this consumes (bronze) + the serving path it complements
  • docs/architecture/ai-recommendation-engine-validation.md — the 5-domain engine that also benefits from this corpus
  • docs/architecture/rogue-user-detection-system.md — user_action_events, the per-inference-audit + no-PHI-in-prompt stance mirrored here
  • docs/architecture/ekardex-from-journey-cache.md — cohort re-identification risk + named-owner governance precedent
  • services/llm/.../_shared/redaction.ts — the regex scrub reused as the §6 backstop
  • infrastructure/medbase/migrations/013_gold_layer.sql + functions/gold-layer-refresh/ — the medallion refresh pattern mirrored
  • infrastructure/medbase/migrations/20260514c_llm_platform.sql — llm_corpora/llm_embeddings for the RAG output path
  • docs/architecture/ambient-clinical-scribe.md — third bronze source: consultation transcripts (de-identified) feed ml_session_trajectories (P4 of the scribe rollout)
Ask Anything