medOS ultra

LLM Platform Deployment & UAT

Deployment + UAT companion to the LLM platform design.

6 min read diagramsUpdated 2026-05-14docs/llm-platform-deployment.md

Companion to: docs/architecture/llm-platform.md (design) Status: v1 (2026-05-14) — Ollama-only, self-pacing operator setup Audience: super-admin / DevOps

This walks an operator from “fresh medOS deployment” to “playground returns a real answer.”

1. What got built

Surface Path Notes
Supabase schema infrastructure/medbase/migrations/20260514c_llm_platform.sql 7 tables, pgvector, RLS, 2 RPCs
Default seed infrastructure/medbase/migrations/20260514d_llm_platform_seed.sql 5 models, 5 corpora, 11 use cases (bilingual)
Mongo entities packages/platform-api-schema/src/llm/ LlmModel, LlmUseCase, LlmCorpus, LlmAuditLog
Backend service services/llm/ Moleculer-native, exposes llm.* actions
REST routes services/public-api/.../modules/llm/ /api/llm/*, /api/admin/llm/*
Ollama runtime infrastructure/docker-compose*.yml ollama container + volume
Super-admin UI web/src/{containers,common/components}/super-admin/llm-*/ 4 pages, 4 routes
Service layer web/src/services/super-admin/llmPlatform.service.ts Supabase + REST glue

2. Prerequisites

  • Docker (with enough RAM/disk for the models — see §4)
  • Supabase project for the region (or shared dev project)
  • The medOS backend already running (Mongo, NATS, gateway, public-api)
  • Super-admin role for the user who’ll operate the dashboard

3. Deploy (one-time)

3a. Apply the Supabase migrations

# From the repo root
psql "$SUPABASE_DB_URL" -f infrastructure/medbase/migrations/20260514c_llm_platform.sql
psql "$SUPABASE_DB_URL" -f infrastructure/medbase/migrations/20260514d_llm_platform_seed.sql

Verifies:

SELECT count(*) FROM llm_models;       -- expect 5
SELECT count(*) FROM llm_use_cases;    -- expect 11
SELECT count(*) FROM llm_corpora;      -- expect 5
SELECT count(*) FROM llm_embeddings;   -- expect 0 until reindex

3b. Start Ollama + the LLM backend service

Local dev:

cd infrastructure
docker compose up -d ollama        # starts Ollama on :11434

cd ../services/llm
yarn                                 # install
yarn dev                             # starts the api-llm service

On-premise (production):

cd infrastructure
docker compose --env-file .env.onpremise.${REGION} -f docker-compose-onpremise.yml up -d ollama api-llm

3c. Configure env vars for the LLM service

Add to infrastructure/.env.{region} (or env-files/ever/.env for dev):

OLLAMA_URL=http://ollama:11434
SUPABASE_URL=https://<your-project>.supabase.co
SUPABASE_SERVICE_ROLE_KEY=eyJ...        # service role, NOT anon
LLM_EMBEDDING_DIM=768                    # or 1024 for bge-m3
LLM_REDACTION_ENABLED=true               # PHI redaction in audit log

3d. Frontend

Already deploys automatically on git push origin main (Vercel). The 4 routes appear at:

  • /super-admin/llm-models
  • /super-admin/llm-use-cases
  • /super-admin/llm-corpora
  • /super-admin/llm-playground

Tiles are also added to /super-admin-dashboard.

4. First-time setup (operator)

Step 1 — Pull the models

Go to /super-admin/llm-models. Default rows are visible with status registered. Click the download icon (⬇) on a row to call ollama pull. The status updates in realtime to pulling → available.

Suggested order:

  1. nomic-embed-text (~270 MB) — needed before any corpus reindex
  2. qwen2.5:7b-instruct-q4_K_M (~5 GB) — default chat model
  3. llama3.1:8b-instruct-q4_K_M (~5 GB) — fallback
  4. qwen2.5:14b-instruct-q4_K_M (~9 GB) — long-context, optional
  5. bge-m3 (~1.2 GB) — multilingual embed, optional

Disk needed: ~20 GB for all five. CPU works; GPU is faster (uncomment the deploy.resources block in the compose file).

Step 2 — Reindex the corpora

Go to /super-admin/llm-corpora. Each row is pending and shows the source collection. Optionally set a Reindex limit (e.g. 100) in the toolbar to dry-run first.

Click the play icon (▶) on a corpus row. Status: pending → indexing → ready. The “Chunks” column updates when done. Errors surface in the status chip’s tooltip.

If a Mongo collection is empty, the corpus stays at 0 chunks — the seed templates name the canonical collections; substitute via the edit dialog if your region uses different names.

Step 3 — Try the playground

Go to /super-admin/llm-playground.

  1. Pick a use case — e.g. master.icd_lookup.
  2. Replace the sample context if your test patient differs.
  3. Type a prompt — for ICD lookup: “chest pain with troponin elevation”.
  4. Click Send.

Expect: response within a few seconds (CPU) or sub-second (GPU), with retrieved ICD entries shown as numbered sources. Token counts + latency display at the top of the response panel.

Step 4 — A/B-test models

Same use case, but pick a different model in the Override model dropdown. Send the same prompt. Compare:

  • response content
  • latency
  • which sources got cited

Use this to validate a model swap before editing the use case permanently.

Step 5 — Edit a use case

Go to /super-admin/llm-use-cases. Click the edit icon (✏) on medication.order_suggest. Tweak:

  • the system prompt (mustache template — use the preview icon 👁 to see it rendered against a sample context)
  • the primary / fallback / embedding model
  • RAG knobs: topK, minScore, mmrLambda
  • generation: temperature, maxTokens

Save. The change is live for the next /api/llm/chat call with that use_case.

5. Call from a feature

From any frontend miniapp:

import { llmChat } from '@services/super-admin/llmPlatform.service';

const res = await llmChat({
  use_case: 'medication.order_suggest',
  messages: [{ role: 'user', content: 'Suggest 3 antihypertensives for this patient.' }],
  context: {
    patient: { name: patient.name, hn: patient.hn, age: patient.age, sex: patient.sex, allergies: patient.allergies },
    encounter: { type: 'OPD', cc: encounter.cc, diagnosis: encounter.diagnosis },
    orders: currentRx,
  },
});

console.log(res.message.content);   // assistant text
console.log(res.sources);            // retrieval cites
console.log(res.usage);              // token + latency

From backend (any other Moleculer service):

const res = await ctx.broker.call('llm.chat', {
  use_case: 'medication.interaction_check',
  messages: [{ role: 'user', content: 'Adding warfarin.' }],
  context: { patient: { allergies: [...] }, orders: [...] },
});

6. Audit

Every /api/llm/* call writes to llm_audit_log. To check recent activity:

SELECT created_at, user_id, use_case_code, action, status, total_tokens, latency_ms
FROM llm_audit_log
ORDER BY created_at DESC LIMIT 50;

PHI redaction is on by default (LLM_REDACTION_ENABLED=true); only redacted strings end up in request_payload / response_summary. The model still sees the raw context.

7. Troubleshooting

Symptom Likely cause Fix
Playground returns 500 Embedding model not available Pull nomic-embed-text in /super-admin/llm-models
Reindex fails with embedding model not available Same Same
Reindex finishes with 0 chunks Source collection is empty or template field names don’t match Open the corpus edit dialog and inspect source.textTemplate against actual document shape
Chat returns empty sources minScore too high, or corpus not reindexed Lower minScore in the use case, or check corpus status
Latency >10s on CPU Expected for 7B-14B models without GPU Enable GPU pass-through in docker-compose, or downsize to a smaller model
403 on /api/admin/llm/* JWT missing super-admin role Check user’s app_metadata.roles includes super_admin

8. Per-region rollout

  1. Run both migrations against the region’s Supabase project (the seed is idempotent).
  2. Optionally swap defaults in the seed file before applying — e.g. for Japan, the primary chat model can be set to qwen2.5:14b (better Japanese) instead of qwen2.5:7b.
  3. Operator opens /super-admin/llm-* and pulls the models for that region.
  4. No frontend changes needed — Vercel rewrites already proxy /api/llm/* to the regional ALB.

9. What’s not in v1

See §13 Out of scope in docs/architecture/llm-platform.md:

  • Streaming SSE responses
  • Tool / function calling end-to-end execution
  • Vision / multi-modal runtime
  • vLLM provider
  • Per-tenant fine-tuning
  • Cost accounting (we log tokens, not dollars)

These are reserved for a v2 design.

10. Quick reference — important paths

Want Path
Architecture docs/architecture/llm-platform.md
Migrations infrastructure/medbase/migrations/20260514{c,d}_llm_platform*.sql
Backend service services/llm/
Public REST services/public-api/src/api/publicapi/modules/llm/
Frontend pages web/src/common/components/super-admin/llm-*/
Frontend service layer web/src/services/super-admin/llmPlatform.service.ts
Routes web/src/routes-integrated.tsx — search for super-admin/llm-
Dashboard tile web/src/common/components/super-admin/SuperAdminDashboard.tsx — search “LLM platform”
Ask Anything