medOS ultra

LLM Platform

Self-hosted, RAG-enabled, per-use-case model selection.

18 min read diagramsUpdated 2026-05-14docs/architecture/llm-platform.md

Status: design (2026-05-14) → 20-loop implementation in progress Owner: super-admin (operator config); platform/infra (runtime) Scope: new services/llm Moleculer service + Ollama runtime + Supabase pgvector RAG + 4 super-admin pages + public REST API

1. Why

medOS-ultra needs a first-party LLM platform so the same model + prompt + retrieval setup works in every region (Thailand, Japan, Philippines, on-premise), without leaking PHI to third-party APIs and without per-feature ad-hoc OpenAI calls.

Constraints driving this design:

  1. PHI must not leave the hospital. On-premise deployments often have no outbound internet. The platform must run fully air-gapped.
  2. Per-region locale. Japan needs ja, Philippines needs fil/en, Thailand needs th. The same use case must resolve to a model that speaks the right language.
  3. Per-use-case model. A drug-interaction check needs a small/fast model with a curated drug corpus. A discharge-summary draft needs a bigger long-context model. Clinical-code lookup needs only RAG, no generation.
  4. Super-admin is the operator. Adding a new model, changing the system prompt for “lab panel suggest”, or rebuilding the ICD-10 corpus — all of these are config edits in the dashboard, not code deploys.
  5. Audit + rate-limit. Every call goes through a single audited gateway. No miniapp talks to Ollama directly.

2. Non-goals

  • Training or fine-tuning models in the cluster (use external ops).
  • Replacing the existing OpenAI-backed services/ai/ clinical-note flow on day one — that path keeps working; new use cases default to the self-hosted gateway.
  • Multi-modal (vision/audio) in v1 — schema reserves capability flags, but runtime only ships text.
  • Streaming SSE in v1 — /chat returns the full message. Streaming added in a later loop if the playground needs it.

3. Architecture

┌──────────────────────────────────────────────────────────────────────────┐
│                  Super-Admin Dashboard (web/)                             │
│  /admin/super-admin/llm-models       /admin/super-admin/llm-use-cases    │
│  /admin/super-admin/llm-corpora       /admin/super-admin/llm-playground  │
└────────────────────────┬─────────────────────────────────────────────────┘
                         │ Supabase (direct, RLS-protected) for config CRUD
                         │ REST (gateway → public-api) for chat/search
                         ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  services/gateway  (existing)  →  services/public-api  (existing)        │
│  exposes /api/llm/* and forwards to Moleculer action `llm.*`             │
└────────────────────────┬─────────────────────────────────────────────────┘
                         │ NATS (Moleculer)
                         ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                  services/llm   (NEW, Moleculer-native)                   │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│  │ ModelRegistry│ │UseCaseRegistry│ │CorpusRegistry│ │  ChatOrchestrator │ │
│  │  Mongo +     │ │  Mongo +     │ │  Mongo +     │ │  retrieve→rerank │ │
│  │  Ollama pull │ │  Supabase    │ │  ingestion   │ │  →complete →log  │ │
│  └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────┘ │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│  │OllamaProvider│ │ EmbeddingSvc │ │ Retriever    │ │  AuditLogger     │ │
│  │  HTTP client │ │  → pgvector  │ │  cosine + MMR│ │  reuses aaa.     │ │
│  │              │ │              │ │              │ │  auditLog        │ │
│  └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────┘ │
└──────┬────────────────────────┬─────────────────────────┬────────────────┘
       │ HTTP                   │ pg (service_role)       │ Moleculer events
       ▼                        ▼                          ▼
┌──────────────────┐ ┌────────────────────────┐ ┌─────────────────────────┐
│  Ollama          │ │ Supabase (pgvector)    │ │  MongoDB (medos-db)     │
│  :11434          │ │  llm_embeddings,       │ │  llm_models,            │
│  models on disk  │ │  llm_corpora,          │ │  llm_use_cases,         │
│  pulled on demand│ │  llm_conversations,    │ │  llm_corpora (mirror),  │
│                  │ │  llm_audit_log         │ │  audit_logs (shared)    │
└──────────────────┘ └────────────────────────┘ └─────────────────────────┘

Why Ollama (not vLLM, llama.cpp, OpenAI-compat shim)

Option Pros Cons Decision
Ollama Single container, OpenAI-compatible API at /v1/chat/completions, ollama pull from registry, runs CPU or GPU, native quantization (Q4/Q8), Modelfile for custom prompts Lower throughput than vLLM at scale (acceptable for hospital-scale traffic) Chosen
vLLM Best GPU throughput GPU-required, heavier ops, no easy model registry Reject for v1
llama.cpp (server mode) Tiny binary, CPU-friendly One model per process, manual model management Reject
LM Studio / GPT4All UX-friendly Desktop apps, not server-grade Reject
External (OpenAI/Anthropic/etc) Best quality Sends PHI off-site, breaks air-gap Already covered by existing services/ai/; this platform is the on-prem alternative

Ollama can be swapped at the provider-layer if a region wants vLLM — llm_models.provider is an enum that the dispatcher reads. v1 only implements ollama; the seam is there.

4. Data model

4.1 MongoDB (write source-of-truth for config — collection per entity)

Schemas live in packages/platform-api-schema/src/llm/.

// LlmModel — what models we know about
{
  _id: ObjectId,
  provider: 'ollama' | 'vllm' | 'openai-compat',
  modelName: string,                  // 'llama3.1:8b', 'qwen2.5:14b', 'nomic-embed-text'
  displayName: string,
  displayName_th?: string,
  displayName_ja?: string,
  capabilities: {
    chat: boolean,
    completion: boolean,
    embedding: boolean,
    vision: boolean,
    functionCalling: boolean,
  },
  contextWindow: number,              // 8192, 32768, 131072
  parameterCount?: number,
  status: 'registered' | 'pulling' | 'available' | 'error',
  sizeBytes?: number,
  endpointUrl: string,                // 'http://ollama:11434'
  defaultTemperature: number,         // 0.3
  active: boolean,
  createdAt, updatedAt,
}

// LlmUseCase — mapping of feature/domain → model + system prompt + RAG
{
  _id: ObjectId,
  code: string,                       // 'medication.order_suggest', 'lab.panel_suggest',
                                      // 'master.icd_lookup', 'clinical.discharge_summary'
  displayName, displayName_th, displayName_ja,
  description?: string,
  primaryModel: ObjectId,             // → LlmModel
  fallbackModel?: ObjectId,
  embeddingModel?: ObjectId,          // for RAG; usually 'nomic-embed-text'
  systemPrompt: string,               // mustache-style {{patient.age}} {{encounter.id}}
  rag: {
    corpusIds: ObjectId[],            // which corpora to query
    topK: number,                     // default 5
    minScore: number,                 // default 0.4 (cosine sim)
    mmrLambda: number,                // default 0.5 (diversity vs relevance)
  },
  generation: {
    temperature: number,
    maxTokens: number,
    topP?: number,
    stopSequences?: string[],
  },
  tools: any[],                       // OpenAI-format function defs (passthrough)
  rateLimitPerMin: number,            // 30
  active: boolean,
  createdAt, updatedAt,
}

// LlmCorpus — RAG corpus definitions
{
  _id: ObjectId,
  code: string,                       // 'icd10', 'drug-db', 'pathology-catalog',
                                      // 'hospital-policies', 'clinical-pathways'
  displayName, displayName_th, displayName_ja,
  source: {
    type: 'mongo-collection' | 'supabase-table' | 'static-jsonl' | 'manual',
    collection?: string,              // e.g. 'icd10_codes' (Mongo) or 'master_drug' (Supabase)
    query?: string,                   // optional filter
    textTemplate: string,             // mustache: '{{code}} — {{description}} ({{description_th}})'
    metadataKeys: string[],           // which fields to copy into embedding metadata
    idField: string,                  // primary key in source
  },
  chunking: {
    strategy: 'whole' | 'fixed' | 'sentence',
    chunkSize: number,                // 512 chars for 'fixed'
    chunkOverlap: number,             // 64
  },
  embeddingModel: ObjectId,
  totalChunks: number,
  lastIndexedAt?: Date,
  status: 'pending' | 'indexing' | 'ready' | 'error',
  errorMessage?: string,
  active: boolean,
  createdAt, updatedAt,
}

4.2 Supabase (read model + vector store)

Migration: infrastructure/medbase/migrations/045_llm_platform.sql

CREATE EXTENSION IF NOT EXISTS vector;

-- 1. Config mirrors (lightweight read model so the UI doesn't hit Mongo)
CREATE TABLE public.llm_models (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  mongo_id TEXT UNIQUE,                       -- ObjectId hex of LlmModel
  provider TEXT NOT NULL,
  model_name TEXT NOT NULL,
  display_name TEXT NOT NULL,
  display_name_th TEXT,
  display_name_ja TEXT,
  capabilities JSONB NOT NULL DEFAULT '{}',
  context_window INT NOT NULL DEFAULT 4096,
  parameter_count BIGINT,
  status TEXT NOT NULL DEFAULT 'registered',
  size_bytes BIGINT,
  endpoint_url TEXT NOT NULL,
  default_temperature NUMERIC NOT NULL DEFAULT 0.3,
  active BOOLEAN NOT NULL DEFAULT true,
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE public.llm_use_cases (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  mongo_id TEXT UNIQUE,
  code TEXT UNIQUE NOT NULL,
  display_name TEXT NOT NULL,
  display_name_th TEXT,
  display_name_ja TEXT,
  description TEXT,
  primary_model_id UUID REFERENCES public.llm_models(id),
  fallback_model_id UUID REFERENCES public.llm_models(id),
  embedding_model_id UUID REFERENCES public.llm_models(id),
  system_prompt TEXT NOT NULL,
  rag_config JSONB NOT NULL DEFAULT '{}',     -- {corpusIds, topK, minScore, mmrLambda}
  generation_config JSONB NOT NULL DEFAULT '{}',
  tools JSONB NOT NULL DEFAULT '[]',
  rate_limit_per_min INT NOT NULL DEFAULT 30,
  active BOOLEAN NOT NULL DEFAULT true,
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE public.llm_corpora (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  mongo_id TEXT UNIQUE,
  code TEXT UNIQUE NOT NULL,
  display_name TEXT NOT NULL,
  display_name_th TEXT,
  display_name_ja TEXT,
  source JSONB NOT NULL,                      -- {type, collection, query, textTemplate, ...}
  chunking JSONB NOT NULL DEFAULT '{}',
  embedding_model_id UUID REFERENCES public.llm_models(id),
  total_chunks INT NOT NULL DEFAULT 0,
  last_indexed_at TIMESTAMPTZ,
  status TEXT NOT NULL DEFAULT 'pending',
  error_message TEXT,
  active BOOLEAN NOT NULL DEFAULT true,
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
);

-- 2. Vector store (the actual RAG payload)
CREATE TABLE public.llm_embeddings (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  corpus_id UUID NOT NULL REFERENCES public.llm_corpora(id) ON DELETE CASCADE,
  source_id TEXT NOT NULL,                    -- the source row's primary key
  chunk_index INT NOT NULL DEFAULT 0,
  content TEXT NOT NULL,
  content_summary TEXT,
  metadata JSONB NOT NULL DEFAULT '{}',
  embedding vector(768),                      -- nomic-embed-text default; 1024 for bge-large
  created_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX llm_embeddings_hnsw ON public.llm_embeddings
  USING hnsw (embedding vector_cosine_ops);
CREATE INDEX llm_embeddings_corpus_idx ON public.llm_embeddings (corpus_id);
CREATE INDEX llm_embeddings_source_idx ON public.llm_embeddings (corpus_id, source_id);

-- 3. Conversations (multi-turn chat use cases)
CREATE TABLE public.llm_conversations (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  use_case_code TEXT NOT NULL,
  user_id TEXT NOT NULL,
  patient_id TEXT,
  encounter_id TEXT,
  context JSONB NOT NULL DEFAULT '{}',
  title TEXT,
  status TEXT NOT NULL DEFAULT 'active',      -- active | archived
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE public.llm_messages (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  conversation_id UUID NOT NULL REFERENCES public.llm_conversations(id) ON DELETE CASCADE,
  role TEXT NOT NULL,                         -- system | user | assistant | tool
  content TEXT NOT NULL,
  tool_calls JSONB,
  tool_call_id TEXT,
  model_id UUID REFERENCES public.llm_models(id),
  prompt_tokens INT,
  completion_tokens INT,
  total_tokens INT,
  retrieval_sources JSONB,                    -- [{corpus_code, source_id, score, snippet}]
  latency_ms INT,
  created_at TIMESTAMPTZ DEFAULT now()
);

-- 4. Audit log (every inference call, every config change)
CREATE TABLE public.llm_audit_log (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id TEXT NOT NULL,
  user_display TEXT,
  use_case_code TEXT,
  model_id UUID,
  action TEXT NOT NULL,                       -- chat | complete | search | embed
                                              -- | model.pull | model.delete
                                              -- | use_case.upsert | corpus.reindex
  request_payload JSONB,
  response_summary TEXT,
  prompt_tokens INT,
  completion_tokens INT,
  total_tokens INT,
  latency_ms INT,
  status TEXT NOT NULL DEFAULT 'ok',          -- ok | error | rate_limited | blocked
  error_message TEXT,
  remote_address TEXT,
  created_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX llm_audit_user_idx ON public.llm_audit_log (user_id, created_at DESC);
CREATE INDEX llm_audit_use_case_idx ON public.llm_audit_log (use_case_code, created_at DESC);

-- 5. RLS — super-admin only for config + audit; user-scoped for conversations
ALTER TABLE public.llm_models      ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.llm_use_cases   ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.llm_corpora     ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.llm_embeddings  ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.llm_conversations ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.llm_messages    ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.llm_audit_log   ENABLE ROW LEVEL SECURITY;

CREATE OR REPLACE FUNCTION public.llm_is_super_admin() RETURNS boolean AS $$
  SELECT coalesce(
    (auth.jwt() -> 'app_metadata' -> 'roles' ? 'super_admin')
    OR (auth.jwt() ->> 'role' = 'super_admin')
    OR (auth.jwt() ->> 'role' = 'service_role'),
    false
  );
$$ LANGUAGE sql STABLE;

CREATE POLICY llm_models_super_admin       ON public.llm_models       FOR ALL USING (public.llm_is_super_admin());
CREATE POLICY llm_use_cases_super_admin    ON public.llm_use_cases    FOR ALL USING (public.llm_is_super_admin());
CREATE POLICY llm_corpora_super_admin      ON public.llm_corpora      FOR ALL USING (public.llm_is_super_admin());
CREATE POLICY llm_embeddings_super_admin   ON public.llm_embeddings   FOR ALL USING (public.llm_is_super_admin());
CREATE POLICY llm_audit_super_admin_select ON public.llm_audit_log    FOR SELECT USING (public.llm_is_super_admin());

-- Conversations: super-admin OR owner
CREATE POLICY llm_conv_owner ON public.llm_conversations
  FOR ALL USING (
    public.llm_is_super_admin()
    OR user_id = coalesce(auth.jwt() ->> 'sub', auth.jwt() ->> 'user_id')
  );
CREATE POLICY llm_msg_owner ON public.llm_messages
  FOR ALL USING (
    EXISTS (
      SELECT 1 FROM public.llm_conversations c
      WHERE c.id = llm_messages.conversation_id
        AND (public.llm_is_super_admin()
             OR c.user_id = coalesce(auth.jwt() ->> 'sub', auth.jwt() ->> 'user_id'))
    )
  );

-- 6. RPC helpers (called from the service via service_role key)
CREATE OR REPLACE FUNCTION public.llm_search_embeddings(
  p_corpus_ids UUID[],
  p_query_embedding vector(768),
  p_top_k INT DEFAULT 5,
  p_min_score NUMERIC DEFAULT 0.4
) RETURNS TABLE (
  id UUID,
  corpus_id UUID,
  source_id TEXT,
  content TEXT,
  metadata JSONB,
  score NUMERIC
) LANGUAGE sql STABLE AS $$
  SELECT
    e.id,
    e.corpus_id,
    e.source_id,
    e.content,
    e.metadata,
    1 - (e.embedding <=> p_query_embedding) AS score
  FROM public.llm_embeddings e
  WHERE e.corpus_id = ANY(p_corpus_ids)
    AND 1 - (e.embedding <=> p_query_embedding) >= p_min_score
  ORDER BY e.embedding <=> p_query_embedding
  LIMIT p_top_k;
$$;

4.3 Why two stores?

  • Mongo is the write source-of-truth for config — matches the rest of medOS, lets services/llm boot from Mongo without a Supabase dependency, and keeps the audit pattern uniform with aaa.auditLog.
  • Supabase is the read model + vector store — the super-admin UI talks to it directly (matches the cron-jobs / dispense-cycles pattern), realtime subscriptions work out of the box, and pgvector is already a dependency.

services/llm writes to Mongo, then projects the row into Supabase via service_role. Same pattern as the existing read-model edge functions; just shorter because both stores live next to each other.

5. API surface

5.1 REST (via services/public-api — exposed through gateway at /api/llm/*)

All routes are JWT-authenticated. Super-admin-only routes are tagged.

# Inference
POST /api/llm/chat
  body: {
    use_case: string,                       // 'medication.order_suggest'
    messages: [{role, content}],            // prior turns; first user msg required
    context?: {                             // template substitutions
      patient?: any,
      encounter?: any,
      orders?: any[],
      ...
    },
    conversation_id?: string,               // continue existing thread
    overrides?: {                           // optional admin overrides (super-admin only)
      model_id?: string,
      system_prompt?: string,
      temperature?: number,
    },
  }
  returns: {
    conversation_id: string,
    message: { role: 'assistant', content: string, tool_calls?: any[] },
    sources: [{corpus, source_id, snippet, score}],
    usage: {prompt_tokens, completion_tokens, total_tokens, latency_ms},
    model: {id, display_name, provider},
  }

POST /api/llm/complete
  body: { use_case, prompt, context? }
  returns: { text, sources, usage, model }

POST /api/llm/search
  body: { corpus_codes: string[], query: string, top_k?: number }
  returns: { results: [{corpus, source_id, content, metadata, score}] }

POST /api/llm/embed
  body: { texts: string[], model_code?: string }
  returns: { embeddings: number[][], model, usage }

GET  /api/llm/models                        # list active models user can choose from
GET  /api/llm/use-cases                     # list use cases the user has access to

# Pre-baked agents (thin wrappers over /chat with a fixed use_case)
POST /api/llm/agents/medication-suggest
POST /api/llm/agents/lab-panel-suggest
POST /api/llm/agents/master-data-lookup
POST /api/llm/agents/clinical-summary

# Super-admin
GET    /api/admin/llm/models
POST   /api/admin/llm/models                # register
POST   /api/admin/llm/models/:id/pull       # trigger ollama pull
DELETE /api/admin/llm/models/:id

GET    /api/admin/llm/use-cases
POST   /api/admin/llm/use-cases
PATCH  /api/admin/llm/use-cases/:id
DELETE /api/admin/llm/use-cases/:id

GET    /api/admin/llm/corpora
POST   /api/admin/llm/corpora
PATCH  /api/admin/llm/corpora/:id
POST   /api/admin/llm/corpora/:id/reindex   # rebuild embeddings
DELETE /api/admin/llm/corpora/:id

GET    /api/admin/llm/audit                 # paginated audit log

5.2 Moleculer actions (internal — other services call these)

llm.chat                  { useCase, messages, context }     → {message, sources, usage}
llm.complete              { useCase, prompt, context }       → {text, sources, usage}
llm.search                { corpusCodes, query, topK }       → {results}
llm.embed                 { texts, modelCode }               → {embeddings}
llm.models.list           {}                                 → [...]
llm.models.pull           { id }                             → {status}
llm.useCases.list         {}                                 → [...]
llm.useCases.upsert       { payload }                        → {...}
llm.corpora.list          {}                                 → [...]
llm.corpora.reindex       { id }                             → {jobId}

6. RAG pipeline

1. Ingestion (background; runs on corpus.upsert and corpus.reindex)
   ┌─────────────────────────────────────────────────────────────────┐
   │ a. Read source rows (Mongo collection / Supabase table / JSONL)  │
   │ b. For each row:                                                 │
   │      text = render(textTemplate, row)        // mustache         │
   │      chunks = chunk(text, chunking.strategy)                     │
   │      for chunk in chunks:                                        │
   │        vec = embed(chunk, embeddingModel)                        │
   │        upsert llm_embeddings (corpus_id, source_id, chunk_index, │
   │           content, metadata, embedding=vec)                      │
   │ c. Set corpus.totalChunks, lastIndexedAt, status='ready'         │
   └─────────────────────────────────────────────────────────────────┘

2. Retrieval (per /chat or /search call)
   ┌─────────────────────────────────────────────────────────────────┐
   │ a. Build query string: last user msg + relevant context fields   │
   │ b. embed(query, useCase.embeddingModel)                          │
   │ c. RPC llm_search_embeddings(corpus_ids, vec, topK*2, minScore)  │
   │ d. MMR rerank with mmrLambda → topK final results                │
   │ e. Build prompt: system + RAG-context-block + user messages      │
   └─────────────────────────────────────────────────────────────────┘

3. Generation
   ┌─────────────────────────────────────────────────────────────────┐
   │ a. POST ollama:11434/v1/chat/completions                         │
   │    {model, messages, temperature, max_tokens, tools}             │
   │ b. Parse response, persist conversation + message rows           │
   │ c. Log audit row with usage + latency                            │
   │ d. Return response with retrieval_sources                        │
   └─────────────────────────────────────────────────────────────────┘

MMR (Maximal Marginal Relevance) reranking is implemented in TS — small enough to keep in-process. The formula: pick the doc that maximizes λ·sim(d, q) - (1-λ)·max sim(d, picked).

7. Default seed (loop 19)

Default registry shipped with the platform — operators can edit/delete in the UI.

Models (llm_models)

code provider size purpose
llama3.1:8b-instruct-q4_K_M ollama ~5 GB general chat, summary, fallback
qwen2.5:14b-instruct-q4_K_M ollama ~9 GB longer-context clinical drafts
qwen2.5:7b-instruct-q4_K_M ollama ~5 GB fast, multilingual (good at ja/th/fil)
nomic-embed-text ollama ~270 MB embedding (768 dim)
bge-m3 ollama ~1.2 GB multilingual embedding (1024 dim, optional)

Use cases (llm_use_cases)

code model corpus purpose
medication.order_suggest qwen2.5:7b drug-db propose Rx given diagnosis + patient ctx
medication.interaction_check llama3.1:8b drug-db flag DDI / contraindication
lab.panel_suggest qwen2.5:7b pathology-catalog suggest lab panel from CC/Dx
lab.result_interpret llama3.1:8b pathology-catalog summarize abnormal result
radiology.protocol_suggest qwen2.5:7b radiology-catalog propose imaging protocol
master.icd_lookup qwen2.5:7b icd10 RAG-only ICD-10/9 lookup
master.drug_lookup qwen2.5:7b drug-db drug ingredient/strength/route lookup
clinical.encounter_summary qwen2.5:14b clinical-policies discharge / encounter summary draft
clinical.handover_note qwen2.5:14b clinical-policies shift-handover summary
billing.code_assist llama3.1:8b icd10 + insurance-rules suggest billing codes from notes
nutrition.menu_advise qwen2.5:7b nutrition-catalog tailor menu to constraints
policy.gate_explain qwen2.5:7b hospital-policies explain why a policy gate fires

Corpora (llm_corpora)

code source language
icd10 mongo icd10_codes (template uses code, description, description_th, description_ja) multilingual
drug-db mongo master_drug multilingual
pathology-catalog mongo pathology_test multilingual
radiology-catalog mongo radiology_procedure multilingual
nutrition-catalog supabase nutrition_menu_items per-region
clinical-policies manual (admin uploads JSONL) per-tenant
hospital-policies supabase policy_gates (rendered) per-tenant
insurance-rules mongo insurance_rate + kaigo_rate + philhealth_rate per-region

All seed data follows CLAUDE.md rule #7: bilingual (local language + English).

8. Super-admin UI (loops 13–18)

Routes registered in web/src/routes/AdminRoutes.tsx. All match the existing super-admin pattern: container wrapper with super-admin guard → page component using raw MUI + Supabase direct.

Route Files Purpose
/admin/super-admin/llm-models containers/super-admin-llm-models/page.tsx + common/components/super-admin/llm-models/LlmModelsPage.tsx + services/super-admin/llmModels.service.ts Register/pull/delete models, see status (registered/pulling/available/error), pull progress via realtime
/admin/super-admin/llm-use-cases parallel layout Mapping of code → model + system prompt + RAG; system-prompt editor with mustache preview against a sample context
/admin/super-admin/llm-corpora parallel layout Corpus list, reindex button (shows progress), source picker, text-template editor
/admin/super-admin/llm-playground parallel layout Pick a use-case, paste sample context, get response + see retrieved sources + token usage + latency; admin overrides for model/temperature

Tile added to SuperAdminDashboard in loop 18.

9. Deployment

9.1 Docker compose changes (loop 12)

infrastructure/docker-compose.yml (dev) — add ollama service + api-llm service.

infrastructure/docker-compose-onpremise.yml (on-prem all-in-one) — same, plus volume for model cache.

# infrastructure/docker-compose-onpremise.yml (excerpt)
services:
  ollama:
    image: ollama/ollama:latest
    restart: always
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=2
      - OLLAMA_MAX_LOADED_MODELS=2
    volumes:
      - ollama-data:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  api-llm:
    build:
      context: ..
      dockerfile: ./docker/api/Dockerfile
      args:
        SCOPE: ever-api-llm
    image: medos-api
    restart: always
    environment:
      - TZ=${TZ:-Asia/Bangkok}
      - SCOPE=ever-api-llm
      - TRANSPORTER=nats://nats:4222
      - MONGO_URI=mongodb://mongo/medos-db?replicaSet=rs0
      - OLLAMA_URL=http://ollama:11434
      - SUPABASE_URL=${SUPABASE_URL}
      - SUPABASE_SERVICE_ROLE_KEY=${SUPABASE_SERVICE_ROLE_KEY}
    depends_on:
      mongo: { condition: service_healthy }
      nats:  { condition: service_healthy }
      ollama: { condition: service_started }

volumes:
  ollama-data:

9.2 Per-region defaults

Region Default chat model Default embed model Locale field in templates
Thailand qwen2.5:7b-instruct bge-m3 description_th
Japan qwen2.5:14b-instruct bge-m3 description_ja
Philippines llama3.1:8b-instruct nomic-embed-text description (English-first)

Selected at seed time; super-admin can change later.

9.3 GPU vs CPU

Ollama runs CPU-only by default. To enable GPU (NVIDIA), add to ollama service:

deploy:
  resources:
    reservations:
      devices: [{ driver: nvidia, count: 1, capabilities: [gpu] }]

This is documented in docs/llm-platform-deployment.md (created in loop 20).

10. Security & RBAC

Layer Mechanism
Config CRUD (UI) Container guard + Supabase RLS (llm_is_super_admin())
/api/llm/chat, /search, /embed JWT required; rate limited per useCase.rateLimitPerMin
/api/admin/llm/* JWT + super-admin role check in public-api controller
PHI in context Pass-through to local Ollama only; never sent to external API; redaction hook in chat orchestrator (regex for ID numbers, configurable per region)
Audit Every inference logs to llm_audit_log; super-admin can review
Tenant isolation Use-case code is global; corpora can be scoped by adding tenant_id in metadata in v2 if needed

11. Multi-region considerations

  • Each region’s Supabase project gets the same migration (045) via cross-region-policy-gates-deployment.md pattern.
  • Seed data lives in infrastructure/market-packs/{region}/seed-llm-defaults.sql per region (loop 19).
  • Vercel projects don’t need changes — /api/llm/* proxies via existing vercel.json rewrites to the same ALB.
  • On-prem deployments get Ollama on the same Docker network; models are pulled once and cached in the ollama-data volume.

12. Phased rollout (loop ordering)

Loop Deliverable
1 This design doc
2 Supabase migration 045_llm_platform.sql
3 packages/platform-api-schema/src/llm/ entities
4 services/llm/ skeleton
5 Ollama HTTP provider
6 Model registry actions
7 Use-case registry actions
8 Corpus registry + chunker
9 Embedding service + retriever
10 Chat orchestrator
11 Public-API route surface
12 Docker compose
13 Frontend service layer
14 Models admin page
15 Use-cases admin page
16 Corpora admin page
17 Playground
18 Route reg + dashboard tile
19 Default seed (bilingual, per region)
20 UAT doc + verification report

13. Out of scope (post-20-loop)

  • Streaming SSE responses
  • Tool / function calling end-to-end (schema is reserved; orchestrator passes through but no built-in tool dispatcher)
  • Image-input (vision) capability flag exists; runtime path skipped
  • Auto-train / DPO loops
  • Per-tenant fine-tuning
  • vLLM provider (Ollama-only in v1)
  • Cost accounting in currency units (token counts only)

These belong in a v2 design once the v1 platform is in operator hands.

Ask Anything