medOS ultra

ELK Stack Exploration

Exploration of Hospital AI, BI, FP&A, and healthcare use cases on the ELK stack.

9 min read diagramsUpdated 2026-04-15docs/architecture/elk-stack-exploration.md

Status: Exploration / Assessment
Date: April 2026
Scope: Evaluate ELK stack applicability for medOS-ultra hospital platform

Executive Summary

The ELK stack (Elasticsearch, Logstash, Kibana) is a strong fit for 5 of 11 evaluated use cases in the medOS-ultra hospital platform. It excels at full-text clinical search, microservice log aggregation, HIPAA audit trail search, real-time operational dashboards, and clinico-genomic research queries. However, it is not the right tool for FP&A core workflows, RCM financial analytics, or infrastructure metrics — those are better served by the existing PostgreSQL/Supabase Gold Layer, the FPAEngine, and Prometheus+Grafana respectively.

The platform was architecturally prepared for this integration: the HealthSearchEngine explicitly states it mirrors ES query DSL, the PipelineEngine already declares 'elasticsearch' as an output target, and aggregation types (terms, date_histogram, percentile) map directly to ES aggregation DSL.

What is the ELK Stack?

ELK is an acronym for three open-source projects deployed together for log management, observability, and full-text search:

  • Elasticsearch — Distributed search and analytics engine built on Apache Lucene. Optimized for millisecond text search across billions of records.
  • Logstash — Server-side data pipeline that ingests, transforms, and routes data from multiple sources into Elasticsearch.
  • Kibana — Frontend visualization layer for dashboards, charts, alerts, and interactive data exploration.
  • Beats — Lightweight data shippers (Filebeat for logs, Metricbeat for metrics, Packetbeat for network).

The stack is often called the Elastic Stack to include Beats. As of late 2024, Elasticsearch is licensed under AGPL, returning to true open-source.

Alternatives in the Ecosystem

Tool Best For vs. ELK
OpenSearch Apache 2.0 alternative to ELK Fork of ES 7.10, drop-in replacement
Grafana Loki Cost-effective log aggregation in K8s Indexes only labels, not full text
ClickHouse Analytical queries over massive datasets Columnar storage, 10-100x cheaper for analytics
Fluent Bit / Vector Lightweight log shipping Replaces Logstash for routing only

Use Case Assessment

1. Hospital AI

Sub-Use Case ELK Fit Rationale
Clinical full-text search Strong Current HealthSearchEngine.fullTextSearch() does JSON.stringify().includes() — linear O(N) scan. ES inverted index provides millisecond search at scale.
ICD-10/SNOMED fuzzy code search Strong ES BM25 scoring + field boosting for contextual code suggestions, complementing the existing Snowstorm terminology server.
Clinico-genomic queries Strong ResearchIntelligenceEngine.clinicoGenomicQuery() iterates O(N) over all patients. A denormalized ES index handles “BRCA1 patients on pembrolizumab with PFS >12mo” in one query.
LLM inference (clinical notes, drug-gene) No fit Computation, not search. Keep OpenAI/Claude/Google AI providers.
AI performance monitoring Supporting Kibana dashboards for real-time coding AI acceptance rates complement the daily gold_coding_ai_effectiveness materialized view.

2. BI Data Research

Sub-Use Case ELK Fit Rationale
Real-time operational dashboards Strong Gold Layer refreshes daily (24-hour blind spot). ES + Kibana provides minute-level updates for bed census, ED wait times, admission counts.
Ad-hoc exploratory analytics Strong Kibana Lens allows non-technical department heads to build charts without SQL. Currently a gap in the system.
Population health analytics Strong Cross-correlate demographics, conditions, and geography with ES aggregations for disease registries and CQM.
Complex multi-table joins (P&L) No fit Fundamentally relational. PostgreSQL/Supabase handles this correctly.
Exact financial compliance reporting No fit ES aggregations approximate high-cardinality fields (HyperLogLog). Gold Layer provides exact numbers required for regulatory compliance.

3. FP&A (Financial Planning & Analysis)

ELK is NOT the primary tool for FP&A. The existing FPAEngine handles chart of accounts, budgets, transactions, forecasting, P&L, runway calculations, and financial KPIs — all fundamentally relational operations with currency conversion and multi-entity joins.

ES plays a supporting role only:

  • Real-time financial transaction anomaly detection (ES ML jobs)
  • Claim-to-payment latency percentile distributions (P50/P90/P99)
  • Financial audit trail search

4. General Healthcare

Sub-Use Case ELK Fit Rationale
Microservice log aggregation Strongest Industry standard for 18-service microservices. pino-elasticsearch already proven in eForm builder.
HIPAA audit trail search Strong Compliance officers need faceted search over immutable audit events (patient, actor, action, date range).
FHIR resource indexing Strong Powers FHIR _search API compliance with proper search parameters.
Clinical quality measures (CQM) Strong Continuous queries against large patient populations with complex filter criteria.
Infrastructure metrics No fit Use Prometheus + Grafana (Moleculer already has Prometheus reporter).

Decision Matrix

Capability Adopt ELK? Priority Alternative
Microservice log aggregation Yes P0 —
FHIR resource full-text search Yes P1 —
HIPAA audit trail search Yes P1 —
Real-time operational dashboards Yes (Kibana) P2 —
Clinico-genomic research queries Yes P2 —
AI performance monitoring Yes (Kibana) P3 —
RCM financial analytics No — Keep Supabase Gold Layer
FP&A core workflows No — Keep PostgreSQL/FPAEngine
Infrastructure metrics No — Prometheus + Grafana
High-volume debug logs No — Grafana Loki
Multi-year cohort analytics No — ClickHouse

Platform Readiness

Already in Place (High Readiness)

  1. HealthSearchEngine — Comments explicitly say: “In production, this would be backed by Elasticsearch.” Aggregation types mirror ES DSL exactly.
  2. PipelineEngine OutputTarget — Already declares 'elasticsearch' as output type. Now implements real bulk indexing via the ES client.
  3. AggregationDef types — count, sum, avg, min, max, terms, date_histogram, percentile map directly to ES aggregations.
  4. Pino-Elasticsearch — Already configured in eForm builder for structured logging.
  5. Elasticsearch 7.17 — Already deployed for Snowstorm (SNOMED CT terminology server).
  6. SidecarService — Now supports dual-write to ES after FHIR transformation.

What Was Added in This Exploration

  1. docker/elk/docker-compose.elk.yml — ES 8.x + Kibana + Logstash deployment (separate from Snowstorm ES 7.17)
  2. packages/health-data-platform/src/elasticsearch/ — Full ES integration module:
    • elasticsearch-client.ts — Client abstraction with production and mock implementations
    • index-management.ts — Index templates (FHIR, audit, logs) and ILM policies
    • datalake-adapter.ts — Bridges DataLakeStore interface to ES with full-text search, aggregations, and HIPAA audit operations
  3. PipelineEngine — Real ES bulk indexing output (replaced no-op stub)
  4. SidecarService — Elasticsearch dual-write for FHIR resource ingestion
  5. Logstash pipelines — Service logs and HIPAA audit event processing

Architecture Integration

                          ┌──────────────────────────────────┐
                          │         medOS Services           │
                          │  (18 Moleculer Microservices)    │
                          └──────┬────────────┬──────────────┘
                                 │            │
                    Pino-ES logs │            │ FHIR resources
                                 │            │
                          ┌──────▼──────┐  ┌──▼──────────────┐
                          │  Logstash   │  │  SidecarService  │
                          │ (Pipeline)  │  │ (Dual-Write)     │
                          └──────┬──────┘  └──┬──────────┬───┘
                                 │            │          │
                                 │            │          │ (source of truth)
                          ┌──────▼────────────▼──┐   ┌──▼──────────────┐
                          │    Elasticsearch      │   │   MongoDB /     │
                          │    (Search Replica)    │   │   Supabase      │
                          │                        │   │   (Write Model) │
                          │  medos-logs-*          │   └─────────────────┘
                          │  medos-fhir-{tenant}   │
                          │  medos-audit-{year}    │
                          └──────────┬─────────────┘
                                     │
                          ┌──────────▼─────────────┐
                          │       Kibana            │
                          │  ┌─────────────────┐    │
                          │  │ Ops Dashboards   │   │
                          │  │ Clinical Search  │   │
                          │  │ Compliance Space │   │
                          │  │ Research Explorer│   │
                          │  └─────────────────┘    │
                          └─────────────────────────┘

Key principle: Elasticsearch is a search replica, never the source of truth. All indices can be rebuilt from MongoDB (write model) and Supabase (read model) at any time.

HIPAA Compliance

PHI in Search Indices

  • Field-level security for PHI fields (name, SSN, DOB, address) via ES Security
  • anonymize processor step in PipelineEngine creates de-identified research indices
  • ES audit logging enabled for meta-audit (“who searched for what”)
  • Per-tenant indices for data isolation

Data Retention (ILM Policies)

Data Category Retention ILM Strategy
PHI audit logs 7 years Hot 1yr → Warm 5yr → Frozen 1yr → Delete
Clinical FHIR resources 10 years Hot 2yr → Warm 5yr → Cold 3yr → Delete
Operational logs (no PHI) 90 days Hot 7d → Warm 30d → Delete 90d
Research indices (anonymized) Indefinite Warm tier, no auto-delete

Access Control (RBAC)

medOS Role ES Access
admin / compliance_officer Full read on audit indices
clinician Read on patient FHIR indices (filtered by department)
coder Read on coding worklist indices
researcher Read on anonymized research indices only
service_role Write access for ingestion pipelines

Phased Rollout

Phase Weeks Scope
0: Foundation 1-3 Deploy ES 8.x + Kibana cluster, add ES client to platform
1: Operational Logging 3-6 Centralize 18 service logs, build error rate dashboards
2: FHIR Resource Search 6-12 Replace in-memory search with ES, enable clinical full-text search
3: HIPAA Audit 10-14 Mirror audit events to ES, build compliance dashboards + alerts
4: Clinical Intelligence 14-20 Clinico-genomic queries, trend detection, anomaly ML jobs
5: Real-time Analytics 18-24 NATS-to-Logstash bridge, operational dashboards, alerting

Files Modified/Created

File Status Purpose
docker/elk/docker-compose.elk.yml New ES 8.x + Kibana + Logstash deployment
docker/elk/config/logstash.yml New Logstash configuration
docker/elk/pipeline/medos-services.conf New Service log ingestion pipeline
docker/elk/pipeline/medos-audit.conf New HIPAA audit event pipeline
packages/health-data-platform/src/elasticsearch/elasticsearch-client.ts New ES client abstraction (prod + mock)
packages/health-data-platform/src/elasticsearch/index-management.ts New Index templates, ILM policies, IndexManager
packages/health-data-platform/src/elasticsearch/datalake-adapter.ts New DataLakeStore → ES adapter with HIPAA audit
packages/health-data-platform/src/elasticsearch/index.ts New Barrel exports
packages/health-data-platform/src/models/platform-types.ts Modified Added elasticsearch config to OutputTarget
packages/health-data-platform/src/pipeline/pipeline-engine.ts Modified Real ES bulk indexing output
packages/health-data-platform/src/sidecar/sidecar-service.ts Modified ES dual-write on ingestion
packages/health-data-platform/src/index.ts Modified Export elasticsearch module
Ask Anything