GCP Cost Downgrade Runbook

Cost-downgrade runbook for the GCP asia-southeast1 deployment.

8 min read diagramsUpdated 2026-06-01docs/gcp-cost-downgrade-runbook.md

Created: 2026-05-31 · Billing account: Ever Healthcare Billing Main (011454-FFB485-492B2E) · Operator: plan@ever.healthcare Why: April 2026 invoice = ฿223,806/mo (~$6,831). ~48% of spend was non-production. Strategy = scream-test: shut nonprod down, reopen on request. Live prod (hie-prod) was right-sized, not killed. FX used: 32.7645 THB/USD. GKE cluster management fee: ฿2,359/mo (~$72) per cluster.

⭐ HOW TO REOPEN A PROJECT (read this first)

Everything below was stopped, not deleted. All disks + data are intact (and snapshotted). To bring a project back, start its VMs and scale its GKE pool back up. Pods reschedule, DBs remount — it comes back as it was.

Reopen `his-nonprod` (the engineering-sandbox farm)

# Start the VMs we stopped
for N in dev-engineering-sandbox-db dev-ever-bi dev-ever-demo-medos engineering-sandbox-prod \
         engineering-sandbox-uat-1 ever-supabase jumphost test-engineering-sandbox-db \
         uat-engineering-sandbox-db dev-ever-dify; do
  Z=$(gcloud compute instances list --project=his-nonprod --filter="name=$N" --format="value(zone.basename())")
  [ -n "$Z" ] && echo ">> start $N ($Z)" && gcloud compute instances start "$N" --project=his-nonprod --zone="$Z" --quiet
done
# Bring the GKE cluster back (pool-2 was the only active pool, ~4 nodes of e2-custom-8-8192)
gcloud container clusters resize his-nonprod-apps --node-pool=pool-2 --location=asia-southeast1 --num-nodes=2 --quiet
#   ^ num-nodes is PER ZONE on this regional cluster (≈6 total at 2). If pods stay Pending, bump up; if nodes idle, lower.

If a single service is needed (not the whole farm), just start that one VM — e.g. master data lives on ever-supabase and the *-engineering-sandbox-db disks.

⚠️ If Phase 2 has run (VMs/disks/cluster deleted), instances start won’t work — the VMs no longer exist. Use “Restore a project from snapshots” below instead. Data is still 100% recoverable from the snap-* snapshots.

Reopen `hie-nonprod` (HIE dev mirror)

for N in dev-hie-center-mongodb dev-hie-datacenter-mongodb; do
  Z=$(gcloud compute instances list --project=hie-nonprod --filter="name=$N" --format="value(zone.basename())")
  [ -n "$Z" ] && gcloud compute instances start "$N" --project=hie-nonprod --zone="$Z" --quiet
done
gcloud container clusters resize hie-nonprod-apps --node-pool=default-pool --location=asia-southeast1 --num-nodes=3 --quiet
#   ^ was 8 nodes (e2-medium). Restart at 3 and scale up only if needed — 8 was over-provisioned.

Reopen `devops-prod` (CI/CD tooling) — start only what’s needed

# Pick the one(s) you need; you rarely need all at once:
gcloud compute instances start prod-jenkins   --project=devops-409016 --zone=asia-southeast1-b --quiet   # CI builds
gcloud compute instances start prod-sonarqube --project=devops-409016 --zone=asia-southeast1-b --quiet   # code scan
gcloud compute instances start airbyte        --project=devops-409016 --zone=asia-southeast1-b --quiet   # data sync
gcloud compute instances start prod-uptime    --project=devops-409016 --zone=asia-southeast1-b --quiet   # monitoring
# (verify zones with: gcloud compute instances list --project=devops-409016 --filter="name=NAME" --format="value(zone.basename())")
# GKE (if you need the devops-apps cluster back):
for POOL in pool-01 pool-1-sport; do gcloud container clusters resize devops-apps --node-pool="$POOL" --location=asia-southeast1 --num-nodes=1 --quiet; done

Reopen `telemed-nonprod` `test-build`

Z=$(gcloud compute instances list --project=telemed-nonprod --filter="name=test-build" --format="value(zone.basename())")
gcloud compute instances start test-build --project=telemed-nonprod --zone="$Z" --quiet

Restore a project from snapshots (after Phase 2 deletion)

Once disks/VMs are deleted, reopen = recreate the disk from its snapshot, then recreate the VM on it. Per VM:

# 1. Recreate the disk from its snapshot (machine types are in the inventory appendix below)
gcloud compute disks create <DISK> --project=his-nonprod --zone=asia-southeast1-b \
  --source-snapshot=snap-<DISK>-<YYYYMMDD>
# 2. Recreate the VM on that disk (adjust --machine-type to the original)
gcloud compute instances create <VM> --project=his-nonprod --zone=asia-southeast1-b \
  --machine-type=<e.g. e2-standard-2> --disk=name=<DISK>,boot=yes,auto-delete=no

List available snapshots to restore from:

gcloud compute snapshots list --project=his-nonprod --format="table(name,sourceDisk.basename(),diskSizeGb,creationTimestamp)"

For the GKE cluster, recreate it (or restore from GKE Backup if enabled) — but for a sandbox farm it’s usually faster to redeploy fresh than to rebuild the cluster.

Billing snapshot — April 2026 (per project)

Rank	Project	฿/mo	~USD/mo	%	Tier	Disposition
1	his-nonprod	66,535	$2,031	29.7%	nonprod	🔴 SHUT DOWN (scream test)
2	hie-prod	56,502	$1,724	25.2%	prod (live HIE)	🟢 right-sized (12→6 GKE), keep running
3	devops-prod (`devops-409016`)	27,959	$853	12.5%	ops	🔴 STOPPED ALL (start on demand)
4	hie-nonprod	20,906	$638	9.3%	nonprod	🔴 SHUT DOWN (scream test)
5	telemed-nonprod	19,172	$585	8.6%	nonprod	🟡 test-build stopped; rest pending Vajira-tunnel check
6	edh-prod	10,749	$328	4.8%	prod	⚪ small wins only (stop `demo-meditron`)
7	consumer-hc-prod	8,113	$248	3.6%	prod	⚪ orphan-disk cleanup
8	vr-triage-prod	6,124	$187	2.7%	prod	⚪ stop `dev-vr-triage-apps`
9	management-project (`...-408809`)	5,695	$174	2.5%	ops	⚪ unreviewed
10	consumer-hc-nonprod	1,947	$59	0.9%	nonprod	💀 dead (all VMs already off) — Phase 2 delete
—	his-prod, kinetic-silicon, health-data-platform, etc.	~106	—	—	mixed	mostly empty/negligible

Target: ฿223,806 → roughly ฿110–120k/mo (about half).

What was shut down — detail + data safety

Data safety pattern (applied to every shutdown): snapshot all data disks → stop VMs (instances stop, reversible, disks retained) → scale GKE node pools to 0. Nothing deleted. Disks stay attached to stopped VMs; GKE PersistentVolumes are retained. Reopen = start + resize (see top).

Snapshot naming: snap-<diskname>-. Verify with:

for P in his-nonprod hie-nonprod devops-409016; do
  echo "== $P =="; gcloud compute snapshots list --project=$P --format="table(name,diskSizeGb,status)" 2>/dev/null
done

his-nonprod — engineering-sandbox farm (~฿50k/mo saved, Phase 1)

What it was: dev/test/uat/prod sandboxes — engineering-sandbox-{prod,uat-1}, {dev,test,uat}-engineering-sandbox-db, dev-ever-bi, dev-ever-demo-medos, ever-supabase (self-hosted Supabase — likely the “master data”), dev-ever-dify (LLM app), jumphost + GKE his-nonprod-apps (6 pools defined; only pool-2 active = 4× e2-custom-8-8192).
Confirmed serves nothing operational (demo/dev traffic goes to AWS PH ALB, not here). Only concern was a junior worried about master-data loss → resolved by snapshots.
Action: snapshot all → stop 10 VMs → scale pools 1–5 + spot-pool-1 to 0.
Already-terminated (left off): engineering-sandbox-1, engineering-sandbox-test-1, k6-loadtest (32-vCPU), web-server-20250908-040335 (c2-standard-16).
Phase 2 (after snapshot verify): delete live disks (−~฿5k), delete cluster (−฿2,359 control-plane), purge Artifact Registry (฿5,924/mo of old image versions) → floor ~฿3k/mo.

hie-nonprod — HIE dev mirror (~฿12k/mo saved)

What it was: GKE hie-nonprod-apps default-pool = 8× e2-medium (over-provisioned) + 2 running mongo VMs (dev-hie-center-mongodb, dev-hie-datacenter-mongodb). ~14 other dev-hie-*-mongodb VMs (incl. a 6-node replica set) were already terminated but their 50GB disks bill.
Action: snapshot all → stop the 2 mongos → scale default-pool 8→0.
Phase 2: delete the ~14 dead mongo disks (after snapshot) + cluster.

devops-prod (`devops-409016`) — CI/CD tooling (~฿16k/mo saved)

What it was: GKE devops-apps (pool-1-sport = 1× e2-custom-8-16384, 400GB disk) + airbyte (8-vCPU) + prod-jenkins + prod-sonarqube + prod-uptime. ฿4,506/mo Cloud Logging.
Decision: none need 24/7 → stop all; start on demand.
⚠️ prod-uptime may be the monitor for the live hie-prod HIE. If so, keep it running (or confirm monitoring is done elsewhere) — don’t go blind on a national referral system to save ~฿150/mo.
Note: prod-jenkins is likely legacy — medOS backend now deploys via GitHub Actions, not Jenkins.
Cloud Logging cost largely drops once these VMs/GKE stop emitting logs; trim retention/exclusions later if still high.

telemed-nonprod — telemedicine dev (partial)

Stopped the idle test-build (n1-standard-4).
HELD: full shutdown of 3× e2-medium GKE + dev DBs (dev-telemed-mongodb-main, dev-telemed-postgres-main, dev-mdc-oma-consumer-{mongodb,postgres}) pending one check:
⚠️ instance-forward-ssh-tunnel-vajira is an SSH tunnel to Vajira hospital — OPEN QUESTION: does any live job/feed depend on it? Do not stop it until confirmed. When clear, shut telemed down the same way (skip that one VM).

🚫 DO NOT shut down (live production)

hie-prod — live multi-hospital HIE (9 sites + MOPH national referral). Mid-migration to AWS ap-southeast-7; capacity only becomes disposable site-by-site AFTER each cutover. Already right-sized GKE 12→6 nodes + autoscaling (3–12); the ~10 per-site Mongo VMs + brokers must keep running.
his-prod, edh-prod, consumer-hc-prod, vr-triage-prod — prod; only small/idle-resource cleanup, not shutdown.

Smaller immediate wins (low ฿ but pure waste)

# Release idle reserved IP in dead consumer-hc-nonprod (~฿230/mo)
gcloud compute addresses delete exip-nginx-ingress-gke --project=consumer-hc-nonprod --region=asia-southeast1 --quiet
# Stop demo VM in edh-prod (if not actively demoing the Meditron medical LLM)
gcloud compute instances stop demo-meditron --project=edh-prod --zone=asia-southeast1-b
# Stop the stray dev VM living in vr-triage-PROD
gcloud compute instances stop dev-vr-triage-apps --project=vr-triage-prod --zone=asia-southeast1-b

Orphaned unattached disks to delete after a snapshot: consumer-hc-nonprod (7× 8GB PVCs + terminated mongo/postgres/jumphost disks), consumer-hc-prod (4× 8GB PVCs + 2 terminated DB disks), devops-409016 base-disk-jenkins (100GB), his-nonprod engineering-sandbox-uat-1 (100GB orphan).

Pending / next actions

[ ] Run the his-nonprod / hie-nonprod / devops shutdown commands (if not already) + verify snapshots READY.
[ ] Answer: does anything depend on instance-forward-ssh-tunnel-vajira? → then shut telemed-nonprod.
[ ] Confirm prod-uptime monitoring target before leaving it stopped.
[ ] Phase 2 (commands issued 2026-05-31, snapshot-gated): delete his-nonprod VMs+disks + GKE cluster his-nonprod-apps + purge Artifact Registry repos → ~฿2–3k/mo floor. Gate: run the P2.0 check first; only proceed on ✅. After Phase 2, reopen = restore-from-snapshot (section above), not instances start.
[ ] Apply smaller-win commands above.

Full project list (24 total, billing account 011454-FFB485-492B2E)

consumer-hc-nonprod, consumer-hc-prod, devops-409016 (devops-prod), edh-prod, ever-global-network, halogen-trilogy-298108, health-data-platform-408707, hie-nonprod, hie-prod, his-nonprod, his-nonprod-7b47b, his-prod, kinetic-silicon-408707 (Ever New HIE), management-project-408809, mdc-dev-c29b1, medos-global-app-store, potent-terminal-289019, probable-sunup-408707, telemed-dev-96798, telemed-kalasin-dev, telemed-kalasin-prod, telemed-nonprod, telemed-prod-408808, vr-triage-prod. (Empty/no-compute: mdc-dev, his-nonprod-7b47b, telemed-kalasin-{dev,prod}, telemed-dev, medos-global-app-store, ever-global-network.)