Voice Attestation for AI Sign-Off

Status: capability shipped + sandbox-verified (2026-06-09); live AI-order wiring is a documented, opt-in next step. Flag-off by default — nothing changes until a model + flag are configured.

Adds voice as a second biometric modality to @ever-medos/bio-identity-kit, mirroring the shipped face modality, so a clinician can attest an AI-proposed order by voice. The attestation is a signed BiometricVerification Verifiable Credential bound to the order (target_ref) — a non-repudiable “Dr. X confirmed this AI order by voice.”

Origin: a request to “bring the User + Voice registration over from ever-sovereign-wallet.” Recon found the wallet’s voice code is biomarkers (energy/mood/vitality, self-vs-self, no speaker identity) — not attestation. So voice-as-identity is net-new; this doc is what was actually built. The wallet’s record-orb UX was the inspiration; its biomarker math was not used.

Why this shape

The kit is already modality-agnostic: BiometricModality includes 'voice'; engine.enroll/verify/identify are modality-neutral; verification_grants.modality is free text; the cosine EmbeddingMatcher compares any embeddings. So voice needed only new adapters + two additive edits (a new ai-order-signoff context). No engine change.

Face ↔ voice file parallel:

Concern	Face (shipped)	Voice (new)
Capture composition	capture.ts EmbeddingCapture	voice-capture.ts VoiceEmbeddingCapture (+ averageEmbeddings, captureEnrollment)
Acoustic/visual front-end	preprocessFaceFrame (in onnx-face-embedder.ts)	voiceFeatures.ts log-Mel fbank (own radix-2 FFT) + WESPEAKER_FBANK_OPTIONS / TITANET_FBANK_OPTIONS presets
ONNX embedder (dep-free, injected ort)	adapters/onnx-face-embedder.ts	adapters/onnx-voice-embedder.ts
Platform capture source (DOM, direct-import only)	adapters/browser-capture.ts	adapters/browser-mic.ts
Liveness	liveness.ts (gesture / passive)	voiceLiveness.ts (phrase-challenge / passive anti-spoof)
Matcher / engine / stores	reused unchanged (modality-agnostic)	reused unchanged
Frontend capture factory	web/src/services/ever-aaa/faceCapture.ts	web/src/services/ever-aaa/voiceCapture.ts
Backend trust anchor	services/auth/.../bioFaceLogin.helper.ts	services/auth/.../bioVoiceLogin.helper.ts
Backend endpoints	/v2/aaa/auth/face/{challenge,sign-in}	/v2/aaa/auth/voice/{challenge,verify,enroll}
UI	login/FaceLoginButton.tsx	bio-attestation/{VoiceEnrollmentPanel,VoiceSignoffButton,VoiceOrb}.tsx

The speaker model (research spike 2026-06-09)

Speaker verification runs client-side in onnxruntime-web, exactly like the face MobileFaceNet path.

• Prototype model = WeSpeaker ResNet34-LM (onnx-community/wespeaker-voxceleb-resnet34-LM): 256-d, EER 0.72%, pure-JS fbank (no STFT op in the graph) so it runs in ort-web. transformers.js ships the exact WeSpeakerFeatureExtractor, so the fbank params are verified and reproduced byte-for-byte in voiceFeatures.ts (WESPEAKER_FBANK_OPTIONS).
• ⚠️ Commercial licensing: WeSpeaker / ECAPA / pyannote / Resemblyzer weights are VoxCeleb-derived → non-commercial (CC BY-NC-ND cloud), despite repos tagged CC-BY/Apache. For a commercial deployment ship NVIDIA TitaNet (nvidia/speakerverification_en_titanet_large, CC-BY-4.0 commercial granted, 192-d, EER 0.66% — TITANET_FBANK_OPTIONS) or self-train on VoxBlink2 / consented staff voice. The model is pluggable via VITE_VOICE_MODEL_URL + VITE_VOICE_MODEL_KIND, so the swap is an env change, zero code.
• Exact WeSpeaker fbank: 16 kHz mono, 80-d kaldi log-Mel (1127·ln(1+f/700) ≡ htk), 25/10 ms, n_fft 512, hamming (periodic=false), power 2.0, pre-emph 0.97 per-frame, remove_dc_offset, natural log, mel_floor 1.19e-7, ×32768 scale, snip_edges, CMVN = per-bin mean. Input [1,T,80] → output embs [1,256]. (TitaNet preset = hann / slaney / dither 1e-5 / per-feature norm.)
• Threshold: cosine ≥ 0.50 to start (VOICE_LOGIN_MIN_SCORE), recalibrate to the real mic/booth at FAR ≤ 0.1%. Enroll 3–5 utterances, average L2-normalized embeddings.
• Liveness v1 = text-dependent challenge: a fresh random digit phrase per attempt; the user reads it; a local STT transcript must match (PhraseChallengeLivenessProvider). A stolen recording fails a fresh challenge. Do NOT use webkitSpeechRecognition for the phrase check — Chrome ships audio to Google (PHI leak); use local Whisper-tiny (transformers.js, MIT). Passive anti-spoof (AASIST/RawNet2) is a v2 OnnxVoiceAntiSpoofProvider slot (ASVspoof license + ort-web SincConv export still to validate).

Data flow

Enrollment (staff registration):

VoiceEnrollmentPanel → captureVoiceProbe ×N → averageEmbeddings (kit)
  → POST /v2/aaa/auth/voice/enroll {subjectRef, template, sampleCount, contexts}
  → enrollVoice(): upsert bio_identities (+'voice') + verification_grants
      (source='enrollment', modality='voice', scope_contexts=['ai-order-signoff','staff-login'], standing expiry)
      + BiometricEnrollment attestation

Sign-off (AI order):

clinician accepts an AI-proposed order
  → VoiceSignoffButton → POST /voice/challenge → {challengeId, phrase}
  → record (orb) → captureVoiceProbe → {probe, audio}; local STT(audio) → transcript
  → POST /voice/verify {subjectRef, template, transcript, challengeId, targetRef, context:'ai-order-signoff'}
  → service: consume challenge (single-use) → phraseRatioOk(phrase,transcript)
           → verifyVoice() cosine vs enrolled grant ≥ VOICE_LOGIN_MIN_SCORE
           → appendVoiceAttestation() → bio_attestations row bound to the order (target_ref)
           → emitVoiceEvent() → user_action_events (RUDS) VOICE_SIGNOFF_SUCCESS/DENIED
  → {verified, score, attestationId}

The engine invariant carries over: no enrolled voiceprint grant ⇒ no verify (no-active-grant), and only successful verifications are filed.

Files

Kit (web/packages/bio-identity-kit/src/): voice-capture.ts, voiceFeatures.ts, voiceLiveness.ts, adapters/onnx-voice-embedder.ts, adapters/browser-mic.ts; additive edits to types.ts (ai-order-signoff) + config.ts; barrels updated (browser-mic excluded, DOM); voice-identity.test.ts. Frontend: web/src/services/ever-aaa/voiceCapture.ts (capture factory), voiceAttestation.service.ts (HTTP client + factories); web/src/common/components/bio-attestation/ (VoiceOrb, VoiceEnrollmentPanel, VoiceSignoffButton, types.ts, index.ts). Backend (services/auth/.../auth/): bioVoiceLogin.helper.ts, dto/voiceSignoff.dto.ts, voice* methods in auth.service.ts, 3 routes in auth.controller.mixin.ts. Sandbox: web/sandbox/targets/VoiceAttestationTarget.tsx (?target=VoiceAttestation). Persistence: existing infrastructure/medbase/migrations/20260531a_bio_identity_attestation.sql — no new migration needed (modality is free text; the phrase reuses the bio_capture_challenges.gesture text column; an optional modality column is a future nicety, deliberately not required so the helper works against an unmigrated gesture-only table).

Config flags

Flag	Default	Purpose
VITE_VOICE_LOGIN_ENABLED	false	Gates the enrollment/sign-off UI (components return null when off → mounting them anywhere is zero-risk)
VITE_VOICE_MODEL_URL	—	Speaker-embedding .onnx. Absent ⇒ capture is null ⇒ UI shows “model not configured”
VITE_VOICE_MODEL_KIND	wespeaker	wespeaker | titanet — picks the fbank preset
VITE_VOICE_MODEL_INPUT/OUTPUT	feats/embs	ONNX tensor names
VITE_VOICE_ANTISPOOF_MODEL_URL	—	Optional passive anti-spoof model
VOICE_LOGIN_ENABLED (backend)	false	Gates the endpoints (else 403)
VOICE_LOGIN_MIN_SCORE	0.5	Cosine accept threshold (tune per model + booth)
VOICE_REQUIRE_PHRASE	true	Require the phrase-challenge liveness
BIO_ISSUER_DID	did:web:hospital	Attestation issuer DID

Wiring AI order sign-off (the next step)

The VoiceSignoffButton is designed to drop into the AI-order accept UX with injected client/capture. Because it returns null when VITE_VOICE_LOGIN_ENABLED is off, mounting it is safe even before the model is provisioned. Two natural hosts:

• Voice orders: web/src/common/components/medical/order/VoiceOrderBridge.tsx / VoiceOrderDialogChrome.tsx — gate the submitForReview accept on a successful sign-off.
• AI cowork proposals: web/src/common/components/medical/builder/acknowledgements/CoworkProposalPanel.tsx — add a sign-off step to Accept (ties into cowork_proposals / agent_proposal).

Recipe:

import { VoiceSignoffButton } from '@/common/components/bio-attestation';
import { makeVoiceSignoffClient, voiceCaptureFn } from '@/services/ever-aaa/voiceAttestation.service';

<VoiceSignoffButton
  client={makeVoiceSignoffClient()}
  capture={voiceCaptureFn}
  subjectRef={currentUser.username}
  targetRef={`OrderRequest/${proposal.id}`}
  orderSummary={proposal.summary}
  onSignedOff={({ attestationId }) => acceptOrder({ attestationId })}  // gate the real accept on this
/>

Pass a local-Whisper transcribe prop to enable the phrase-liveness content check.

Staff enrollment wizard (add-user page)

Enrollment is surfaced as a wallet-style setup wizard (in the spirit of the Ever wallet’s first-run capture moments) that opens after creating a new user in user management (UsersPage.tsx → “Add New User” → on success). This required a face-enrollment twin of the voice path (we’d previously built face login only):

• Backend: enrollFace in bioFaceLogin.helper.ts + FaceEnrollRequestDto + faceEnroll service method + POST /v2/aaa/auth/face/enroll (mirrors enrollVoice, standing grant scoped staff-login).
• Frontend capture: getFaceEmbedder + captureFaceProbe + releaseFaceCamera added to faceCapture.ts (additive; login path untouched); faceAttestation.service.ts (faceEnroll client + faceCaptureFn).
• UI: bio-attestation/{FaceScanFrame, FaceEnrollmentPanel} (face twins of VoiceOrb/VoiceEnrollmentPanel) + BiometricEnrollmentWizard (MUI Stepper: Face → Voice → Done, each skippable, per-modality flag-gated) + BiometricEnrollmentWizardConnected (wires the real services; what UsersPage renders).
• Wiring (3 surfaces, all flag-gated by BIO_ENROLL_ENABLED = VITE_FACE_LOGIN_ENABLED || VITE_VOICE_LOGIN_ENABLED; the wizard also returns null when no modality is enabled, so every surface is byte-identical until biometrics are on):
  1. Add-user: UsersPage.submitAdd sets enrollFor={username,name} on success → ``.
  2. Existing staff: a per-row “Enroll face / voice” IconButton in UsersPage opens the same wizard for any existing user (subjectRef = user.username).
  3. Patient registration: patient-roster FormRegister createPatient onSuccess opens the wizard (subjectType="patient", contexts=['patient-unlock','med-admin-five-rights']), deferring the dialog-close/visit-create until the wizard finishes — only when bioEnrollEnabled (else the registration flow is unchanged).
• subjectType threaded end-to-end (types → panels → wizard → Connected → backend Face/VoiceEnrollRequestDto → service → enrollFace/enrollVoice, default practitioner) so patient enrollments create a patient identity, staff a practitioner one.
• Look & feel (ever-sovereign-wallet, hybrid): the wallet’s signature aesthetic, brought over truthfully but with the medOS palette (which already matches: TEAL #17b8e0 / GREEN #22c55e). VoiceOrb reproduces the wallet’s Fibonacci-sphere particle orb (breathing, amplitude-reactive, per-particle identiconColors homogenised to the username) in pure 2D canvas — no Three.js dependency (the hybrid choice). GradientPillButton = the wallet’s teal→green / red→orange gradient capture pills with glow + press-dip. walletStyle.ts ports the wallet’s identiconColors verbatim. The wizard adds a custom gradient progress rail + framer-motion step transitions + gradient accent bar. FaceScanFrame = brand-gradient reticle. All in web/src/common/components/bio-attestation/.
• Sandbox: ?target=BiometricEnrollmentWizard (mock clients + captures) — browser-verified: face 3 frames → voice 3 samples → “Biometric setup complete” (Face/Voice enrolled), 0 console errors.

Verified vs. remaining

Verified now: kit strict tsc = 0 errors + a 10-check tsx runtime proof (enroll-average → standing ai-order-signoff grant → signed attestation bound to the order; non-match rejected; fbank shape/finite for all presets; 48→16 kHz resample; injected-ort embedder; phrase-challenge anti-replay) + a jest test; the auth service tsc -p adds 0 errors (the 2 pre-existing ever.moleculer.config errors are unrelated); the sandbox ?target=VoiceAttestation is browser-verified (enroll 3 samples → enrolled; sign-off VERIFIED 93% + success banner; wrong-voice → “did not match”; 0 console errors).

Remaining to go live (mirrors how face shipped — code-complete, flag-off, model-pluggable): provision a speaker .onnx (pnpm add onnxruntime-web is already declared for face) + set VITE_VOICE_MODEL_URL; resolve the commercial license (WeSpeaker → TitaNet/self-train); apply migration 20260531a; wire a local Whisper-tiny transcribe; recalibrate the threshold on real mics; add passive anti-spoof; security review; then flip VOICE_LOGIN_ENABLED + VITE_VOICE_LOGIN_ENABLED; finally mount VoiceSignoffButton in the chosen accept host. Backend is not committed/pushed (a push auto-deploys services/auth).