Currently Healthcare Data Platform — Operator Console

Document ID: OPS-001
Classification: Internal — Restricted
Version: 1.0
Scope: Full operational surface for platform engineering, SRE, and support teams

---

Table of Contents

1. Admin Dashboard
2. Internal Tooling Inventory
3. Operational Workflows
4. Monitoring and Alerting
5. Support Workflows
6. Environment Management
7. Secret Management
8. Health Checks and Status

---

1. Admin Dashboard

1.1 Platform Admin Shell

The platform administrator interface is rendered under the /admin route group, served by the Next.js application on Vercel. The shell component is defined in web/src/app/(app)/(platform)/admin/_components/platform-admin-shell.tsx and provides the navigation frame for all platform-scoped administrative views. Access requires a Clerk session with the platform_admin role.

1.2 Admin Route Inventory

All admin routes are prefixed under /admin (UI) and /api/admin or /api/v1/admin (API). Routes without a v1 prefix are considered legacy and will be migrated.

UI Routes

API Routes — Legacy Prefix (/api/admin/)

API Routes — Versioned Prefix (/api/v1/admin/)

1.3 HIE Network Dashboard

HIE operators access a separate scoped interface under /dashboard/network. This shell is defined in web/src/app/(app)/(hie)/dashboard/network/layout.tsx and is gated to users with the HIE role in Clerk.

1.4 Developer-Only Routes

This route must not be accessible in production. Verify that environment checks gate this handler before any production deployment.

---

2. Internal Tooling Inventory

2.1 Currently CLI (cli/)

The currently CLI is a TypeScript command-line tool built with tsx, compiled via tsup, and distributed from cli/dist/bin/currently.js. The entry point is cli/bin/currently.ts. It communicates with the platform API via cli/src/client.ts using API keys stored locally by cli/src/auth/store.ts.

Authentication

Command Surface

CLI Configuration

Default values are defined in cli/src/config/defaults.ts. Named profiles (e.g., production, staging) are managed in cli/src/config/profiles.ts, enabling operators to switch target environments without modifying credentials.

2.2 GitHub Actions Workflows

All CI/CD automation lives under .github/workflows/.

workflow_dispatch triggers are available on pipeline-deploy and pipeline-worker-deploy, allowing manual deployment from the GitHub Actions UI without a code push.

2.3 Docker Images

All images run as non-root user appuser. The base image is python:3.11-slim. System dependencies installed at build time: gcc, postgresql-client, curl.

2.4 Configuration Files

2.5 Procfile

A Procfile is present at the repository root, indicating Railway or Heroku-compatible process definitions as a secondary deployment target alongside GKE.

---

3. Operational Workflows

3.1 Deploy — Pipeline Service

Trigger: Push to main or manual workflow_dispatch on pipeline-deploy.yml.

Steps:

1. GitHub Actions builds the Docker image from Dockerfile.pipeline.
2. Image is pushed to the GCP Container Registry.
3. GKE deployment is updated via kubectl set image or equivalent Helm rollout.
4. smoke-test.yml fires automatically post-deploy, hitting /health on the newly deployed pods.
5. If smoke tests fail, the pipeline is halted and the team is notified.

Manual trigger:

GitHub → Actions → pipeline-deploy → Run workflow → Select branch

3.2 Deploy — Celery Workers

Trigger: Push to main or manual workflow_dispatch on pipeline-worker-deploy.yml.

Workers are deployed independently from the API service. This allows scaling worker replicas without touching the API surface. The Celery broker is Redis; ensure Redis connectivity is confirmed before scaling workers.

3.3 Deploy — Web Application

The Next.js web application deploys automatically to Vercel on push to main. No manual workflow is required. Environment variables are configured in the Vercel project dashboard.

3.4 Rollback

Pipeline/Worker rollback (GKE):

kubectl rollout undo deployment/currently-pipeline
kubectl rollout undo deployment/currently-worker

Confirm rollback status:

kubectl rollout status deployment/currently-pipeline

Web rollback (Vercel): Navigate to the Vercel dashboard → Deployments → select the previous successful deployment → Promote to Production.

3.5 Feature Flags

Feature flags are managed via the settings JSONB column on the organizations table in the Platform Cloud SQL database. Module-level toggles for HIE organizations are surfaced in the UI at /dashboard/network/settings/modules.

To enable or disable a module for a specific organization directly in the database:

UPDATE organizations
SET settings = jsonb_set(settings, '{modules,omop_export}', 'true')
WHERE slug = '<org-slug>';

The conformance_toggle feature is tested end-to-end in tests/e2e/test_conformance_toggle.py, confirming runtime behavior responds correctly to flag state.

3.6 User Management

User lifecycle is owned by Clerk. The platform database (users table on orchestrator 192.168.0.251) mirrors Clerk user state via the webhook handler at POST /api/auth/sync-user.

Administrative user operations:

Invitations use tokenized links; the token resolution handler is at GET /api/invitations/:token.

3.7 API Key Management

Platform-level API keys are managed via /api/v1/admin/api-keys. HIE-scoped keys are managed by HIE administrators at /dashboard/network/settings/api-keys, backed by the hie-api-keys-page-client.tsx component. Patient-scoped API access is configurable at /portal/settings/api-access.

Aggregate API key usage metrics are available at GET /api/v1/admin/api-keys/metrics/platform.

3.8 NPI Verification

Provider NPI numbers are verified in real-time via GET /api/npi/verify, which calls the NPPES NPI Registry. This is invoked during facility onboarding.

---

4. Monitoring and Alerting

4.1 What Is Observed

4.2 Docker Health Check

The FastAPI container health check is defined in Dockerfile:

HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

A container that fails 3 consecutive checks is marked unhealthy by the container runtime. GKE will restart unhealthy pods automatically.

4.3 End-to-End Monitoring Tests

The test file tests/e2e/test_monitoring.py validates that monitoring instrumentation is functioning. These tests run in the pipeline-test workflow and should pass before any production deployment is promoted.

4.4 Smoke Tests

Post-deploy smoke tests are defined in .github/workflows/smoke-test.yml. They run automatically after pipeline-deploy and pipeline-worker-deploy complete. A smoke test failure must block promotion and trigger immediate investigation.

4.5 Alerting and Escalation

The repository does not contain a dedicated alerting configuration file (e.g., PagerDuty, Alertmanager). The following escalation path is inferred from the architecture:

---

5. Support Workflows

5.1 Ticket Triage

Incoming support requests should be categorized against the following matrix before assignment:

5.2 Common Issues and Resolution Playbooks

PLAY-001: Facility User Cannot Log In

1. Confirm user exists in Clerk dashboard with correct email.
2. Check users table: SELECT * FROM users WHERE email = '<email>';
3. Verify is_active = true and organization_id is not null.
4. If the users row is missing, the Clerk webhook sync may have failed. Manually trigger re-sync or re-invite via POST /api/admin/invitations/facility-user.
5. Verify the user's Clerk role (clerk_role) matches the expected scope.

PLAY-002: Medical Record Stuck in processing

1. Query file_registry: SELECT id, status, created_at FROM file_registry WHERE status = 'processing' AND created_at < NOW() - INTERVAL '1 hour';
2. Check Celery worker logs for the corresponding job ID.
3. If the job is in the DLQ, retrieve the error trace from the DLQ topic and examine the payload.
4. Common causes: malformed HL7 segment, unsupported message type, vocabulary lookup failure against OMOP Athena.
5. For malformed payloads, update status to failed, notify the uploading organization, and provide the error detail.
6. For vocabulary misses, check config/fhir_mappings/vocabulary_maps.json and config/crosswalk/hl7_table_contexts.json for missing mappings.

PLAY-003: HIE Pending Approval Not Resolving

1. Navigate to /admin/registrations/hie in the platform admin UI, or GET /api/admin/registrations/hie.
2. Confirm the registration record exists and its status.
3. Use PATCH /api/admin/registrations/hie/:id to update state, or POST /api/admin/org-links/approve to approve the link.
4. Verify the associated Clerk organization exists and clerk_org_id is populated in the organizations table.

PLAY-004: API Key Returns 401

1. Confirm the key exists: GET /api/v1/admin/api-keys.
2. Check the key's scope — HIE keys, platform keys, and patient keys are distinct and not interchangeable.
3. Verify the key has not been revoked. If revoked, issue a new key via the appropriate admin UI or API.
4. Check GET /api/v1/admin/api-keys/metrics/platform for recent usage — no activity may indicate the wrong key is being used.

PLAY-005: Subscription / Checkout Failure

1. Check GET /api/admin/revenue/stats for subscription state.
2. Use POST /api/subscriptions/checkout to re-initiate checkout, or POST /api/subscriptions/cancel if the customer is cancelling.
3. Verify pricing tier configuration at GET /api/admin/pricing/tiers.

PLAY-006: PHI Audit Log Review

1. Retrieve audit events: GET /api/v1/admin/audit/phi-access with appropriate date range parameters.
2. Cross-reference with user sessions in Clerk.
3. Any unexpected access event must be escalated to Compliance immediately and documented. Do not delete audit records.

5.3 NPI Verification Failure

If a facility cannot complete onboarding due to NPI verification failure:

1. Manually verify the NPI at https://npiregistry.cms.hhs.gov/.
2. If the NPI is valid but the API is returning errors, check whether the NPPES API is experiencing an outage.
3. If NPPES is down, temporarily bypass NPI verification with a manual approval via PATCH /api/admin/registrations/facility/:id and document the exception.

---

6. Environment Management

6.1 Environment Overview

The orchestrator database (non-PHI) runs on the on-premise host at 192.168.0.251. The PHI Cloud SQL instance is GCP-managed and access is restricted to the FastAPI/Celery GKE workloads via Workload Identity (see commit fix: use workload identity for GCS access).

6.2 Environment Variable Configuration

The template for remote environment configuration is config/env.remote-example. Copy this file to .env and populate values before running services outside of CI.

Critical variables by service:

6.3 Developer Environment

Local development uses the profiles system in cli/src/config/profiles.ts. Developers should configure a development profile pointing to localhost:8000. PHI data must never be loaded into local development databases; use synthetic HL7/CCDA test fixtures from the tests/ directory.

6.4 Ignored Files by Environment

---

7. Secret Management

7.1 Secret Inventory

7.2 Secret Scanning

GitLeaks is configured via .gitleaks.toml and runs as a pre-commit hook (.pre-commit-config.yaml). Any commit containing a pattern matching a known secret format is blocked. The code-quality.yml workflow also runs secret scanning in CI.

If a secret is detected in git history:

1. Immediately rotate the affected credential.
2. Force-push a cleaned history using git filter-repo or BFG Repo Cleaner.
3. Notify the security team and document the incident.
4. Audit access logs for the affected credential's usage window.

7.3 PHI Isolation

The PHI Cloud SQL instance is never accessible from the Next.js web tier. Only the FastAPI and Celery workloads running on GKE with Workload Identity bindings can connect. The NEXT_PUBLIC_BACKEND_URL variable is guarded at the call site in the web application to prevent accidental direct PHI database exposure.

De-identification rules are maintained in config/deid/default_rules.csv. Any change to these rules requires review by the Compliance lead before deployment.

---

8. Health Checks and Status

8.1 FastAPI Health Endpoint

Route: GET http://<api-host>:8000/health
Defined in: src/api/main.py (FastAPI application entry point)
Expected response: HTTP 200 with a JSON body indicating service status.
Docker HEALTHCHECK: Every 30 seconds, 10-second timeout, 40-second start period, 3 retries before marking unhealthy.

# Manual check
curl -f http://localhost:8000/health

8.2 Next.js Web Health Endpoint

Route: GET /api/web/health
File: web/src/app/api/health/route.ts
Expected response: HTTP 200.
Use: Vercel deployment health checks and uptime monitoring.

curl -f https://<web-host>/api/health

8.3 Database Connectivity

Database connectivity is implicitly verified by the API health check — if the platform database is unreachable, the health endpoint should reflect that. Operators can manually verify connectivity using postgresql-client (included in the API Docker image):

# From within a running API container
psql $DATABASE_URL -c "SELECT 1;"

The orchestrator database host is 192.168.0.251. Ensure network policy allows GKE pods egress to this host if on-premise connectivity is required.

8.4 Pipeline Job Health

Job health is observable through the file_registry table. The following query identifies jobs that have been stuck in processing for more than 30 minutes, which indicates a likely worker failure:

SELECT id, file_name, file_type, status, created_at, updated_at
FROM file_registry
WHERE status = 'processing'
  AND updated_at < NOW() - INTERVAL '30 minutes'
ORDER BY updated_at ASC;

A high count of failed records indicates a systemic pipeline issue and should trigger P2 escalation.

8.5 Post-Deploy Verification Checklist

After any production deployment, complete the following checks before declaring the release stable:

• GET /api/web/health returns HTTP 200
• GET http://<api-host>:8000/health returns HTTP 200
• Smoke test workflow (smoke-test.yml) passed in GitHub Actions
• /dashboard/network/jobs shows no unexpected increase in failed jobs
• GET /api/v1/admin/api-keys/metrics/platform shows no authentication errors
• Celery workers are consuming from the queue (no queue depth growth in Redis)
• PHI database connectivity confirmed via API health check
• No unexpected entries in GET /api/v1/admin/audit/phi-access since deployment

---

This document reflects the repository state as of commit fc207dd. Update after each significant infrastructure change. Owner: Platform Engineering.