Currently — Live Project Memory

Document Version: Live (auto-snapshot) Last Materialized From: Commit fc207dd (HEAD) Methodology: PRD-driven Context Engineering v0.1+ Classification: Internal — No PHI

---

1. Project Identity

Currently is an AI-powered healthcare data integration platform that ingests raw clinical documents (HL7 v2 messages, CDA/CCDA XML) from health information exchanges (HIEs) and transforms them into FHIR R4 resources and OMOP CDM 5.4 records. Terminology codes are resolved against the OMOP Athena vocabulary (6.3M concepts) covering LOINC, SNOMED CT, ICD-10-CM, and RxNorm.

The platform serves three distinct user personas through four authenticated shells: platform administrators (/admin), HIE operators (/dashboard/network), facility providers (/dashboard), and patients (/portal).

Current lifecycle stage: v0.7–v0.8 (Beta / Pre-Growth). Core pipeline is production-deployed. Web platform is feature-complete at the shell level. Active work is concentrated on production hardening, auth wiring, and branding polish.

---

2. System Architecture Snapshot

2.1 Runtime Topology

2.2 Data Boundary Decision (TECH-001)

Decision: PHI and non-PHI are partitioned into two separate Cloud SQL instances. The Platform DB (orchestrator, 192.168.0.251) holds organizations, users, file registry metadata, job tracking, and CRM/CMS data. The PHI DB holds medical_records, FHIR bundles, and the full omop_cdm54 schema including the 6.3M-concept Athena vocabulary table.

Rationale: Compliance isolation. Any accidental logging, query introspection, or lateral movement within the platform layer cannot expose PHI. FHIR/OMOP data never enters the Platform DB.

Impact: Every API route that touches clinical data must connect to the PHI DB via a separate connection string. This has been the source of several recent fix: commits around GCS auth and worker wiring.

2.3 Four-Shell UI Architecture (TECH-002)

web/src/app/(app)/
├── (platform)/admin/          # Platform super-admins
├── (hie)/dashboard/network/   # HIE operators
├── (provider)/dashboard/      # Facility/provider users
└── (patient)/portal/          # Patients

Each shell has its own layout, navigation, and role guard. Role resolution flows from Clerk organization membership → clerk_role → platform DB users.role.

---

3. Current Project State

3.1 What Has Been Built

Pipeline (Python / FastAPI / Celery)

• HL7 v2 parser supporting 18+ message types: ADT (A01–A31), ORU R01/R32, ORM O01, RDE O11, MDM T02, DFT P03, SIU S12, VXU V04 — profiles in config/hl7_profiles/
• CDA/CCDA XML parser with full C-CDA section support
• FHIR R4 output: Patient, Observation, Condition, MedicationRequest, AllergyIntolerance
• OMOP CDM 5.4 output: person, measurement, condition_occurrence, drug_exposure, visit_occurrence
• Athena vocabulary resolver (6.3M concepts, omop_cdm54.concept)
• FHIR datetime normalization — HL7 compact dates (YYYYMMDD) → FHIR dashed form (YYYY-MM-DD) fixed in commit b955f31
• De-identification rules via config/deid/default_rules.csv
• Crosswalk tables: config/crosswalk/hl7_table_contexts.json, config/fhir_mappings/vocabulary_maps.json
• Y7/Y8/Y9/R2 pipeline stages added in commit 9df7a3c with associated web schema changes
• Dead-letter queue (DLQ) routing with dedicated test coverage (tests/e2e/test_dlq_routing.py)
• Conformance toggle (runtime-switchable strictness level, tests/e2e/test_conformance_toggle.py)
• Rule audit trail (tests/e2e/test_rule_audit.py)
• Phase 3 orchestrator (tests/e2e/test_phase3_orchestrator.py)

Web Platform (Next.js)

• Clerk webhook sync at /api/auth/sync-user — real-time user/org sync into Platform DB
• Admin shell: registrations (HIE + facility), invitations, org-link approvals, pricing tiers, revenue stats, analytics, API key management, PHI access audit, provider approval
• HIE shell: network overview, analytics, approvals, patient list with MRN-hash routing, provider management, records, jobs, upload, export, API keys, Google Workspace integration
• Patient portal: profile, records, sharing, API access self-service
• NPI verification endpoint (/api/npi/verify)
• Subscription lifecycle: checkout + cancel (/api/subscriptions/checkout, /api/subscriptions/cancel)
• Developer email preview endpoint (/api/dev/email-preview)
• CLI tool (cli/) with commands: allergies, analytics, auth, billing, config, diagnoses, encounters, keys, medications, orgs, patient, pipeline, quality, records, storage, webhooks
• Internal API docs page with code generators (web/src/app/(app)/(platform)/admin/api-docs/)
• ATS (applicant tracking) module under /admin/ats/ — unusual for a healthcare platform; indicates internal hiring tooling was embedded in the admin shell

Infrastructure

• 4 Dockerfiles: Dockerfile (API), Dockerfile.pipeline, Dockerfile.worker, Dockerfile.nlp
• 7 GitHub Actions workflows: code quality, pipeline deploy, pipeline test, worker deploy, smoke test, docs sync, Terraform plan
• Terraform plan workflow exists; full apply not confirmed from available context
• SBOM attestation step in CI (marked continue-on-error in commit 2942e46)
• Secrets management via .gitleaks.toml (secret scanning pre-commit hook)
• .hypothesis/ directory committed — property-based test corpus is version-controlled

3.2 What Is In Progress

3.3 What Is Blocked / At Risk

---

4. Active Decisions and Rationale

TECH-001 — PHI / Non-PHI Database Split

See §2.2 above. Settled. Not under reconsideration.

TECH-002 — Four-Shell UI via Next.js Route Groups

Decision: Use Next.js parallel route groups (platform), (hie), (provider), (patient) within a single Next.js app on Vercel rather than four separate applications. Rationale: Shared component library, single deployment unit, Clerk context available across all shells. Role-based shell isolation is enforced at the layout level via Clerk session claims. Trade-off accepted: A single compromised layout or middleware could leak cross-shell data. This is mitigated by Clerk's server-side auth checks in each route handler.

TECH-005 — Remove Cloud Tasks SDK, Use REST Enqueue

Decision: Cloud Tasks client library removed (ada7aae). Pipeline tasks are now enqueued via direct HTTP REST calls to the Cloud Tasks API. Rationale: Reduces dependency surface, avoids Cloud Tasks SDK version conflicts with the Python 3.11 runtime, enables easier local simulation. Open question (UJ-001): Are retries and dead-lettering semantics fully replicated under the REST enqueue pattern? The DLQ routing tests (tests/e2e/test_dlq_routing.py) should confirm this, but it is not clear from commit messages alone whether those tests were re-validated after the migration.

TECH-009 — Clerk as Sole Identity Provider

Decision: Clerk manages organizations, users, roles, and invitations. The platform DB mirrors Clerk state via webhook (/api/auth/sync-user). There is no separate identity service. Rationale: Clerk's organization model maps cleanly to HIE → facility hierarchy. Invitation flows for facility users and patients (/api/admin/invitations/facility-user, /api/admin/invitations/patient) are handled via Clerk invitation tokens. Trade-off accepted: Platform is tightly coupled to Clerk. Migration to another IdP would require rewriting the entire auth layer.

TECH-010 — Hypothesis Property-Based Testing for Pipeline

Decision: .hypothesis/ corpus is committed to the repository. Property-based tests (tests/conformance/test_property_based.py, tests/conformance/test_fuzz.py) run as part of CI. Rationale: Clinical data is highly variable. Property-based testing catches edge cases that example-based tests miss (e.g., the HL7 compact date bug fixed in b955f31 was likely surfaced through fuzz/property tests).

TECH-011 — ATS Module Inside Admin Shell

Decision: An applicant tracking system is embedded under web/src/app/(app)/(platform)/admin/ats/. This is a non-clinical internal tooling module co-located with the healthcare platform admin. Rationale (inferred): Convenience — reuse of the existing admin shell auth and UI patterns. Avoids maintaining a separate application. Trade-off: Increases admin shell complexity. ATS data could inadvertently appear in platform analytics if not carefully scoped.

---

5. Recent Changes and Their Impact

---

6. Open Questions and Unresolved Trade-offs

---

7. Technical Debt Inventory

---

8. Test Coverage Architecture

tests/
├── conformance/
│   ├── test_fuzz.py                    # Fuzz testing via Hypothesis
│   ├── test_hl7_conformance_integration.py
│   └── test_property_based.py         # Property-based testing
├── e2e/
│   ├── test_canonical_consistency.py
│   ├── test_conformance_toggle.py      # Runtime conformance strictness
│   ├── test_dlq_routing.py             # Dead-letter queue behavior
│   ├── test_fhir_full_pipeline.py      # Full FHIR output validation
│   ├── test_hl7_full_pipeline.py       # Full HL7 ingestion → output
│   ├── test_monitoring.py
│   ├── test_omop_csv_round_trip.py     # OMOP round-trip fidelity
│   ├── test_omop_full_pipeline.py
│   ├── test_performance.py             # Performance baselines
│   ├── test_phase3_orchestrator.py     # Phase 3 orchestrator
│   ├── test_rule_audit.py              # Rule audit trail
│   └── test_y1_y2_edge_cases.py        # Y1/Y2 edge case regression
└── conftest.py

Coverage observations:

• E2E tests cover the full pipeline from HL7 ingest through FHIR/OMOP output. This is strong for a healthcare data platform.
• test_y1_y2_edge_cases.py covers Y1/Y2 stages. test_phase3_orchestrator.py covers phase 3. Y7/Y8/Y9/R2 stages (added in 9df7a3c) need corresponding test files — none are visible in the current file tree.
• test_performance.py exists but performance baselines are not documented in available context (see §9).
• 110 test files across Python. Frontend test coverage is not visible in the available file tree — this may be a gap.

---

9. Performance Baselines

Performance baselines are tracked in tests/e2e/test_performance.py. Specific numbers are not available from the current repository context snapshot. The following are known constraints from architecture:

Action needed (TECH-012): Extract and document baseline numbers from test_performance.py into this document and into the project status report.

---

10. Integration Points and Health

---

11. Team Conventions and Patterns

11.1 Cursor Rules (.cursor/rules/)

Four explicit convention files are maintained:

• project-conventions.mdc — General project conventions
• python-pipeline.mdc — Python pipeline-specific rules
• testing.mdc — Testing standards
• web-frontend.mdc — Next.js frontend patterns

This indicates the team uses Cursor AI tooling and has codified conventions for AI-assisted development. These files should be treated as the canonical style guide.

11.2 Commit Message Convention

Commits follow Conventional Commits: fix:, feat:, ci:, fix(web):. This is enforced via .pre-commit-config.yaml.

11.3 Secret Scanning

.gitleaks.toml is present — secret scanning runs pre-commit. This is appropriate for a HIPAA-adjacent codebase.

11.4 Environment Config

config/env.remote-example provides the template for remote environment configuration. Local vs. remote environments are explicitly distinguished.

11.5 API Versioning Pattern

Newer admin endpoints are under /api/v1/admin/ (e.g., /api/v1/admin/api-keys, /api/v1/admin/approve-provider). Older endpoints are unversioned (e.g., /api/admin/registrations/hie). Convention: New admin API routes should be under /api/v1/. Existing unversioned routes represent migration debt.

11.6 PHI Access Audit

/api/v1/admin/audit/phi-access exists as a dedicated audit endpoint. Any access to PHI-adjacent data must be logged through this mechanism.

11.7 MRN Hashing

Patient routes use hashed MRNs: web/src/app/(app)/(hie)/dashboard/network/patients/[mrnHash]/. MRNs are never stored or transmitted as plaintext in the URL space.

---

12. Architectural Evolution

Phase 1 — Core Pipeline (Pre-current)

Initial HL7 and CDA parsers. FHIR R4 output only. Single database. Deployed via Fly.io (config/fly.toml and Procfile are artifacts of this era).

Phase 2 — PHI Isolation + OMOP

PHI DB separated from Platform DB. OMOP CDM 5.4 output added. Athena vocabulary loaded. GKE deployment introduced. Celery workers added for async processing.

Phase 3 — Multi-Tenant Web Platform

Four-shell Next.js app built. Clerk integration for multi-tenant org management. File registry, job tracking, and user management in Platform DB. Admin shell with registration workflows for HIEs and facilities.

Phase 4 (Current) — Production Hardening

Workload Identity replacing service account keys. Cloud Tasks SDK removed in favor of REST enqueue. FHIR validation authenticated. Pipeline enqueue failure surfacing. Y7/Y8/Y9/R2 pipeline stages added (scope unclear). CLI tooling shipped for developer access. SBOM attestation attempted (blocked).

Phase 5 (Planned/Emerging)

Evidence: pricing tiers endpoint, subscription checkout/cancel, revenue stats endpoint, ATS module. Suggests monetization infrastructure is being built. Google Workspace integration under HIE settings. Patient API access self-service. These indicate a move toward a developer-facing API product and patient-controlled data sharing.

---

13. Key Learnings From Recent Development

1. Production worker wiring is fragile at this stack depth. Three sequential fix: commits (6d31ec1, 2ee79b5, 1ca44ba) targeted worker deployment within a short window. The GKE → Celery → Redis → GCS → Cloud Tasks chain has multiple auth surfaces that each need independent validation in CI. The smoke test workflow (.github/workflows/smoke-test.yml) should be extended to exercise the full enqueue-to-completion path.

2. GCS auth migration to Workload Identity was the right call but introduced transient failures. The sequence a4fd140 → e4b22ee suggests the initial Workload Identity commit (a4fd140) did not fully cover file sync downloads — a second commit was needed. When migrating GCS auth, enumerate every code path that touches GCS independently, not just the primary upload path.

3. FHIR datetime normalization is a systemic concern. The fix in b955f31 (_normalize_fhir_datetime) corrects compact HL7 date formats. Any field that accepts a date from an HL7 source must be routed through this normalizer. This should be a linting rule in python-pipeline.mdc.

4. Deferred environment variable guards create distributed risk. Moving NEXT_PUBLIC_BACKEND_URL guard to call sites (commit 19a5e90) trades a single initialization failure for N potential runtime failures. A better pattern is a typed configuration module that validates all required env vars at startup and throws a typed error, so the failure surface is small and explicit.

5. Rig bot PRs need a merge strategy. 10 open PRs (#51–#60) from an automated tool making overlapping changes to the same files will create conflicts. Establish a policy: bot PRs must be squash-merged or auto-closed after 48 hours if superseded.

6. The .hypothesis/ corpus being version-controlled helps reproduce failures (the corpus contains examples that previously caused failures, per Hypothesis design). However, the binary blob growth should be monitored. Consider a .hypothesis/.gitignore that keeps examples/ (failure corpus) but excludes constants/ (internal Hypothesis state).

---

14. Next Recommended Actions

---

This document reflects the repository state as of commit fc207dd. It should be updated after each sprint boundary or any commit that touches architecture, schema, or deployment configuration.