Colony — Live Project Memory

Document version: 2026-04-28 (post Phase 2 wave execution) Methodology: PRD-driven Context Engineering v0.1→v1.0 Owner: midwestco/colony

---

1. Project Identity

Colony is the command interface for the ZoomProp GTM Agentic Stack. It is a CRM-plus-agent-control-plane: a single surface where founders orchestrate outbound, content, pipeline management, recording intelligence, onboarding, and analytics through a streaming chat interface backed by specialist AI agents. The canonical spec lives at docs/ZP_GTM_Agentic_Stack_Draft_V1.pdf.

Stage: Active multi-phase build. Phase 1 (foundation + all specialist agents) is structurally complete. Phase 2 (discovery expansion, multi-channel, deliverability hardening, org management) is in execution with test scaffolding written and batch runner artifacts present.

---

2. Current Project State

2.1 What Has Been Built (Phase 1 — Structurally Complete)

2.2 Phase 2 — Current Execution State

Phase 2 planning is fully documented across 27 action plans in docs/phase2/action_plans/. Test specs for all 27 items are written in docs/phase2/testing/. Batch runner artifacts in runners/.artifacts/ confirm active execution as of 2026-04-28.

Key finding: Items 02 (Pipedrive LeadBooster scraper) and 07 (Playwright generic runner) have confirmed active runner execution. The remaining 25 Phase 2 items have test scaffolding written but no runner artifact evidence yet.

2.3 What Is In Progress

• TECH-001 Pipedrive LeadBooster batch scraping — Five batch runs executed on 2026-04-28, timestamps ranging 16:31–18:48 UTC. Artifacts include batch.log, collect-N.log, import-N.log, and summary.json per run. The evolution from dry-run (captures + screenshots) to full batch (import logs) indicates the pipeline matured intraday.
• TECH-002 Playwright generic runner — Dry-run sessions produced candidates.json, captures.json, organizations.json, people.json, and final.png screenshots. Session progressed from partial (16:31 — no organizations/people JSON) to full data capture (16:39 — all files present).
• TECH-003 Phase 2 wave orchestration — docs/phase2/action_plans/00a_WAVE_ORCHESTRATION.md exists, indicating deliberate sequencing. Items are being executed in waves rather than all at once.

2.4 What Is Blocked

---

3. Architecture Snapshot

3.1 Current Architecture (as deployed)

Clerk (Auth) ──Svix webhook──► Postgres 16 (colony-39989:…:colony)
                               ├── pgvector (1536-dim embeddings)
                               ├── pgcrypto (vault encryption)
                               └── api_keys (KMS-encrypted per-org)

Next.js 16 / React 19
├── /overview  ── SSE streaming ──► GTM Orchestrator (Inngest)
└── App Router                      ├── Campaign Orchestrator
                                    └── Specialist Agents (×8)

GCP Secret Manager (platform secrets)
Colony Vault (per-org: Pipedrive, Unipile, Resend, Google, Notion)

Integrations: Pipedrive · Unipile · Resend · Google Drive · Langfuse · Sentry
Storage: gs://colony-assets (CMEK + signed URLs)

3.2 How Architecture Has Evolved

Epoch 1 (pre-Phase 1): Pure spec. Architecture lived only in docs/ZP_GTM_Agentic_Stack_Draft_V1.pdf.

Epoch 2 (Phase 1 execution): All 26 action plans executed. Infrastructure fully Terraform-managed (infrastructure/gcp/). Two-secret-store pattern locked in: GCP Secret Manager for platform keys, Colony Vault for per-org keys. This was a deliberate security decision to prevent cross-org credential leakage.

Epoch 3 (Phase 2 — current): Discovery stack is being layered on top. Playwright is now a first-class infrastructure component (not a test-only tool), running as a generic scraping runner in runners/. The runners/.artifacts/ directory structure emerging intraday (2026-04-28) represents a new runtime artifact pattern — batch jobs now produce structured logs + summary JSON that feed back into the pipeline.

TECH-ARCH-001 Key architectural decision: Playwright as a production runner (not just CI test harness) was validated on 2026-04-28 with the Pipedrive LeadBooster scraper. The .github/workflows/playwright-runner.yml workflow formalizes this.

3.3 Two-Secret-Store Pattern (Active Decision)

Decision TECH-DEC-001: Maintain two separate secret stores.

• GCP Secret Manager — Clerk, Inngest, bootstrap LLM keys, Langfuse, Sentry. Injected at deploy time via gcloud run deploy --update-secrets=….
• Colony Vault — Per-org credentials (Pipedrive, Unipile, Resend, Google OAuth tokens). Stored as KMS-encrypted rows in api_keys table.

Rationale: Prevents a compromised org token from escalating to platform-level access. GCP Secret Manager has no concept of org-scoping; the vault pattern makes org isolation enforceable at the data layer.

Trade-off unresolved: KMS key rotation cadence for Colony Vault rows is not yet documented in any runbook. This is a latent TECH-DEBT item.

---

4. Active Decisions and Rationale

---

5. Recent Changes and Their Impact

2026-04-28 — Pipedrive Batch Runner Goes Live

Change: First full batch run of the Pipedrive LeadBooster scraper executed at 16:31 UTC, followed by four more runs through 18:48 UTC. Each run produced collect + import logs and a summary.json.

Impact:

• Confirms runners/ as a new production runtime directory, not just a scratch space.
• The progression from dry-run (no import logs) → batch (import logs present) in a single day indicates rapid iteration.
• Import artifacts (import-0.log, import-1.log, etc.) in runs 2 and 5 confirm data is actually landing in the database.
• Runs 3 and 4 (18:19, 18:22) have only batch.log + collect-0.log with no import logs — these may represent partial runs or runs that failed at the import stage. This is an open question (OQ-001).

2026-04-28 — Playwright Dry-Run Artifacts Evolve

Change: Three Playwright dry-run sessions ran between 16:31 and 16:41 UTC. Session at 16:39 was the first to produce organizations.json and people.json alongside candidates.json and captures.json.

Impact:

• Validates that the Playwright runner can extract structured entity data (orgs + people), not just raw page captures.
• final.png screenshots provide visual regression baseline for runner sessions.

2026-04-28 — Playwright MCP Console Log

Change: .playwright-mcp/console-2026-04-28T16-11-43-345Z.log and .playwright-mcp/page-2026-04-28T16-11-45-519Z.yml created.

Impact: Suggests Playwright MCP (Model Context Protocol) tooling is being used interactively, likely for developing/debugging runner scripts before committing them. This is a new development workflow pattern.

---

6. Open Questions and Unresolved Trade-offs

---

7. Technical Debt Inventory

---

8. Key Learnings from Recent Development Sessions

LEARN-001 — Playwright as a first-class production tool, not a test-only dependency. The decision to use Playwright for Pipedrive LeadBooster scraping (Phase 2, AP-07/02) confirms that in the agentic GTM context, browser automation is a data-collection primitive. The .github/workflows/playwright-runner.yml workflow formalizes this. Future agent capabilities that require authenticated SaaS data extraction should default to the Playwright runner pattern before assuming an API exists.

LEARN-002 — Batch runner maturation happens fast. Five batch runs in under 2.5 hours (16:31–18:48 UTC) with evolving artifact structures indicates a rapid iteration loop. The team moved from dry-run validation to full import within the same working session. This suggests the Playwright + batch runner pattern has low iteration cost once the initial session structure is established.

LEARN-003 — Artifact directories need a retention and gitignore policy immediately. Runner artifacts accumulating in runners/.artifacts/ and .playwright-mcp/ represent a real data governance gap. Prospect data (organizations.json, people.json, candidates.json) from real Pipedrive accounts should not live in git history. This is both a GDPR-adjacent concern and a repo hygiene issue.

LEARN-004 — Wave orchestration reduces integration failures. The explicit 00a_WAVE_ORCHESTRATION.md document for Phase 2 reflects a learned pattern from Phase 1: running all 26 action plans in parallel creates dependency failures (e.g., agents trying to call Knowledge Core before pgvector is seeded). Sequenced waves with explicit dependency gates reduce rework.

LEARN-005 — Codacy static analysis is configured but its CI integration health is unknown. .codacy/ is fully configured (ESLint, Semgrep, Trivy, Lizard, Pylint, Revive) but the actual CI trigger is not visible in the provided GitHub Actions workflows (only playwright-runner.yml and rbac-check.yml are listed). Codacy may be running as a webhook-triggered check rather than a workflow step.

LEARN-006 — The two-secret-store architecture creates a bootstrapping dependency. When a new org joins Colony, their per-org credentials must be added to the Colony Vault before any agent can act on their behalf. This onboarding step is manual today and is not represented in the Deployment Kit generation flow. It should be.

---

9. Team Conventions and Patterns

9.1 Documentation Conventions

• Action plans live in docs/phaseN/action_plans/ — numbered 00 (overview/orchestration) through N (feature). Each plan is a standalone markdown document with implementation instructions.
• Test specs mirror action plans 1:1 in docs/phaseN/testing/ with matching numeric prefixes. Spec files use TypeScript (.spec.ts) for Phase 2; Phase 1 used mixed .spec.md (prose specs) and .spec.ts.
• Runbooks live in docs/phaseN/runbooks/ — operational procedures for manual steps (e.g., DNS configuration, Google Workspace admin, Pipedrive field setup).
• ID prefix convention: Action plans use numeric prefixes (00–27). No BR/UJ/API/DBT-style IDs have been adopted in the codebase documentation yet — these appear only in external PRD tooling.

9.2 Infrastructure Conventions

• Terraform providers pinned via .terraform.lock.hcl — no terraform init -upgrade without explicit decision.
• All GCP resources managed via Terraform in infrastructure/gcp/. Direct console changes are treated as drift.
• Local Postgres for development via infrastructure/docker-compose.yml + infrastructure/postgres/init/.
• Terraform variables in infrastructure/gcp/terraform.tfvars (not terraform.tfvars.example — actual values file present, which reinforces OQ-002/DEBT-001 concerns).

9.3 Agent Runtime Conventions

• Every specialist agent has an action plan before implementation begins.
• Inngest durable functions are the execution primitive for all multi-turn agent loops.
• Circuit breakers are per-org — never global. This prevents one org's bad data or runaway agent from affecting another org's sending reputation or API quotas.
• Knowledge Core is consulted by every message/content/brief generation — exemplar retrieval via pgvector is a mandatory step, not optional.

9.4 Security Conventions

• KMS encryption at rest for all per-org API keys in Colony Vault.
• Signed URLs (not public reads) for all GCS asset access.
• CMEK for GCS bucket — customer-managed encryption keys via GCP KMS.
• RBAC enforced in CI via .github/workflows/rbac-check.yml — no bypass.

9.5 Runner/Scraper Conventions (Emerging — Phase 2)

• Dry-run before batch — Playwright sessions run in dry-run mode first, producing candidates.json and captures.json for validation, before batch mode with import logs.
• Timestamped artifact directories — each run gets a UTC ISO timestamp directory under runners/.artifacts/[job-type]/.
• summary.json is the canonical output of a completed batch run — downstream processes should read this, not individual collect/import logs.

---

10. Integration Points and Current Health