DataZoom — Progressive Product Requirements Document

---

PRD Metadata

---

v0.1 Spark — Problem & Outcomes

Problem Statement

Legal and business documents are dense, voluminous, and structurally inconsistent. During due diligence, fundraising, M&A review, or regulatory compliance, professionals must read hundreds of pages of contracts, equity agreements, IP assignments, and financial disclosures to extract specific facts, identify risks, and map relationships between parties. This process is slow (days to weeks), expensive (senior attorney time), and error-prone (key clauses missed under time pressure).

Existing document management tools (Google Drive, Dropbox, SharePoint) provide storage and search but offer no semantic understanding. Generic AI chatbots (ChatGPT, Claude) can summarize single documents but cannot reason across a corpus of 50–200 interrelated files, maintain source citations, manage organizational data isolation, or reconstruct equity ownership chains from fragmented transaction records.

DataZoom solves this by combining vector-based semantic search over chunked document corpora with a retrieval-augmented generation (RAG) pipeline and a structured data extraction layer. Users upload documents, and DataZoom extracts entities, dates, parties, and terms; embeds content into a pgvector-backed similarity index; and exposes a conversational interface where every answer cites the exact source chunk. Specialized modules for cap table reconstruction, due diligence checklist automation, timeline visualization, and strategic advisory extend the core RAG layer into domain-specific workflows.

Desired Outcomes

Constraints

• BR-001 Data Residency: All document processing must be executable locally (Ollama path) to satisfy customers with data residency requirements.
• BR-002 Citation Integrity: Every RAG response must surface source document and chunk-level references. No uncited assertions permitted.
• BR-003 Org Isolation: PostgreSQL Row-Level Security (RLS) enforces organization-based data isolation. Clerk organization ID is the primary tenant discriminator.
• BR-004 Vector Dimension Lock: Embeddings are fixed at 384 dimensions (all-MiniLM-L6-v2). Schema changes to document_chunks.embedding VECTOR(384) require a full re-embedding migration.
• BR-005 Cap Table Review Gate: AI-extracted cap table transactions require human approval via the review queue (/api/cap-table/review) before they are committed to the ledger. Automatic approval is prohibited.
• BR-006 LLM Routing: The model router (product/lib/__tests__/model-router.test.ts) determines whether queries are dispatched to Modal (Qwen2.5:32B cloud) or local Ollama. Routing logic must be deterministic and testable.

---

v0.2 Market Definition — ICP & Segments

Ideal Customer Profile

The primary buyer is a deal-side professional at a firm that regularly conducts document-intensive review cycles — either as a buyer (acquiring company, investor) or as a subject (startup raising capital, company undergoing legal audit). They have access to 10–500 documents per deal, a legal or finance team that currently reviews them manually, and a mandate to move faster without adding headcount.

Segments Table

"Not For"

• Consumer users seeking personal document help (lease review, personal tax prep). DataZoom's multi-tenant architecture, Clerk org management, and professional workflow depth are excessive and confusing for single-user consumer needs.
• Pure litigation / discovery workflows requiring Bates-numbering, privilege logging, or EDRM chain-of-custody. DataZoom processes transactional documents, not litigation production sets.
• Real-time collaboration document editors (Google Docs competitors). While a collaboration-ws WebSocket service exists (product/collaboration-ws/Dockerfile), the core product is a read/query/analyze system, not a collaborative authoring tool.
• Generic knowledge base / wiki search use cases where documents are not legal or business agreements. The domain-specific extractors (cap table, timeline, due diligence checklist) provide no value outside transactional document contexts.

---

v0.3 Commercial Model — Pricing & Positioning

Positioning Statement

DataZoom is the AI-native deal room for teams that need to understand their documents, not just store them. Where traditional virtual data rooms (Intralinks, Ansarada, Firmex) provide secure file hosting with basic search, DataZoom provides semantic understanding: ask any question about your deal in plain English and receive a cited answer in seconds. Where generic AI tools require copy-pasting documents and cannot reason across a corpus, DataZoom maintains a persistent, org-isolated vector index that grows with every upload.

Monetization Model

Billing Infrastructure

The repository includes a dedicated billing and wallet subsystem (docs/billing_wallet/BILLING_WALLET_IMPLEMENTATION_MASTER.md, docs/billing_wallet/WALLET_ACTION_PLAN.md). This suggests a credit/wallet model for AI query consumption layered atop the subscription tiers — specifically, Modal cloud LLM calls are metered and deducted from a wallet balance, as evidenced by the 06_wallet_deduction.md action plan in the Modal migration optimization series.

Moat Thesis

1. Corpus Statefulness: DataZoom's pgvector index persists and grows across sessions. Each uploaded document increases the semantic retrieval surface. A competitor starting fresh from a document dump cannot match the indexed, relationship-mapped state built over months of use.
2. Domain-Specific Extractors: Cap table reconstruction (/api/cap-table/extract, /api/cap-table/auto-populate), due diligence checklist generation (/api/business-types/[typeKey]/checklist), and party-level analysis (/api/analysis/party) are not achievable with a generic LLM wrapper. Each extractor encodes domain logic validated against real document fixtures (product/lib/cap-table/__tests__/fixture-corpus.ts).
3. Hybrid Deployment: The ability to run the full stack locally (Ollama + Supabase self-hosted) addresses a blocker that eliminates cloud-only competitors from regulated market segments.
4. Citation Enforcement: The citation-system.test.ts test file and BR-002 constraint indicate citation integrity is a hard architectural requirement. Hallucination-prone competitors cannot meet the evidentiary standard required in legal and financial contexts.

---

v0.4 User Journeys

UJ-001 — Document Upload and Indexing

Actor: Legal analyst at a PE firm
Entry point: product/app/(app)/documents/page.tsx → Documents view
Steps:

1. User navigates to the Documents section and selects a folder or creates a new one (product/app/(app)/documents/folder/[id]/page.tsx).
2. User uploads one or more PDFs or documents via the upload interface (PR #101: Upload fixes indicates active work on this flow).
3. The document is stored in Supabase Storage; metadata is written to the documents table (filename, document_type, parties, key_terms, page_count).
4. The async ingest worker (docker/Dockerfile.worker, BullMQ queue via Redis/Upstash) processes the document: extracts full text, generates an LLM summary via Modal or Ollama, and splits the text into chunks.
5. sentence-transformers (all-MiniLM-L6-v2) generates 384-dimensional embeddings for each chunk; chunks are written to document_chunks with their embedding vectors.
6. The ivfflat cosine index on document_chunks.embedding is updated, making the document immediately queryable.
7. User sees the document appear in the Documents list with extracted document_type, effective_date, and parties metadata rendered in product/app/(app)/[type]/[id]/views/document-view.tsx.

Success metric: Document available for semantic query within 60 seconds of upload for documents ≤ 50 pages.

---

UJ-002 — Natural Language Document Query (RAG Chat)

Actor: Founder preparing for investor Q&A
Entry point: product/app/(app)/context/page.tsx → Context / Chat interface
Steps:

1. User opens the Context page and selects or creates a conversation thread via product/app/(app)/context/components/thread-selector.tsx.
2. User types a natural language question (e.g., "Who are the parties to the IP assignment signed in February 2025, and what rights were transferred?").
3. The query is embedded using the same all-MiniLM-L6-v2 model; a vector cosine similarity search against document_chunks retrieves the top-k most relevant chunks.
4. Retrieved chunks are assembled into a prompt context and dispatched to the model router, which selects Modal (Qwen2.5:32B) or local Ollama based on routing rules (product/lib/__tests__/model-router.test.ts).
5. The LLM generates a response grounded in the retrieved chunks; the citation system (product/lib/__tests__/citation-system.test.ts) attaches source references to each factual claim.
6. The response is streamed to product/app/(app)/context/components/conversation-panel.tsx with inline citations linking back to the source document and chunk.
7. Suggested follow-up questions are surfaced via product/app/(app)/context/components/suggested-followups.tsx.
8. The user can save a decision or finding to the decision log (product/app/(app)/context/components/decision-log.tsx, product/app/(app)/context/components/save-decision-form.tsx).
9. AI interaction is tracked via /api/ai/track-interaction and sent to Mixpanel.

Success metric: P95 response latency < 8 seconds; citation present on 100% of factual responses.

---

UJ-003 — Cap Table Extraction and Review

Actor: M&A associate reconstructing target company equity history
Entry point: product/app/(app)/cap-table/page.tsx
Steps:

1. Associate navigates to Cap Table and triggers extraction against already-uploaded equity documents via /api/cap-table/extract.
2. The extraction pipeline (Modal endpoint, triggered via /api/cap-table/auto-populate) parses equity agreements, SAFEs, and stock purchase agreements to identify transactions: issuances, transfers, conversions, option grants.
3. Extracted transaction candidates are written to a review queue; associate sees them in the review interface surfaced via /api/cap-table/review.
4. Associate reviews each candidate transaction: approves via /api/cap-table/review/[id]/approve or rejects via /api/cap-table/review/[id]/reject. Bulk review is also supported.
5. Approved transactions are committed to the cap table ledger. The calculation engine (product/lib/cap-table/__tests__/calc.test.ts) computes ownership percentages and dilution.
6. Associate queries the cap table as of any historical date via /api/cap-table/as-of to reconstruct point-in-time ownership.
7. Current state is available via /api/cap-table/current. Transactions can be voided via /api/cap-table/transactions/[id]/void.
8. Cap table health checks run via /api/cap-table/health to surface reconciliation mismatches.
9. The associate views the rendered cap table in product/app/(app)/cap-table/page.tsx with ownership percentages, transaction history, and parties.

Success metric: Extraction accuracy ≥ 90% against human-verified corpus (product/lib/cap-table/__tests__/fixture-corpus.test.ts).

---

UJ-004 — Due Diligence Checklist Generation

Actor: VC analyst conducting Series B diligence
Entry point: product/app/(app)/analysis/page.tsx → Analysis module
Steps:

1. Analyst selects a business type profile (e.g., SaaS, healthcare, fintech) via /api/business-types and retrieves the associated checklist via /api/business-types/[typeKey]/checklist.
2. DataZoom generates a due diligence checklist tailored to the business type, drawing from the document catalog (docs/action_plans/business_type_fix/REFERENCE_DOCUMENT_CATALOG.md).
3. The system matches checklist items against the documents already uploaded to the org, surfacing coverage gaps (documents present vs. documents expected).
4. The DD match panel (product/app/(app)/[type]/[id]/views/dd-match-panel.tsx) shows which checklist items are satisfied by which documents.
5. Analyst requests regeneration of specific analysis sections via /api/analysis/regenerate or party-level analysis via /api/analysis/party.
6. Strategic options are generated by the Advisor module (/api/advisor/strategic-options), synthesizing findings across the corpus into a structured memo.
7. A risk memo is produced via /api/advisor/risk-memo, compiled from findings logged during review.
8. The analyst exports activity data via /api/activity/export for client reporting.

Success metric: Checklist coverage computation completes within 10 seconds for corpora up to 200 documents.

---

UJ-005 — Timeline Visualization

Actor: Legal counsel reconstructing chronology for a dispute or deal closing
Entry point: product/app/(app)/analysis/overview/page.tsx or analysis strategy view
Steps:

1. Counsel uploads all relevant agreements; the ingest pipeline extracts timeline_events records from each document, populating event_date, event_type (e.g., equity_change, agreement_signed, ip_assignment), description, parties_involved, and impact level.
2. The Timeline view renders events chronologically, filterable by event_type and impact severity.
3. Counsel clicks an event to navigate to the source document chunk, viewing the exact text that triggered the event extraction.
4. Cross-document event correlations (e.g., an IP assignment and a subsequent equity grant to the same party) are surfaced.
5. The full timeline is exportable via the activity export API (/api/activity/export).

Success metric: Timeline events extracted for ≥ 95% of documents containing dates and described events.

---

UJ-006 — Admin Pipeline Monitoring

Actor: Platform administrator or DevOps engineer
Entry point: product/app/(app)/admin/pipeline/page.tsx
Steps:

1. Admin navigates to the Pipeline admin page to monitor the document processing queue.
2. Pipeline health is checked via /api/admin/pipeline/health; cloud worker status (RunPod fleet, pod warmup state: provisioning → warming → ready) is retrieved from /api/admin/pipeline/cloud.
3. Admin views routing statistics (Modal vs. Ollama dispatch ratios, queue depths) via /api/admin/routing-stats.
4. Failed jobs are retried via /api/admin/pipeline/retry.
5. Observability metrics (embedding throughput, LLM call latency, queue backlog) are visible in the pipeline view.

---

v0.5 Red Team Review — Risk Register

---

v0.6 Architecture

TECH-001 — Frontend Application

Path: product/ (Next.js 15 App Router)
Runtime: React 19, TypeScript, Tailwind CSS, shadcn/ui (Radix UI primitives, e.g., @radix-ui/react-context-menu)
Theming: next-themes (dark/light mode)
Routing: App Router with dynamic segments product/app/(app)/[type]/[id]/page.tsx for polymorphic entity rendering (documents, companies, findings, options)
State Management: React Context providers scoped per feature (product/app/(app)/analysis/analysis-provider.tsx, product/app/(app)/documents/documents-provider.tsx, product/app/(app)/activity/activity-provider.tsx)
Deployment: Vercel (.vercelignore present) with Cloudflare edge (PR #111: Cloudflare deployment)

TECH-002 — Backend API Layer

Path: product/app/api/ (50 Next.js API Route handlers)
Auth: Clerk (/api/clerk/proxy/route.ts); organization-scoped via withOrgAuth middleware
Database Client: @supabase/supabase-js ^2.95.3 (SUPABASE_URL, SUPABASE_SERVICE_KEY from .env.services)
Key API Groups:

• Activity: activity/, activity/unified/ (calendar, day, feed, refresh)
• Admin: admin/pipeline/ (cloud, health, retry), admin/routing-stats/
• Advisor: advisor/ (batch, process-queue, risk-memo, strategic-options)
• Cap Table: cap-table/ (as-of, auto-populate, current, extract, health, review, transactions)
• Analysis: analysis/ (party, regenerate)
• Business Types: business-types/, business-type-profiles/
• Clauses: clauses/compare/
• Collaboration: collaboration/token/
• Company Settings: company-settings/, company-settings/linkage-mismatches/

TECH-003 — Database Layer

Engine: PostgreSQL via Supabase
Extension: pgvector for 384-dimensional vector storage and cosine similarity search
Key Tables:

Indexes:

• ivfflat on document_chunks.embedding with vector_cosine_ops (lists=100) for ANN search
• GIN on documents.parties for array containment queries
• B-tree on documents.document_type, documents.effective_date
• B-tree on timeline_events.event_date, timeline_events.event_type
• B-tree on document_chunks.document_id

RLS: Row-Level Security policies enforce org-level isolation using Clerk organization ID as the discriminant.

TECH-004 — AI / ML Pipeline

Embedding Service: sentence-transformers all-MiniLM-L6-v2 (384-dim); runs in the GPU Docker image (docker/Dockerfile.gpu)
LLM (Cloud): Modal deployment of Qwen2.5:32B; dispatched from Next.js API routes via the model router
LLM (Local): Ollama (ollama/ollama:latest), exposed on port 11434; OLLAMA_NUM_PARALLEL=4, OLLAMA_MAX_LOADED_MODELS=3
LLM Proxy: llm service in docker-compose.yml, image ghcr.io/midwestco/datazoom-base:latest, port 8001, proxies to http://ollama:11434
Model Router: Deterministic routing logic tested in product/lib/__tests__/model-router.test.ts; selects Modal vs. Ollama based on query type, org configuration, and availability
Ingest Worker: Python 3.10+ worker (docker/Dockerfile.worker, ghcr.io/midwestco/datazoom-worker:latest); processes BullMQ jobs from Redis/Upstash; performs text extraction, chunking, embedding generation, and database writes
Cloud Worker: docker/cloud-worker/Dockerfile; deployed to Fly.io (docker/cloud-worker/fly.toml); handles cloud-path document processing

TECH-005 — Queue Infrastructure

Queue: BullMQ backed by Redis (Upstash in production, with TLS via ssl_cert_reqs=None fix in commit b718932)
Job Types: Document ingest, re-embedding, batch advisor processing (/api/advisor/batch, /api/advisor/process-queue), cap table extraction
Retry: Failed jobs retried via /api/admin/pipeline/retry; pipeline health monitored via /api/admin/pipeline/health

TECH-006 — Infrastructure and Deployment

Container Registry: ghcr.io/midwestco/ (GitHub Container Registry)
CI/CD: .github/workflows/build-images.yml builds and pushes datazoom-base, datazoom-worker, datazoom-gpu images
Orchestrator: Fly.io (fly/orchestrator/Dockerfile); manages RunPod GPU fleet for cloud inference
GPU Workers: RunPod (pod warmup states: provisioning → warming → ready); status exposed via /api/admin/pipeline/cloud
Collaboration Service: Standalone WebSocket server (product/collaboration-ws/Dockerfile) for real-time co-editing
Networking: WireGuard VPN option for private network connectivity (docker/wireguard/wg0.conf.example)
Docker Compose Profiles:

• full: All services including Ollama, LLM proxy, worker
• gpu: Embed + reranker + Ollama
• worker: Ingest worker only
• infra: Monitoring (future)

API-001 — RAG Query API

Route: Inferred from context module; query dispatched through /api/advisor or context page
Method: POST
Input: { query: string, conversationId: string, orgId: string }
Processing: Embed query → cosine similarity search on document_chunks → retrieve top-k chunks → assemble prompt → dispatch to model router → stream response with citations
Output: { answer: string, citations: [{ documentId, chunkId, chunkContent, filename }] }

API-002 — Cap Table Extract API

Route: /api/cap-table/extract
Method: POST
Input: { documentIds: string[] }
Processing: Dispatch extraction job to Modal endpoint; parse equity transaction details from document full text; write candidates to review queue
Output: { jobId: string, candidateCount: number }
Test Coverage: product/lib/__tests__/cap-table-extract-trigger.test.ts

API-003 — Activity Unified Feed API

Route: /api/activity/unified/feed
Method: GET
Input: { orgId, from, to } (query params)
Output: Paginated activity events from activity_log materialized view; supports calendar view (/api/activity/unified/calendar) and day drill-down (/api/activity/unified/day)

DBT-001 — Activity Materialized Views

Location: docs/activity_page/action_plans/02_create_materialized_views_COMPLETE.md
Purpose: Pre-aggregate activity log records by day and event type to power the Activity calendar and metrics APIs (/api/activity/metrics) without full-table scans on activity_log
Refresh: Triggered via /api/activity/unified/refresh (incremental refresh on new activity inserts)

DBT-002 — Cap Table Snapshot Read Model

Location: docs/cap-table/action_plans/14_snapshot_read_model_optimization.md
Purpose: Maintain point-in-time cap table snapshots to