DataZoom Operator Console

Document ID: OPS-001
Version: 1.0
Classification: Internal — Operations
Audience: Platform engineers, SRE, support engineers, on-call responders

---

Table of Contents

1. System Overview
2. Admin Dashboard & Routes
3. Internal Tooling Inventory
4. Operational Workflows
5. Monitoring & Alerting
6. Support Workflows
7. Environment Management
8. Secret Management
9. Health Check Endpoints & Status

---

1. System Overview

DataZoom (midwestco/datazoom) is an AI-powered document analysis platform built on Next.js 15 (App Router), Supabase (PostgreSQL + pgvector), Clerk multi-tenant authentication, and a distributed Python worker fleet for document ingestion and LLM inference. The operational surface spans three distinct compute tiers: the Next.js web application, the Docker-based worker fleet (embedding, reranking, ingest), and the cloud inference layer (Modal/RunPod via orchestrator on Fly.io).

Production topology:

---

2. Admin Dashboard & Routes

2.1 Admin UI Pages

The admin surface is scoped under the (app) route group and protected by Clerk organization authentication. The primary operator page is:

2.2 Admin API Routes

All admin API routes live under /api/admin/ and require Clerk authentication with organization-scoped tokens. The cloud proxy endpoint uses Clerk auth directly (not withOrgAuth) per commit d037000.

2.3 Supporting Operational Routes

These routes are used by operators for data inspection and corrective actions:

---

3. Internal Tooling Inventory

3.1 Shell Scripts

3.2 Docker Compose Profiles

The docker-compose.yml supports four named profiles for selective service startup:

docker-compose.module1.yml and docker/docker-compose.module1-ports.yml provide port-exposed variants for module 1 development.

3.3 CI/CD Tooling

Located in .github/workflows/:

Image registry: ghcr.io/midwestco/

ghcr.io/midwestco/datazoom-base:latest
ghcr.io/midwestco/datazoom-worker:latest
ghcr.io/midwestco/datazoom-gpu:latest

Cloud worker image is built directly on GitHub Actions and pushed to ghcr.io (commit 07c91b9).

3.4 Database Utilities (SQL Scripts)

Operational SQL scripts are maintained under docs/archive/ for manual execution against Supabase:

3.5 Git Hooks

3.6 Queue Infrastructure

Document processing jobs are managed via BullMQ backed by Upstash Redis. Workers connect using redis.asyncio with ssl_cert_reqs=None for Upstash TLS compatibility (fixed in commit b718932). The cloud worker entrypoint initializes this connection on container start.

---

4. Operational Workflows

4.1 Deployment

Next.js Application (Vercel)

1. Merge pull request to main branch.
2. Vercel auto-deploys on merge. Preview deployments are generated for all PRs.
3. .vercelignore controls which files are excluded from the deployment bundle.
4. Environment variables must be configured in the Vercel project dashboard before deployment (see Section 8).

Docker Worker Fleet

# 1. Build updated images (CI handles this automatically on push)
docker pull ghcr.io/midwestco/datazoom-base:latest
docker pull ghcr.io/midwestco/datazoom-worker:latest
docker pull ghcr.io/midwestco/datazoom-gpu:latest

# 2. Deploy with appropriate profile
COMPOSE_PROFILES=full docker compose up -d

# 3. Verify health
docker compose ps
curl http://localhost:8001/health   # LLM proxy
curl http://localhost:11434/api/tags  # Ollama

Cloud Worker (Fly.io)

The cloud worker is deployed to Fly.io using the config at docker/cloud-worker/fly.toml. Orchestrator is deployed separately from fly/orchestrator/Dockerfile.

# Deploy orchestrator
cd fly/orchestrator
fly deploy

# Deploy cloud worker
cd docker/cloud-worker
fly deploy

Note: Orchestrator requires 2048 MB memory for performance CPU (see commit 4e7cd14).

4.2 Rollback

Vercel Rollback

1. Navigate to Vercel project → Deployments.
2. Identify the last known-good deployment.
3. Click Promote to Production.
4. Verify health check endpoints respond (see Section 9).

Docker Worker Rollback

# Pull previous image by digest or tag
docker pull ghcr.io/midwestco/datazoom-worker:<previous-sha>

# Update docker-compose.yml image tag, then redeploy
COMPOSE_PROFILES=worker docker compose up -d

# Confirm old container is replaced
docker compose ps

Database Migration Rollback

DataZoom uses Supabase migrations. If a migration must be reversed:

1. Connect to Supabase Studio or psql using the service key.
2. Execute the inverse SQL manually (no automated down-migration scripts are present in the repository).
3. Update the migration tracking table as appropriate.

4.3 Feature Flags

Feature flags are controlled via the Supabase company_settings table and the cap table feature gate system documented in product/lib/cap-table/__tests__/feature-gate.test.ts. The cap table feature is the primary gated feature as of the current version.

Cap Table Feature Gate:

• Gate is evaluated per organization.
• Operators can enable/disable via the company_settings record for a given org.
• The API endpoint GET /api/company-settings returns the current gate state.
• POST /api/company-settings updates settings including feature flag state.

Rollout procedure (per docs/cap-table/action_plans/12_rollout_feature_flag_and_rollback.md):

1. Enable for internal test org first.
2. Monitor GET /api/cap-table/health for data integrity issues.
3. Expand to pilot customer orgs.
4. Full rollout by toggling the default in company_settings.

To disable a feature for a specific org:

UPDATE company_settings
SET cap_table_enabled = false
WHERE org_id = '<clerk-org-id>';

4.4 User Management

User and organization management is handled through Clerk. DataZoom does not maintain a separate user table for auth — Clerk is the system of record.

Common operator actions:

Multi-tenant isolation: All data operations are scoped by Clerk organization ID. The withOrgAuth middleware enforces this at the API layer. The cloud proxy endpoint is an exception and uses Clerk auth directly without withOrgAuth (commit d037000).

---

5. Monitoring & Alerting

5.1 Observability Stack

5.2 What Is Observed

Pipeline health:

• Job queue depth (BullMQ / Upstash Redis)
• Worker pod state: provisioning → warming → ready (pod warmup tracking added in commit 580b80b)
• Cloud inference routing decisions: local vs. Modal/RunPod dispatch (visible at GET /api/admin/routing-stats)
• RunPod fleet status derived from orchestrator cloudStatus (not stale serverless endpoint — commit 088f7b4)

Document processing:

• Embedding coverage across document_chunks (via check_embedding_status.sql)
• Failed and stuck jobs visible at GET /api/admin/pipeline
• Extraction candidate review queue depth at GET /api/cap-table/review

Activity tracking:

• User actions logged to activity log table; materialized views refresh via POST /api/activity/unified/refresh
• Daily metrics accessible at GET /api/activity/metrics

Cap table integrity:

• GET /api/cap-table/health returns data integrity status
• GET /api/company-settings/linkage-mismatches surfaces document-to-company linkage errors

5.3 Ollama Service Health

Ollama is health-checked by Docker Compose:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
  interval: 30s
  timeout: 10s
  retries: 5
  start_period: 30s

If Ollama fails 5 consecutive checks (5 minutes), Docker will mark the container unhealthy. The LLM proxy on port 8001 depends on this service.

5.4 Alerting Escalation Path

DataZoom does not yet have a formalized PagerDuty or OpsGenie integration in the repository. Escalation is currently manual:

---

6. Support Workflows

6.1 Ticket Triage

All inbound support issues should be categorized on receipt:

6.2 Common Issues and Resolution Playbooks

---

ISSUE-001: Document stuck in processing state

Symptoms: Document uploaded but no chunks appear in document_chunks; no embedding visible.

Resolution:

1. Check GET /api/admin/pipeline for the job in the queue.
2. If job is in failed state, use POST /api/admin/pipeline/retry with the job ID.
3. If retry fails, inspect worker container logs: docker compose logs worker --tail=100.
4. Verify Redis/BullMQ connection: confirm Upstash TLS credentials in .env.services.
5. If embedding service is unhealthy, restart GPU profile: COMPOSE_PROFILES=gpu docker compose restart.

---

ISSUE-002: AI chat returning no results / empty context

Symptoms: Chat responds with no citations; RAG retrieval returns zero chunks.

Resolution:

1. Run check_embedding_status.sql against Supabase to confirm document_chunks.embedding is non-null for the document.
2. If embeddings are missing, the ingest worker did not complete — follow ISSUE-001 steps.
3. If embeddings are present, verify the ivfflat index is not corrupted: REINDEX INDEX CONCURRENTLY document_chunks_embedding_idx;
4. Test vector search directly in Supabase SQL editor with a sample query vector.
5. If search is operational but chat fails, check Modal/Ollama routing via GET /api/admin/routing-stats.

---

ISSUE-003: Cloud inference unavailable (Modal/RunPod)

Symptoms: POST /api/admin/pipeline/cloud returns errors; advisor queue stalls.

Resolution:

1. Check GET /api/admin/pipeline/health — inspect cloudStatus field.
2. Verify orchestrator on Fly.io is running: fly status -a <orchestrator-app-name>.
3. Check pod warmup state: pods cycle through provisioning → warming → ready (commit 580b80b). Allow up to 5 minutes for cold start.
4. If RunPod is the issue, confirm RunPod API key in secrets is valid (see Section 8).
5. As fallback, ensure local Ollama is running (COMPOSE_PROFILES=gpu docker compose up -d) — the model router will fall back to local inference.
6. Monitor model routing via GET /api/admin/routing-stats to confirm fallback is active.

---

ISSUE-004: Cap table shows incorrect ownership

Symptoms: Org reports wrong percentages or missing shareholders in cap table view.

Resolution:

1. Review pending extraction candidates at GET /api/cap-table/review.
2. Approve correct candidates via POST /api/cap-table/review/{id}/approve.
3. Reject incorrect candidates via POST /api/cap-table/review/{id}/reject.
4. If a confirmed transaction is wrong, void it: POST /api/cap-table/transactions/{id}/void.
5. Re-run extraction if source document was recently updated: POST /api/cap-table/extract with the document ID.
6. Check for linkage mismatches: GET /api/company-settings/linkage-mismatches.
7. Verify calculation engine output using the test fixtures in product/lib/cap-table/__tests__/calc.test.ts as reference.

---

ISSUE-005: Activity feed not updating

Symptoms: Activity page (/activity) shows stale data; recent actions not reflected.

Resolution:

1. Force refresh materialized views: POST /api/activity/unified/refresh.
2. If feed is still stale, verify delete logging is functioning: run docs/activity_page/VERIFY_DELETE_LOGGING.sql.
3. Check that the activity_log table is being written to by inspecting recent rows in Supabase.
4. Review calendar and day views specifically: GET /api/activity/unified/calendar and GET /api/activity/unified/day.

---

ISSUE-006: Collaboration WebSocket disconnects

Symptoms: Real-time document collaboration drops; users see stale state.

Resolution:

1. Check the collaboration WebSocket service: docker compose logs collaboration-ws --tail=50.
2. Verify the @tiptap/y-tiptap dependency is present (PR #112 added this fix).
3. Check GET /api/collaboration/token is returning valid tokens.
4. Restart the service: docker compose restart collaboration-ws.

---

6.3 Regression Testing

The smoke regression suite at product/lib/__tests__/app-smoke-regression.test.ts should be run after any production incident resolution to confirm system integrity:

cd product
npm test -- app-smoke-regression

---

7. Environment Management

7.1 Environment Configuration

DataZoom operates across three environments. Configuration is driven by environment variables in .env.services (worker fleet) and Vercel project settings (Next.js application).

7.2 Environment Variables

The canonical template is .env.services.example. Operators must copy this to .env.services before starting the worker fleet.

Required variables:

Supabase local development:

supabase start   # starts local Postgres + pgvector + Auth + Storage
supabase stop    # stop local stack
supabase db reset  # reset schema to latest migrations

7.3 Docker Image Configuration

Images are sourced from ghcr.io/midwestco/ in production. For local development, images can be built directly:

docker build -f docker/Dockerfile.base -t datazoom-base .
docker build -f docker/Dockerfile.worker -t datazoom-worker .
docker build -f docker/Dockerfile.gpu -t datazoom-gpu .

The docker-compose.yml OLLAMA_NUM_PARALLEL: "4" and OLLAMA_MAX_LOADED_MODELS: "3" settings should be tuned to the available host hardware in production.

7.4 WireGuard VPN (Worker Networking)

The worker fleet can be configured to communicate over WireGuard for secure inter-service networking. Configuration template is at docker/wireguard/wg0.conf.example. See docker/wireguard/README.md for setup instructions.

---

8. Secret Management

8.1 Secret Inventory

8.2 Secret Storage by Environment

8.3 Rotation Procedures

Rotating SUPABASE_SERVICE_KEY (SEC-002):

1. Generate new service role key in Supabase Dashboard → Settings → API.
2. Update Vercel environment variable (triggers re-deployment).
3. Update .env.services on all worker hosts.
4. Restart worker fleet: COMPOSE_PROFILES=full docker compose down && COMPOSE_PROFILES=full docker compose up -d.
5. Confirm GET /api/admin/pipeline/health returns healthy.

Rotating CLERK_SECRET_KEY (SEC-003):

1. Generate new secret key in Clerk Dashboard → API Keys.
2. Update Vercel environment variable.
3. Vercel will redeploy automatically.
4. Revoke the old key in Clerk Dashboard only after confirming new deployment is healthy.

Rotating Fly.io secrets:

fly secrets set RUNPOD_API_KEY=<new-key> -a <app-name>
fly secrets set MODAL_TOKEN_ID=<id> MODAL_TOKEN_SECRET=<secret> -a <app-name>

Fly.io automatically restarts the application after secret update.

8.4 Security Notes

• .env.services.example is the only secrets-related file committed to the repository. The actual .env.services file must never be committed.
• The .claude/settings.local.json and .mcp.json files present in the repository should be reviewed to ensure no credentials are embedded before any repository access is shared externally.
• SUPABASE_SERVICE_KEY bypasses Row Level Security. Its use must be restricted to server-side API routes and workers only — never exposed to the client.

---

9. Health Check Endpoints & Status

9.1 API Health Endpoints

9.2 Infrastructure Health Checks

Docker Compose service health:

docker compose ps                    # shows health status for all services
docker compose logs ollama --tail=20  # Ollama logs
docker inspect datazoom_ollama_1 --format='{{.State.Health.Status}}'

Supabase local stack:

supabase status   # shows all local Supabase services and their URLs

Fly.io orchestrator:

fly status -a <orchestrator-app-name>
fly logs -a <orchestrator-app-name>

9.3 Vercel Deployment Status

Vercel deployment status is accessible at the Vercel project dashboard. DataZoom does not maintain a public status page as of the current version. The README badge [![Status](https://img.shields.io/badge/status-production-green)]() is static and not connected to a live status endpoint.

Recommended operator checks after any deployment:

1. GET /api/admin/pipeline/health — confirms backend workers are reachable from the Next.js layer.
2. GET /api/cap-table/health — confirms database read path is intact.
3. GET /api/activity/metrics — confirms Supabase query path for aggregated data.
4. GET /api/admin/routing-stats — confirms model routing is operating and recording decisions.
5. Execute smoke regression: cd product && npm test -- app-smoke-regression.

9.4 Queue Health (BullMQ / Upstash)

There is no dedicated BullMQ dashboard endpoint in the repository. Queue state is observable through:

• GET /api/admin/pipeline — returns queue snapshot including failed, waiting, and active job counts.
• Upstash Redis console — direct inspection of queue keys if credentials are available.

Jobs that remain in active state for >10 minutes without completion should be considered stuck and retried via POST /api/admin/pipeline/retry.

---

Last updated: derived from repository state as of commits through 960f88c. Maintain this document in sync with changes to admin routes, Docker infrastructure, and secret requirements.