Documentation source

Cohort question runner

One question across many records, aggregated pattern view

## Problem

Users frequently want to ask one question across many records — "Which of my 84 active patients are stalling on goals?", "Which of these 220 portfolio companies have churn risk above X?", "Score every deal in this view against my new criteria." Today the answer is N manual triggers (one action firing per record) or one long-running agent chat that loses thread halfway through. There is no first-class affordance for "ask the cohort," and no aggregation view of N responses such that a pattern can be seen.

The design exploration "Cohort patterns" (PKG-012, Exp05 in `Amble explorations/e04-06.jsx`) demonstrates the target: a question bar attached to a filtered view, a beeswarm/distribution chart of N answers, an outlier rail with per-record drill-in, and a "pattern across this cohort" summary card. Today none of those surfaces exist as a coherent feature.

## Solution

A "Ask the cohort" question bar attached to any filtered view, saved cohort, or search result. The user types a question (or selects from a templated set) and the platform:

1. Confirms scope and cost — N records × estimated tokens × LLM provider.
2. Creates one `actions` registry row representing the cohort question, with `queue_config` pointing at the view's data source (per ADR-0016).
3. The unified `action-tick` Inngest worker fans out per-record sessions via existing `triggerTask()` machinery.
4. Per-record responses stream back into a `cohort_question` aggregation surface that subscribes to all child sessions, computing rolling aggregates as answers arrive.
5. The aggregation view renders multiple visualizations (distribution, cluster, trajectory, table) over the same response set, surfaces outliers, and offers "save as cohort," "ask follow-up," "capture as context."

The user experience is one question → one waiting state → one aggregated answer, even though under the hood N agent sessions ran. The mechanism is a thin orchestration layer over primitives that already exist (`actions`, `queue_config`, `sessions`, `submitResponse`); the new surface is the question bar plus the aggregation panel.

## Design

### Architecture

```
filtered-view / saved-cohort / search-result
        ↓ user types question
[CohortQuestionBar]
        ↓ confirm scope + cost
POST /api/cohort-questions  (creates actions row, queue_config)
        ↓
action-tick worker (ADR-0016 fan-out)
        ↓ per-record dispatch
N × triggerTask() → N × session (agent_session)
        ↓ each session calls submitResponse for one record
        ↓
[CohortResponseAggregation] subscribes to:
  - sessions WHERE parent_action = <cohort question action>
  - entity_responses WHERE source = <this cohort question>
        ↓
renders: distribution / cluster / trajectory / table
        + outlier rail (per-record drill-in)
        + pattern summary (computed across responses)
```

Three new components, one extension, one orchestration helper:

- `features/views/components/cohort-question-bar.tsx` (planned) — attached to the view header when a view is filtered or a cohort is saved. Templated chip-based question builder ("Which records are X for more than Y?") plus freeform text fallback.
- `features/views/components/cohort-response-aggregation.tsx` (planned) — subscribes via React Query + realtime to all child sessions, computes rolling aggregates client-side for small N (≤ 500) or server-side via `features/sessions/server/aggregate.ts` (planned) for larger N.
- `features/sessions/server/aggregate.ts` (planned) — server helper that groups `entity_responses` by `source_action_id`, computes histograms / clusters / outliers, returns a stable shape for the aggregation view.
- `features/actions/server/dispatch.ts` (extension) — accepts an "ad-hoc cohort question" payload shape, materializes the `actions` row + `queue_config`, returns the new action ID for subscribers.

### API contract

`POST /api/cohort-questions`

```ts
{
  viewId?: string;              // pre-existing view (source of cohort)
  filter?: EntityFilterPayload; // ad-hoc filter (alt to viewId)
  question: string;             // freeform user text
  targetField?: string;         // if structured (per-record submits this field)
  responseSchema?: ZodSchema;   // optional structured response shape
  agentSlug?: string;           // optional; defaults to tenant default cohort agent
  confirmCost: { records: number; estTokens: number; ack: true };
}
→ { actionId: string; estimatedCompletionMs: number }
```

The action row carries `queue_config.dataSource = { kind: "view", viewId }` (or the inline filter equivalent). Per-record sessions are created lazily by `action-tick`, not eagerly at request time, so submission stays fast.

### Aggregation surface

The aggregation view renders four tabs over the same response set (matching the design exploration):

| Tab | What it shows |
|---|---|
| Distribution | Beeswarm / histogram for one numeric or categorical dimension |
| Cluster | Free-text responses grouped (simple `value` exact-match for v1; embeddings clustering deferred) |
| Trajectory | Time-series of per-record answer over repeated cohort questions (deferred to v2) |
| Table | Raw `(record, response, confidence, source_session)` list, sortable |

The outlier rail (right side, per the design ref) lists the top-K records by a "concern" score derived from `response.score` and time-since-last-progress. Each outlier opens the per-record session transcript.

### Permission + RLS

The cohort-question action inherits the caller's tenant + workspace. Per-record sessions are dispatched in the caller's context — RLS on `entities` automatically scopes the fan-out to records the caller can see. Records the caller cannot read are silently absent from the cohort (existing view semantics). No new permission gates; reuse `actions.run` for the question and the agent's `entities.*.read` for per-record access.

## Trade-offs

- **Cost gating is a hard requirement.** A naive "Ask the cohort" button on an 84-record view = ~84 LLM calls. The confirm-before-run dialog with record count × estimated tokens is non-negotiable. v1 caps cohort size at 500 records hard; larger cohorts require an admin permission and async-only mode.
- **Latency vs rate limits.** ADR-0016's `queue_config` already provides per-action concurrency caps; cohort questions reuse this. Default concurrency = 5; configurable per tenant. Surface streaming aggregate updates as responses arrive — never block on "all 84 done."
- **Pattern detection quality.** v1 uses naive grouping (string exact-match for categorical, bucketing for numeric). Embeddings-based clustering of free-text is a v2 follow-up — adds infrastructure dependency (vector store) and per-cohort embedding cost.
- **Reusing `actions` vs new primitive.** The cohort question is a one-shot fan-out, not a persistent scheduled action. We could invent a `cohort_question_runs` table. We don't — per `.claude/rules/no-parallel-systems.md`, `actions` already supports one-shot triggers (`trigger_type = "manual"`), and `queue_config` already handles fan-out. The trade-off is that the `actions` admin UI will show cohort-question rows alongside scheduled actions; the lifecycle field (`completed` once fan-out drains) and a `kind` discriminator distinguish them.
- **Aggregation lives in the view, not the action.** The `actions` row is plumbing; the aggregation panel is its own surface that any caller (chat, dashboard, embed) can subscribe to by `actionId`. This means the aggregation view is reusable for any action whose `queue_config` produces N responses, not just cohort questions.
- **"Save as cohort" creates a saved view, not a new primitive.** Per design ref: clicking "Save as cohort" persists the current filter as a `views` row with the question annotated. No new "cohort" table.

## Acceptance Criteria

- A user on any filtered entity view sees an "Ask the cohort" affordance in the view header.
- Clicking it opens a question bar (templated chips + freeform text + target-field selector).
- Submitting shows a cost-confirmation dialog with N records, estimated tokens, estimated wall-clock.
- On confirm, one `actions` row is created with `queue_config` referencing the current view's data source.
- The unified `action-tick` worker fans out per-record sessions (ADR-0016 path), respecting tenant concurrency caps.
- Per-record sessions call `submitResponse` against the question's target field (or write to an ad-hoc response store when no target field is set).
- The aggregation view subscribes via React Query + realtime and updates as responses arrive — no "blank until all done" state.
- The aggregation view renders Distribution and Table tabs in v1 (Cluster + Trajectory deferred).
- The outlier rail lists top-6 records by concern score with per-record drill-in to the source session transcript.
- The pattern summary card shows a one-sentence agent-generated synthesis once ≥ 80% of responses have returned.
- "Save as cohort" persists the current filter as a `views` row with the question text annotated.
- Hard limit: cohort size ≤ 500 records without admin override.
- All per-record sessions and the parent `actions` row are visible in `/tasks` (existing surface) so the user has a fallback view if the aggregation panel hangs.
- All RLS scoping is by existing `entities` policies; no new permission gates required.

## Files

| File | Role |
|---|---|
| `features/actions/server/dispatch.ts` (extension) | Accept ad-hoc cohort-question payload, materialize `actions` row + `queue_config`, return action ID |
| `features/sessions/server/aggregate.ts` (planned) | Server helper grouping `entity_responses` by `source_action_id`, computing histograms / clusters / outliers |
| `features/views/components/cohort-question-bar.tsx` (planned) | Question-builder UI attached to filtered view headers; templated chips + freeform + target-field selector |
| `features/views/components/cohort-response-aggregation.tsx` (planned) | Aggregation panel subscribing to all child sessions; renders Distribution / Cluster / Trajectory / Table tabs + outlier rail + pattern summary |
| `app/api/cohort-questions/route.ts` (planned) | Thin route handler; Zod-validates payload, dispatches to `dispatch.ts`, returns action ID |
| `content/docs/features/cohort-questions.mdx` (planned, target-doc) | Living feature doc, populated by documentarian on first completed implementation |

## References

- Design exploration: PKG-012 "Amble explorations" Exp05 — `~/SprinterVault/20-Ventures/Amble/03-Product/design-references/claude-design-docs-20260519/extracted/Amble explorations/e04-06.jsx` lines 249-377 (question bar, beeswarm distribution, outlier rail, pattern summary).
- ADR-0016 — Action queues unify orchestration. `queue_config` fan-out is the load-bearing primitive this spec reuses; no new orchestration engine required.
- ADR-0023 — Action vs session executor boundary. Per-record sessions go through `features/sessions/server/executor.ts` (`executeAgentSession`), not a new path.
- ADR-0004 — Task entity and action registry. Cohort questions are `actions` registry rows; the per-record work units are sessions.
- Related work: `documents/work/2026-05-19-docs-intake-canonical-submit/` — `submitResponse` plumbing this spec depends on; cohort questions are the multi-record consumer.
- Rules: `.claude/rules/no-parallel-systems.md` — why we extend `actions` + `queue_config` rather than build a `cohort_question_runs` table. `.claude/rules/core-loop.md` — sessions / actions / tasks primitive boundary.

## Problem Users frequently want to ask one question across many records — "Which of my 84 active patients are stalling on goals?", "Which of these 220 portfolio companies have churn risk above X?", "Score every deal in this view against my new criteria." Today the answer is N manual triggers (one action firing per record) or one long-running agent chat that loses thread halfway through. There is no first-class affordance for "ask the cohort," and no aggregation view of N responses such that a pattern can be seen. The design exploration "Cohort patterns" (PKG-012, Exp05 in `Amble explorations/e04-06.jsx`) demonstrates the target: a question bar attached to a filtered view, a beeswarm/distribution chart of N answers, an outlier rail with per-record drill-in, and a "pattern across this cohort" summary card. Today none of those surfaces exist as a coherent feature. ## Solution A "Ask the cohort" question bar attached to any filtered view, saved cohort, or search result. The user types a question (or selects from a templated set) and the platform: 1. Confirms scope and cost — N records × estimated tokens × LLM provider. 2. Creates one `actions` registry row representing the cohort question, with `queue_config` pointing at the view's data source (per ADR-0016). 3. The unified `action-tick` Inngest worker fans out per-record sessions via existing `triggerTask()` machinery. 4. Per-record responses stream back into a `cohort_question` aggregation surface that subscribes to all child sessions, computing rolling aggregates as answers arrive. 5. The aggregation view renders multiple visualizations (distribution, cluster, trajectory, table) over the same response set, surfaces outliers, and offers "save as cohort," "ask follow-up," "capture as context." The user experience is one question → one waiting state → one aggregated answer, even though under the hood N agent sessions ran. The mechanism is a thin orchestration layer over primitives that already exist (`actions`, `queue_config`, `sessions`, `submitResponse`); the new surface is the question bar plus the aggregation panel. ## Design ### Architecture ``` filtered-view / saved-cohort / search-result ↓ user types question [CohortQuestionBar] ↓ confirm scope + cost POST /api/cohort-questions (creates actions row, queue_config) ↓ action-tick worker (ADR-0016 fan-out) ↓ per-record dispatch N × triggerTask() → N × session (agent_session) ↓ each session calls submitResponse for one record ↓ [CohortResponseAggregation] subscribes to: - sessions WHERE parent_action = <cohort question action> - entity_responses WHERE source = <this cohort question> ↓ renders: distribution / cluster / trajectory / table + outlier rail (per-record drill-in) + pattern summary (computed across responses) ``` Three new components, one extension, one orchestration helper: - `features/views/components/cohort-question-bar.tsx` (planned) — attached to the view header when a view is filtered or a cohort is saved. Templated chip-based question builder ("Which records are X for more than Y?") plus freeform text fallback. - `features/views/components/cohort-response-aggregation.tsx` (planned) — subscribes via React Query + realtime to all child sessions, computes rolling aggregates client-side for small N (≤ 500) or server-side via `features/sessions/server/aggregate.ts` (planned) for larger N. - `features/sessions/server/aggregate.ts` (planned) — server helper that groups `entity_responses` by `source_action_id`, computes histograms / clusters / outliers, returns a stable shape for the aggregation view. - `features/actions/server/dispatch.ts` (extension) — accepts an "ad-hoc cohort question" payload shape, materializes the `actions` row + `queue_config`, returns the new action ID for subscribers. ### API contract `POST /api/cohort-questions` ```ts { viewId?: string; // pre-existing view (source of cohort) filter?: EntityFilterPayload; // ad-hoc filter (alt to viewId) question: string; // freeform user text targetField?: string; // if structured (per-record submits this field) responseSchema?: ZodSchema; // optional structured response shape agentSlug?: string; // optional; defaults to tenant default cohort agent confirmCost: { records: number; estTokens: number; ack: true }; } → { actionId: string; estimatedCompletionMs: number } ``` The action row carries `queue_config.dataSource = { kind: "view", viewId }` (or the inline filter equivalent). Per-record sessions are created lazily by `action-tick`, not eagerly at request time, so submission stays fast. ### Aggregation surface The aggregation view renders four tabs over the same response set (matching the design exploration): | Tab | What it shows | |---|---| | Distribution | Beeswarm / histogram for one numeric or categorical dimension | | Cluster | Free-text responses grouped (simple `value` exact-match for v1; embeddings clustering deferred) | | Trajectory | Time-series of per-record answer over repeated cohort questions (deferred to v2) | | Table | Raw `(record, response, confidence, source_session)` list, sortable | The outlier rail (right side, per the design ref) lists the top-K records by a "concern" score derived from `response.score` and time-since-last-progress. Each outlier opens the per-record session transcript. ### Permission + RLS The cohort-question action inherits the caller's tenant + workspace. Per-record sessions are dispatched in the caller's context — RLS on `entities` automatically scopes the fan-out to records the caller can see. Records the caller cannot read are silently absent from the cohort (existing view semantics). No new permission gates; reuse `actions.run` for the question and the agent's `entities.*.read` for per-record access. ## Trade-offs - **Cost gating is a hard requirement.** A naive "Ask the cohort" button on an 84-record view = ~84 LLM calls. The confirm-before-run dialog with record count × estimated tokens is non-negotiable. v1 caps cohort size at 500 records hard; larger cohorts require an admin permission and async-only mode. - **Latency vs rate limits.** ADR-0016's `queue_config` already provides per-action concurrency caps; cohort questions reuse this. Default concurrency = 5; configurable per tenant. Surface streaming aggregate updates as responses arrive — never block on "all 84 done." - **Pattern detection quality.** v1 uses naive grouping (string exact-match for categorical, bucketing for numeric). Embeddings-based clustering of free-text is a v2 follow-up — adds infrastructure dependency (vector store) and per-cohort embedding cost. - **Reusing `actions` vs new primitive.** The cohort question is a one-shot fan-out, not a persistent scheduled action. We could invent a `cohort_question_runs` table. We don't — per `.claude/rules/no-parallel-systems.md`, `actions` already supports one-shot triggers (`trigger_type = "manual"`), and `queue_config` already handles fan-out. The trade-off is that the `actions` admin UI will show cohort-question rows alongside scheduled actions; the lifecycle field (`completed` once fan-out drains) and a `kind` discriminator distinguish them. - **Aggregation lives in the view, not the action.** The `actions` row is plumbing; the aggregation panel is its own surface that any caller (chat, dashboard, embed) can subscribe to by `actionId`. This means the aggregation view is reusable for any action whose `queue_config` produces N responses, not just cohort questions. - **"Save as cohort" creates a saved view, not a new primitive.** Per design ref: clicking "Save as cohort" persists the current filter as a `views` row with the question annotated. No new "cohort" table. ## Acceptance Criteria - A user on any filtered entity view sees an "Ask the cohort" affordance in the view header. - Clicking it opens a question bar (templated chips + freeform text + target-field selector). - Submitting shows a cost-confirmation dialog with N records, estimated tokens, estimated wall-clock. - On confirm, one `actions` row is created with `queue_config` referencing the current view's data source. - The unified `action-tick` worker fans out per-record sessions (ADR-0016 path), respecting tenant concurrency caps. - Per-record sessions call `submitResponse` against the question's target field (or write to an ad-hoc response store when no target field is set). - The aggregation view subscribes via React Query + realtime and updates as responses arrive — no "blank until all done" state. - The aggregation view renders Distribution and Table tabs in v1 (Cluster + Trajectory deferred). - The outlier rail lists top-6 records by concern score with per-record drill-in to the source session transcript. - The pattern summary card shows a one-sentence agent-generated synthesis once ≥ 80% of responses have returned. - "Save as cohort" persists the current filter as a `views` row with the question text annotated. - Hard limit: cohort size ≤ 500 records without admin override. - All per-record sessions and the parent `actions` row are visible in `/tasks` (existing surface) so the user has a fallback view if the aggregation panel hangs. - All RLS scoping is by existing `entities` policies; no new permission gates required. ## Files | File | Role | |---|---| | `features/actions/server/dispatch.ts` (extension) | Accept ad-hoc cohort-question payload, materialize `actions` row + `queue_config`, return action ID | | `features/sessions/server/aggregate.ts` (planned) | Server helper grouping `entity_responses` by `source_action_id`, computing histograms / clusters / outliers | | `features/views/components/cohort-question-bar.tsx` (planned) | Question-builder UI attached to filtered view headers; templated chips + freeform + target-field selector | | `features/views/components/cohort-response-aggregation.tsx` (planned) | Aggregation panel subscribing to all child sessions; renders Distribution / Cluster / Trajectory / Table tabs + outlier rail + pattern summary | | `app/api/cohort-questions/route.ts` (planned) | Thin route handler; Zod-validates payload, dispatches to `dispatch.ts`, returns action ID | | `content/docs/features/cohort-questions.mdx` (planned, target-doc) | Living feature doc, populated by documentarian on first completed implementation | ## References - Design exploration: PKG-012 "Amble explorations" Exp05 — `~/SprinterVault/20-Ventures/Amble/03-Product/design-references/claude-design-docs-20260519/extracted/Amble explorations/e04-06.jsx` lines 249-377 (question bar, beeswarm distribution, outlier rail, pattern summary). - ADR-0016 — Action queues unify orchestration. `queue_config` fan-out is the load-bearing primitive this spec reuses; no new orchestration engine required. - ADR-0023 — Action vs session executor boundary. Per-record sessions go through `features/sessions/server/executor.ts` (`executeAgentSession`), not a new path. - ADR-0004 — Task entity and action registry. Cohort questions are `actions` registry rows; the per-record work units are sessions. - Related work: `documents/work/2026-05-19-docs-intake-canonical-submit/` — `submitResponse` plumbing this spec depends on; cohort questions are the multi-record consumer. - Rules: `.claude/rules/no-parallel-systems.md` — why we extend `actions` + `queue_config` rather than build a `cohort_question_runs` table. `.claude/rules/core-loop.md` — sessions / actions / tasks primitive boundary.