Documentation source
Documentation source
Documentation source
Topic-driven signal collection, agent triage, and promotion-gated assessment — with an allowlist activation fence that keeps the lens closed by default for every tenant.
## Overview
Trend Intelligence turns a stream of external source material into an evidence-ranked feed of **signals** — source items that an agent has read, scored against the tenant's declared topics, and explained. It answers "what changed in the world that this tenant should care about today, and why."
The module is deliberately split in two:
- **Shared builders** in `features/custom/tenants/lib/` return pure declaration data (criteria sets, agents, cron actions, field extensions, views). They perform no registry writes, no I/O, and no activation.
- **Tenant declarations** spread those builder outputs into a tenant's declaration arrays, supplying only operating parameters and tenant-owned topic/source data.
Sprinter is the origin tenant and the only activated one. Its feed renders at the custom page `/t/sprinter/p/signals`.
The most important thing to understand about this module is that **shipping the code does not turn it on**. Declarations are fenced behind an allowlist, and every cron action ships `status: "paused"`. Activation is a separate, reviewed decision. See [Activation Model](#activation-model).
## Key Concepts
### Topic
A `topic` is the tenant's standing question — the thing it wants watched. Topics are entities of the platform `topic` system type, carrying a definition, inclusion and exclusion criteria, examples, priority, horizon, geography, cadence, keywords, calibration notes, and a status:
```typescript
status: "active" | "paused" | "muted";
```
A topic's status is the reader-facing control. `muted` is enforced structurally in the UI, not merely styled — see [Structural muting](#structural-muting).
### Signal
A signal is not its own entity type. A signal is a `source-item` that carries a **promoted** signal assessment. The `topic-intelligence` builder extends the `source-item` type with the promoted assessment fields:
`confidence`, `topics`, `relevance`, `novelty`, `credibility`, `why_this_matters`, `suggested_action`, plus two relation fields — `relevant_to` (topics) and `evidences` (trends).
`suggested_action` is a closed enum, and every option is deliberately side-effect free:
```typescript
["none", "watch", "investigate", "create-idea", "link-trend"];
```
### Criteria sets
Two agent-scored assessment criteria sets carry the scoring contract. Both are declared with `promotion_mode: "auto"` and governance lane `topic-intelligence` version 1.
- **`signal-assessment`** — entity-scoped on `source-item`. Dimensions: `confidence` (0–1), `matched_topics`, `relevance` (1–10, weight 0.35), `novelty` (1–10, weight 0.2), `credibility` (1–10, weight 0.2), `why_this_matters`, `suggested_action`. Its `display_config.promotion_policy` is the gate:
```typescript
{ mode: "if_confident", confidence_threshold: 0.7, below_threshold: "park-for-review" }
```
- **`signal-relevance`** — relation-scoped (`{ type: "relation", fieldKey: "relevant_to" }`), one score per item-to-topic edge. Dimensions: `relevance` (1–10, the sole weighted dimension), `stance` (`supports` / `contradicts` / `neutral`), and a free-text `note`.
### Agents
`buildTopicIntelligenceDeclarations` emits two tenant-scoped agents, both `role_slug: "editor"`:
- **Intelligence Curator** (`<tenant>-intel-curator` by default) — triages source items and submits promotion-gated assessments and relation scores. Tools: `searchEntities`, `getEntity`, `createRelation`, `submitResponse`, `fetchUrl`, `rssFeed`.
- **Intelligence Reviewer** (`<tenant>-intel-reviewer`) — reviews _parked_ low-confidence assessments and records calibration feedback. Tools are deliberately narrower: `searchEntities`, `getEntity`, `submitResponse`. It cannot create, edit, or delete a relation.
### Trend
A `trend` is a falsifiable claim with accumulating evidence — a movement worth tracking, distinct from a topic (a standing question) and from a signal (a dated observation). Source items link to trends through the `evidences` relation. The platform `trend` type is authored but **not registered**; see [The D-19 boundary](#the-d-19-boundary).
## How It Works
### The loop: collection → triage → promotion
1. **Collection.** `Intelligence Source Sync` (`<tenant>-intel-source-sync-daily`, cron `0 6 * * *`) synchronizes configured active sources through the **existing** source-sync ingestion path, preserving deduplication and provenance. Its instructions say so explicitly: do not create a new fetch or scheduling mechanism. Output is `source-item` entities.
2. **Triage.** `Intelligence Curator Triage` (`<tenant>-intel-curator-triage-daily`, cron `0 7 * * *`) depends on the sync action and processes a bounded batch — `DEFAULT_TRIAGE_BATCH_CAP` is 40; Sprinter uses 40. For each item the curator searches before promoting, marks near-duplicates as corroboration on the _original_ item, skips muted topics, scores one `signal-relevance` edge per item-topic pair, and submits the `signal-assessment` response. New trends stay `proposed`.
The batch cap rides `output_config.triage_batch_cap` and the instructions text, **not** `trigger_config` — `CronTriggerConfigSchema` is strict and a batch size is not a scheduling concern.
3. **Promotion.** Promotion is the criteria system's, not this module's. An assessment with `confidence >= 0.7` promotes and stamps its dimensions onto `source-item.content`; below the threshold it parks for review instead. That single threshold is the difference between "a signal on the feed" and "something a human still has to look at."
4. **Improve (opt-in).** `heartbeat-feedback-review` (cron `0 * * * *`) synthesizes explicit positive and negative signal feedback into `shared_context` lessons for the curator's next run. Its instructions forbid inferring preferences from impressions and forbid standing up a parallel preference store.
### The signal feed surface
`/t/sprinter/p/signals` is a tenant custom page, registered in `features/custom/tenants/sprinter/index.ts` under the renderer key `sprinter-signals` and rendered by `SignalsPage`. It composes existing platform primitives rather than new ones:
- **Feed** — `useFeed(signalFeedConfig(mode, topic))` over `entityTypeSlugs: ["source-item"]`. "For you" uses tab `for-you` sorted by `relevance`; "Latest" uses tab `new` sorted by `recency` and groups by day.
- **Topics** — `useEntities("topic", 200)` (200 is the `/api/entities` hard cap) feeds the rail.
- **Trend chips** — one permission-gated, tenant-filtered `entity_relations` query per visible batch, via the `loadSignalRelationProjections` server action.
Because "for you" ranks by `relevance` — a JSONB content value with no column to `ORDER BY` — the SQL query sorts by recency and `rankFeedItems` re-sorts afterwards. `signalFeedConfig` therefore sets `minRelevance: 0` in that mode, which is the platform's own signal to `hasScoreFilters()` to scan the bounded candidate set instead of paging the SQL order. Without it the feed would silently mean "the most relevant of the 20 newest."
### Structural muting
`excludeAllMutedSignals` runs **before** grouping, counting, and rendering. A card is hidden only when _every_ one of its stamped topics resolves to a muted topic; mixed active/muted cards and unstamped cards stay visible. The list holds its loading/error state until topic status is available, so muting fails closed rather than flashing muted-only cards.
Rail counts describe the currently loaded, visible page — not a global aggregate. While a topic filter is active, every _other_ topic's count is reported as `null` and rendered as nothing, because the feed then contains only the selected topic and a `0` would be a placeholder the reader would misread as "no signals here."
### Relation projections
Visible signal IDs are split into bounded batches of 100 so trend chips survive past five feed pages. A failed batch does not poison its neighbours: successful batches still render, failed ones surface "Trend links unavailable" rather than claiming a confirmed absence, and a confirmed-empty join preserves the existing "No linked trend yet" state.
## Activation Model
**A tenant is activated by a reviewed one-line allowlist change plus a data-op — never by code alone.**
The fence lives in `features/custom/tenants/lib/intelligence-activation.ts`:
```typescript
export const INTELLIGENCE_ACTIVATED_TENANT_SLUGS: readonly string[] = [
"sprinter",
];
export function isIntelligenceActivated(tenantSlug: string): boolean;
export function gateIntelligenceRecords<T>(
tenantSlug: string,
records: readonly T[],
): readonly T[];
```
It is an **allowlist**, so the default is closed: a tenant nobody thought about is fenced, not exposed. A denylist would fail open for every future tenant. The check is exact equality, not a substring or prefix match — `sprinter-staging` is not activated.
The fence is applied at the point declarations enter a shared array, because three separate surfaces read the same declaration arrays and one fence at the source closes all three:
1. **Tenant push** — `tenantCodeRecords` in a tenant's `declarations/index.ts` is the diff source for `pnpm tenant:push <slug>`.
2. **Runtime-default registration** — `declarations.server.ts` projects the same arrays into `TenantDeclarationsModule.declarations`, which `bootstrapTenantModule` registers into the tenant-defaults registry. `ensureDefaultsRegistered()` resolves them on a DB miss, so a cold preview can surface a default with no DB row at all.
3. **Execution** — an action that never reaches either surface has no row to tick.
**`status: "paused"` is not a substitute for the fence.** Paused is a lifecycle flag on a row that already exists, and a manual trigger is exactly the path that ignores it. The two controls stack: the fence decides whether the row exists at all; `paused` decides whether the scheduler ticks it.
That is also why the code shipping does not mean the loop is running. Every action this module declares ships `status: "paused"`. Turning Sprinter's feed from fixtures into live signals is a data-op against the `actions` rows, not a code change.
Activating a new tenant is therefore three coupled things in one reviewed PR: the slug ruling, the one-line allowlist addition, and the data-op runbook.
### The D-19 boundary
Two related decisions, both deliberately unresolved in code:
**The platform `trend` type is authored but UNREGISTERED.** `TREND_SYSTEM_ENTITY_TYPE` exists in `features/entities/system-types/trend-system-entity-type.ts` and is fully specified, but it is _not_ in `SYSTEM_ENTITY_TYPES` and does not appear in `SYSTEM_ENTITY_TYPES_BY_SLUG`. Production already holds one global `trend` row with a legacy schema and live records, and system sync replaces a registered type's `json_schema` wholesale — so automatic registration would reinterpret those records before the separately authorized schema/data reconciliation. The catalog fails closed and a test asserts the absence, so registration cannot creep in by accident.
**Marbella is deliberately absent from the allowlist.** D-19 rules that the activation runbook must not push topic/trend declarations to Marbella before its slug decision lands. `features/custom/tenants/marbella/declarations/index.ts` therefore routes its intelligence records through `gateIntelligenceRecords`, which returns an empty array for a fenced tenant. Marbella is added to the allowlist in the same PR that resolves its slug — not before.
Both boundaries are enforced by tests rather than prose, because prose is not a gate.
## API Reference
Builders in `features/custom/tenants/lib/` — all pure, all returning declaration data only:
```typescript
buildTopicIntelligenceDeclarations(options: {
tenantSlug: string;
topicAffectsTargets: readonly string[];
curatorAgentSlug?: string;
triageBatchCap?: number; // default DEFAULT_TRIAGE_BATCH_CAP = 40
}): TopicIntelligenceDeclarations;
// → entityTypeExtensions, criteriaSets, views, agents, actions
buildTrendTrackingDeclarations(options: {
tenantSlug: string;
trendTypeSlug: string;
});
// → trend-assessment / trend-vitality criteria, the trend-curator agent,
// the trends-radar view, and the trend-evidence-digest /
// trend-freshness-sweep / trend-vitality-rescore actions
buildTrendBasisDeclarations(options: {
trendTypeSlug: string;
decisionTypeSlugs: readonly string[];
roleOptions?: readonly string[];
tenantSlug?: string;
});
// → rests_on_trends extensions on each decision type, the trend-basis /
// trend-basis-review criteria, the basis-reviewer agent, and the
// trend-shift-fanout / decision-basis-review /
// trend-basis-staleness-sweep actions
```
`buildTopicIntelligenceDeclarations` validates its inputs and throws rather than degrading: every slug must match `^[a-z0-9]+(?:-[a-z0-9]+)*$`, `topicAffectsTargets` must be non-empty, `triageBatchCap` must be a positive integer, and the curator slug must not collide with the derived reviewer slug.
The activation fence:
```typescript
isIntelligenceActivated(tenantSlug: string): boolean;
gateIntelligenceRecords<T>(tenantSlug: string, records: readonly T[]): readonly T[];
```
Feed-surface model helpers in `features/custom/components/signal-card/signals-page-model.ts`:
```typescript
readTopicOption(entity: EntityRecord): TopicOption;
countSignalsByStampedTopic(items: FeedItemV2[], topics: TopicOption[]): Record<string, number>;
excludeAllMutedSignals(items: FeedItemV2[], topics: TopicOption[]): FeedItemV2[];
signalFeedConfig(mode: SignalFeedMode, topic: { slug: string; title: string } | null);
loadSignalRelationProjectionBatches(signalIds, load): Promise<SignalRelationProjectionBatchResult>;
latestAssessmentTimestamp(items: FeedItemV2[]): string | null;
formatDataAsOf(value: string | null): string;
```
Constants worth importing rather than retyping: `SIGNAL_ASSESSMENT_CRITERIA_SET_SLUG`, `SIGNAL_RELEVANCE_CRITERIA_SET_SLUG`, `DEFAULT_TRIAGE_BATCH_CAP`, `TRIAGE_CURATOR_TOOLS`, `INTELLIGENCE_REVIEWER_TOOLS`, `INTELLIGENCE_ACTIVATED_TENANT_SLUGS`.
## For Agents
This module registers **no new tools**. The curator and reviewer are DB agent rows configured with existing generic tools — that is the point, and adding a dedicated trend-intelligence tool would be a regression.
- **Curator** — `searchEntities`, `getEntity`, `createRelation`, `submitResponse`, `fetchUrl`, `rssFeed`. Search before you promote. Mark a near-duplicate as corroboration on the original item rather than creating a second signal. Skip muted topics entirely. Score one relevance edge per item-topic pair. Respect the batch cap.
- **Reviewer** — `searchEntities`, `getEntity`, `submitResponse` only. Review parked assessments, submit a corrected response with a calibration note when warranted. Never create, edit, or delete a relation; never publish, send, call webhooks, or manage accounts.
- **Confidence is a promotion decision, not a vibe.** It is scored on the same 0–1 scale as the 0.7 threshold it is compared against. Under-confident output parks for a human rather than reaching the feed.
- **`suggested_action` must stay side-effect free.** Every enum option is an internal next step. An agent proposing an external action is out of contract.
- **Spend is attributable.** Every action carries `metadata.budget_attribution = "trend-intelligence:<tenantSlug>"`, per tenant, never a shared bucket.
Agents extending this module should add dimensions to the existing criteria sets or topics to the tenant's declarations — not stand up a parallel scoring path.
## Design Decisions
**The fence is code, not prose.** The previous version of this boundary was a comment in `intelligence.ts` asserting that nothing was pushed until the slug ruling landed, while `index.ts` spread the declarations unconditionally. A comment cannot stop a push. The allowlist module exists because that failure mode already happened once.
**Allowlist, not denylist.** The default must be closed. A denylist fails open for every tenant invented after it was written.
**Actions ship paused.** A cron definition that lands enabled runs before anyone has reviewed its cost, its prompt, or its blast radius. Paused-by-default makes the first run a decision.
**No decorative fallback views.** `buildTopicIntelligenceDeclarations` returns `views: []`. The generic entity-feed cannot enforce promoted assessments or muted-topic relation filters, and coverage aggregates and the implication matrix have no truthful block projection — so a tenant without a purpose-built surface gets an explicit surface decision rather than a view that looks right and lies.
**Promoted stamps are topic truth; source tags are not.** Rail filters and card counts read only `source-item.content.topics`. Source tags describe collection defaults and can disagree with what the curator actually matched.
**`matched_topics` uses `relation-rank` with a `fieldName` and no relation metadata.** The platform has no rendered string-array response dimension, and a relation picker returns entity IDs where slugs are required. Omitting the relation metadata means promotion stamps the slug array through the `fieldName` seam without also attempting relation-rank writes. It is optional so manual corrections can still submit the remaining dimensions.
**Muting fails closed.** Holding the loading state until topic status resolves is worse UX for a few hundred milliseconds and strictly better than flashing content a user asked never to see.
**Counts are measurements or nothing.** A count that cannot be measured renders as absent rather than as `0`. A placeholder zero is indistinguishable from a real one to a reader.
**Batch failures degrade honestly.** "Trend links unavailable" and "No linked trend yet" are different claims, and the UI never substitutes the second for the first.
## Related Modules
- **Entity System** (`features/entities/`) — owns the `topic` and (authored, unregistered) `trend` system types; `source-item` carries the promoted assessment fields.
- **Source Sync** (`features/source-sync/`) — the ingestion and deduplication path collection reuses; `buildGoogleNewsSearchUrl` builds Sprinter's search-feed sources.
- **Feed System** (`features/feed/`) — `useFeed`, `rankFeedItems`, `groupFeedItemsByDay`, and the `FeedItemV2` shape the signal cards render from.
- **Unified Response System** — criteria sets, agent scoring, promotion policy, and the park-for-review path.
- **Actions** (`features/actions/`) — the cron rows the loop runs on; `status` and `trigger_config` are the lifecycle controls.
- **Custom Pages** (`features/custom-pages/`) — how `/p/signals` resolves through the tenant renderer registry.
- **Analytics & Cost** — `metadata.budget_attribution` is the per-tenant spend key for this feature.