Documentation source

Field-Dimension Unification

Roadmap for fully merging entity type fields and criteria set dimensions into a single system

## Problem

The platform currently has two parallel systems for defining extractable data on entities:

1. **Field configs** (`entity_types.config.fields`) — define extraction behavior (agent, instructions, dependencies) for entity content fields
2. **Criteria set dimensions** (`criteria_sets.dimensions`) — define scoring/evaluation dimensions with weights and scales

These serve the same fundamental purpose: defining what data to collect about an entity and how to collect it. The current approach (Phase 1) bridges them by making dimensions extractable alongside fields, but the two schemas remain separate.

## Vision

Fully unify fields and dimensions into a single schema. Every piece of data on an entity — whether it's a text description, a numeric score, or a categorization — is a **dimension** within a **criteria set**.

### Phase 1 (Current — Implemented)

- Extraction config added to `CriteriaSetDimension`
- DAG compiler creates scoring nodes alongside field nodes
- Scoring criteria sets are extractable through the normal workflow
- All scoring writes go through `submitResponse` for response history
- Legacy `weighted_score` fallback removed

### Phase 2 (Next)

- The "default" criteria set for an entity type contains dimensions that map 1:1 to the entity's fields
- Field extraction writes go through `submitResponse` instead of direct entity.content writes
- Every field change has a response record (full audit trail)
- `ensureDefaultCriteriaSet` already creates default dimensions from the schema — extend it to include extraction config from field configs

### Phase 3 (Final)

- Extraction metadata (agent, instructions, dependencies) moves from `FieldConfig` into `CriteriaSetDimension.extraction`
- `FieldConfig` is deprecated — criteria set dimensions are the single source of truth
- The default criteria set becomes the canonical definition of an entity type's fields
- UI editors (schema editor, field config editor) work directly on criteria set dimensions
- Display metadata (label, displayType, statusMap) either moves to dimensions or becomes a separate display config layer

## Trade-offs

**Benefits:**
- Single mental model: everything is a dimension in a criteria set
- Universal response history for all data changes
- Consistent extraction pipeline for fields and scores
- Simpler code — one system instead of two

**Risks:**
- Large migration of existing field configs
- Criteria set dimensions need to support all current field config features (statusMap, connection, cascadeExtraction, etc.)
- Breaking change for any direct field config consumers

## Migration Strategy

Phase 2 and 3 are backward-compatible internally. The key principle: criteria sets that contain field-mapped dimensions auto-promote values to entity.content, preserving the existing data model. The migration is invisible to the frontend — it reads entity.content the same way.

## Problem The platform currently has two parallel systems for defining extractable data on entities: 1. **Field configs** (`entity_types.config.fields`) — define extraction behavior (agent, instructions, dependencies) for entity content fields 2. **Criteria set dimensions** (`criteria_sets.dimensions`) — define scoring/evaluation dimensions with weights and scales These serve the same fundamental purpose: defining what data to collect about an entity and how to collect it. The current approach (Phase 1) bridges them by making dimensions extractable alongside fields, but the two schemas remain separate. ## Vision Fully unify fields and dimensions into a single schema. Every piece of data on an entity — whether it's a text description, a numeric score, or a categorization — is a **dimension** within a **criteria set**. ### Phase 1 (Current — Implemented) - Extraction config added to `CriteriaSetDimension` - DAG compiler creates scoring nodes alongside field nodes - Scoring criteria sets are extractable through the normal workflow - All scoring writes go through `submitResponse` for response history - Legacy `weighted_score` fallback removed ### Phase 2 (Next) - The "default" criteria set for an entity type contains dimensions that map 1:1 to the entity's fields - Field extraction writes go through `submitResponse` instead of direct entity.content writes - Every field change has a response record (full audit trail) - `ensureDefaultCriteriaSet` already creates default dimensions from the schema — extend it to include extraction config from field configs ### Phase 3 (Final) - Extraction metadata (agent, instructions, dependencies) moves from `FieldConfig` into `CriteriaSetDimension.extraction` - `FieldConfig` is deprecated — criteria set dimensions are the single source of truth - The default criteria set becomes the canonical definition of an entity type's fields - UI editors (schema editor, field config editor) work directly on criteria set dimensions - Display metadata (label, displayType, statusMap) either moves to dimensions or becomes a separate display config layer ## Trade-offs **Benefits:** - Single mental model: everything is a dimension in a criteria set - Universal response history for all data changes - Consistent extraction pipeline for fields and scores - Simpler code — one system instead of two **Risks:** - Large migration of existing field configs - Criteria set dimensions need to support all current field config features (statusMap, connection, cascadeExtraction, etc.) - Breaking change for any direct field config consumers ## Migration Strategy Phase 2 and 3 are backward-compatible internally. The key principle: criteria sets that contain field-mapped dimensions auto-promote values to entity.content, preserving the existing data model. The migration is invisible to the frontend — it reads entity.content the same way.