Add VISA backend foundation
This commit is contained in:
jerryW123
@@ -0,0 +1,95 @@
|
||||
# VISA API And Migration Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a real backend API surface and a database schema migration for the VISA backend foundation.
|
||||
|
||||
**Architecture:** Add an Express app factory with seeded in-memory VISA data so the API can run without introducing a database driver yet. Add a PostgreSQL migration file that creates the normalized VISA schema described in `docs/visa-backend-database-design.md`.
|
||||
|
||||
**Tech Stack:** TypeScript, Express, Node assert tests, PostgreSQL SQL migration.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: API Tests
|
||||
|
||||
**Files:**
|
||||
- Create: `src/server/visaApi.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing API tests**
|
||||
|
||||
Test `GET /api/model-versions/:id/visa`, `GET /api/model-versions/:id/view-model`, `POST /api/model-versions/:id/actions/run-check`, and a missing model version response.
|
||||
|
||||
- [ ] **Step 2: Run the test to verify red**
|
||||
|
||||
Run: `npx.cmd tsx src/server/visaApi.test.ts`.
|
||||
Expected: FAIL because the server app module does not exist.
|
||||
|
||||
### Task 2: API Implementation
|
||||
|
||||
**Files:**
|
||||
- Create: `src/server/visaRepository.ts`
|
||||
- Create: `src/server/visaApi.ts`
|
||||
- Create: `src/server/index.ts`
|
||||
- Modify: `package.json`
|
||||
|
||||
- [ ] **Step 1: Add in-memory repository**
|
||||
|
||||
Seed the repository with `createDefaultVisaSpec()` and expose `getVisaSpec`, `getWorkflowViewModel`, and `runConsistencyCheck`.
|
||||
|
||||
- [ ] **Step 2: Add Express app factory**
|
||||
|
||||
Expose the VISA JSON endpoint, view-model endpoint, and run-check action.
|
||||
|
||||
- [ ] **Step 3: Add server entry and npm script**
|
||||
|
||||
Add `src/server/index.ts` and a `server` script using `tsx`.
|
||||
|
||||
- [ ] **Step 4: Run API tests**
|
||||
|
||||
Run: `npx.cmd tsx src/server/visaApi.test.ts`.
|
||||
Expected: PASS.
|
||||
|
||||
### Task 3: Migration Tests
|
||||
|
||||
**Files:**
|
||||
- Create: `db/migrations/202606230001_create_visa_schema.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing migration test**
|
||||
|
||||
Assert the migration includes the system tables, VISA semantic tables, consistency check table, foreign keys, and indexes.
|
||||
|
||||
- [ ] **Step 2: Run the test to verify red**
|
||||
|
||||
Run: `npx.cmd tsx db/migrations/202606230001_create_visa_schema.test.ts`.
|
||||
Expected: FAIL because the SQL migration does not exist.
|
||||
|
||||
### Task 4: Migration SQL
|
||||
|
||||
**Files:**
|
||||
- Create: `db/migrations/202606230001_create_visa_schema.sql`
|
||||
|
||||
- [ ] **Step 1: Add PostgreSQL schema migration**
|
||||
|
||||
Create model/version tables, VISA T1-T8 normalized tables, consistency check table, foreign keys, uniqueness constraints, and basic indexes.
|
||||
|
||||
- [ ] **Step 2: Run migration test**
|
||||
|
||||
Run: `npx.cmd tsx db/migrations/202606230001_create_visa_schema.test.ts`.
|
||||
Expected: PASS.
|
||||
|
||||
### Task 5: Verification
|
||||
|
||||
**Files:**
|
||||
- Modify as needed based on test results.
|
||||
|
||||
- [ ] **Step 1: Run model and API tests**
|
||||
|
||||
Run:
|
||||
- `npx.cmd tsx src/workflowModel.test.ts`
|
||||
- `npx.cmd tsx src/visaModel.test.ts`
|
||||
- `npx.cmd tsx src/server/visaApi.test.ts`
|
||||
- `npx.cmd tsx db/migrations/202606230001_create_visa_schema.test.ts`
|
||||
|
||||
- [ ] **Step 2: Run TypeScript check**
|
||||
|
||||
Run: `npm.cmd run lint`.
|
||||
@@ -0,0 +1,82 @@
|
||||
# VISA Backend Foundation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a VISA-native backend/database foundation while keeping the current frontend workflow model compatible through an adapter.
|
||||
|
||||
**Architecture:** Define VISA T1-T8 as backend-oriented domain structures, add a canonical sample specification, and convert that specification into the existing `WorkflowModel` view model. Keep the database design in documentation for the next backend phase.
|
||||
|
||||
**Tech Stack:** TypeScript, Node assert tests, Vite React, Markdown database design documentation.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Adapter Tests
|
||||
|
||||
**Files:**
|
||||
- Create: `src/visaModel.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write tests for VISA-to-workflow conversion**
|
||||
|
||||
Create tests that assert the adapter exposes VISA agents as frontend workflow nodes, internal functions as behavior sections, sensing records as observable rows, and model-level tables as schedule/data/I-O/validation rows.
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `npx tsx src/visaModel.test.ts`
|
||||
Expected: FAIL because `visaModel` does not exist yet.
|
||||
|
||||
### Task 2: VISA Domain Model
|
||||
|
||||
**Files:**
|
||||
- Create: `src/visaModel.ts`
|
||||
|
||||
- [ ] **Step 1: Define VISA T1-T8 TypeScript interfaces**
|
||||
|
||||
Define backend-facing types for agents, variables, sensing relations, internal functions, associated data, inputs, outputs, schedule steps, termination conditions, validations, and consistency check results.
|
||||
|
||||
- [ ] **Step 2: Add a canonical default VISA specification**
|
||||
|
||||
Create `createDefaultVisaSpec()` using the existing segregation-style example but expressed as VISA T1-T8 rather than frontend tables.
|
||||
|
||||
### Task 3: Workflow Adapter
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/visaModel.ts`
|
||||
|
||||
- [ ] **Step 1: Implement `buildWorkflowFromVisaSpec`**
|
||||
|
||||
Convert T1 agents into workflow nodes, selected T3/T4 relations into view edges and behavior sections, and T5-T8 records into the existing table rows.
|
||||
|
||||
- [ ] **Step 2: Run adapter tests**
|
||||
|
||||
Run: `npx tsx src/visaModel.test.ts`
|
||||
Expected: PASS.
|
||||
|
||||
### Task 4: Database Design Documentation
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/visa-backend-database-design.md`
|
||||
|
||||
- [ ] **Step 1: Document the database schema**
|
||||
|
||||
Write the scheme as eight VISA semantic modules plus normalized helper tables for sensing/function references, model versions, and consistency check results.
|
||||
|
||||
- [ ] **Step 2: Document frontend boundary**
|
||||
|
||||
State that the frontend must not show the eight tables as raw database tables; it consumes view models derived from VISA data.
|
||||
|
||||
### Task 5: Verification
|
||||
|
||||
**Files:**
|
||||
- Modify as needed based on test results.
|
||||
|
||||
- [ ] **Step 1: Run all model tests**
|
||||
|
||||
Run: `npx tsx src/workflowModel.test.ts` and `npx tsx src/visaModel.test.ts`.
|
||||
|
||||
- [ ] **Step 2: Run TypeScript lint**
|
||||
|
||||
Run: `npm run lint`.
|
||||
|
||||
- [ ] **Step 3: Report status**
|
||||
|
||||
Summarize changed files, tests, and remaining backend work.
|
||||
@@ -0,0 +1,396 @@
|
||||
# VISA Backend Database Design
|
||||
|
||||
## Purpose
|
||||
|
||||
The project should treat VISA as a backend model specification protocol, not as a set of frontend tables. The database is the source of truth for reproducible ABM descriptions. The frontend consumes derived view models for graph editing, summaries, validation reports, and export actions.
|
||||
|
||||
This design follows the selected approach C:
|
||||
|
||||
```text
|
||||
Database and backend: VISA-native T1-T8 specification
|
||||
Adapter API: VISA data -> frontend workflow view model
|
||||
Frontend: modeling interface, not raw database table display
|
||||
```
|
||||
|
||||
## High-Level Architecture
|
||||
|
||||
```text
|
||||
Frontend
|
||||
Canvas, forms, summaries, validation report, export actions
|
||||
|
|
||||
v
|
||||
Adapter API
|
||||
Builds frontend view models from VISA records
|
||||
Converts semantic edit actions into VISA updates
|
||||
|
|
||||
v
|
||||
Backend Services
|
||||
Model versioning
|
||||
VISA T1-T8 CRUD
|
||||
19-rule consistency checking
|
||||
VISA JSON export
|
||||
Future code generation
|
||||
|
|
||||
v
|
||||
Database
|
||||
Model/version metadata
|
||||
VISA semantic modules
|
||||
Normalized helper tables
|
||||
Consistency check results
|
||||
```
|
||||
|
||||
## System Tables
|
||||
|
||||
### `models`
|
||||
|
||||
Stores the model project.
|
||||
|
||||
```text
|
||||
id
|
||||
name
|
||||
description
|
||||
domain
|
||||
author
|
||||
created_at
|
||||
updated_at
|
||||
```
|
||||
|
||||
### `model_versions`
|
||||
|
||||
Stores versioned VISA specifications. Every VISA record belongs to one `model_version_id`.
|
||||
|
||||
```text
|
||||
id
|
||||
model_id
|
||||
version_name
|
||||
status: draft | checked | released | archived
|
||||
source_protocol: VISA
|
||||
created_by
|
||||
created_at
|
||||
notes
|
||||
```
|
||||
|
||||
## VISA Semantic Modules
|
||||
|
||||
The database may use more than eight physical tables, but the backend must expose eight VISA semantic modules.
|
||||
|
||||
## T1 Agent
|
||||
|
||||
Physical table: `visa_agents`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
name
|
||||
symbol_set
|
||||
instance_symbol
|
||||
category: Environment | Space | Decision-maker | Passive
|
||||
description
|
||||
quantity_symbol
|
||||
quantity_type: fixed | variable
|
||||
quantity_value
|
||||
sort_order
|
||||
```
|
||||
|
||||
Design rule: database records should represent model-semantic agents, not implementation classes. Datasets, output records, and reports belong to T5 or T6, not T1.
|
||||
|
||||
## T2 Variable
|
||||
|
||||
Physical table: `visa_variables`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
agent_id
|
||||
name
|
||||
symbol
|
||||
variable_type: exog_homo | exog_hetero | endog_decision | endog_non_decision
|
||||
data_type
|
||||
value_source
|
||||
unit
|
||||
description
|
||||
is_time_indexed
|
||||
sort_order
|
||||
```
|
||||
|
||||
Design rule: endogenous variables should be traceable to T4 function IDs. Exogenous variables marked as `Input` must be covered by T6a.
|
||||
|
||||
## T3 Sensing
|
||||
|
||||
Physical tables:
|
||||
|
||||
```text
|
||||
visa_sensing_relations
|
||||
visa_sensing_variables
|
||||
```
|
||||
|
||||
`visa_sensing_relations`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
observer_agent_id
|
||||
observed_agent_id
|
||||
access_type: none | partial | all
|
||||
scope: self | peer | other | all | singleton
|
||||
condition
|
||||
notes
|
||||
```
|
||||
|
||||
`visa_sensing_variables`
|
||||
|
||||
```text
|
||||
sensing_relation_id
|
||||
variable_id
|
||||
```
|
||||
|
||||
Design rule: store sensing as normalized relations rather than a wide matrix. The backend can reconstruct the VISA matrix for export and can check whether T4 functions use only authorized information.
|
||||
|
||||
## T4 Internal Function
|
||||
|
||||
Physical tables:
|
||||
|
||||
```text
|
||||
visa_internal_functions
|
||||
visa_function_inputs
|
||||
visa_function_updates
|
||||
```
|
||||
|
||||
`visa_internal_functions`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
agent_id
|
||||
function_code
|
||||
name
|
||||
method
|
||||
reference
|
||||
description
|
||||
sort_order
|
||||
```
|
||||
|
||||
`visa_function_inputs`
|
||||
|
||||
```text
|
||||
function_id
|
||||
variable_id
|
||||
source_type: self | sensed | input | output
|
||||
expression
|
||||
```
|
||||
|
||||
`visa_function_updates`
|
||||
|
||||
```text
|
||||
function_id
|
||||
target_agent_id
|
||||
variable_id
|
||||
update_type: self_state | external_effect | create_agent | remove_agent
|
||||
expression
|
||||
```
|
||||
|
||||
Design rule: every internal function must update at least one endogenous variable or create/remove an instance of a variable-quantity agent type.
|
||||
|
||||
## T5 Associated Data
|
||||
|
||||
Physical table: `visa_associated_data`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
data_code
|
||||
title
|
||||
data_type: Empirical | Literature | Generated
|
||||
temporal_type: Static | Dynamic
|
||||
source
|
||||
collection_method: Survey | Administrative | Sensor | Experimental | Computational
|
||||
preprocessing_method: None | Selected | Aggregated | Transformed
|
||||
record_count
|
||||
availability: Open | Restricted | Private
|
||||
license
|
||||
url_or_reference
|
||||
description
|
||||
```
|
||||
|
||||
Design rule: this table records data provenance. It is not merely a source-column lookup table.
|
||||
|
||||
## T6 Input and Output
|
||||
|
||||
Physical tables:
|
||||
|
||||
```text
|
||||
visa_inputs
|
||||
visa_outputs
|
||||
```
|
||||
|
||||
`visa_inputs`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
agent_id
|
||||
variable_id
|
||||
symbol
|
||||
value_or_distribution
|
||||
data_source_id
|
||||
derivation: Direct | Estimated | Computed | Assumed
|
||||
algorithm
|
||||
reference
|
||||
notes
|
||||
```
|
||||
|
||||
`visa_outputs`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
symbol
|
||||
indicator_name
|
||||
formula
|
||||
data_type
|
||||
unit
|
||||
frequency
|
||||
description
|
||||
```
|
||||
|
||||
Design rule: T6a covers exogenous model inputs. T6b covers simulation output indicators. Component-to-component frontend wiring is a derived view, not the canonical T6 design.
|
||||
|
||||
## T7 Schedule
|
||||
|
||||
Physical tables:
|
||||
|
||||
```text
|
||||
visa_schedule_steps
|
||||
visa_termination_conditions
|
||||
```
|
||||
|
||||
`visa_schedule_steps`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
step_number
|
||||
agent_id
|
||||
function_id
|
||||
execution_mode: Synchronous | Sequential | Random-order | Asynchronous
|
||||
execution_parameters
|
||||
condition
|
||||
sort_order
|
||||
```
|
||||
|
||||
`visa_termination_conditions`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
condition_code
|
||||
indicator_symbol
|
||||
condition_expression
|
||||
description
|
||||
value_source
|
||||
termination_logic
|
||||
```
|
||||
|
||||
Design rule: execution mode must be structured because synchronization choices can change ABM behavior.
|
||||
|
||||
## T8 Validation
|
||||
|
||||
Physical table: `visa_validations`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
validation_code
|
||||
level: Agent | Model | Output
|
||||
validation_object
|
||||
benchmark_data_id
|
||||
method
|
||||
indicator
|
||||
passing_condition
|
||||
reference
|
||||
status: pending | passed | failed | blocked
|
||||
notes
|
||||
```
|
||||
|
||||
Design rule: T8 is scientific model validation. It should not be mixed with the 19-rule consistency checker.
|
||||
|
||||
## Consistency Check Results
|
||||
|
||||
Physical table: `visa_consistency_check_results`
|
||||
|
||||
```text
|
||||
id
|
||||
model_version_id
|
||||
rule_code
|
||||
rule_type: within_table | cross_table
|
||||
status: passed | failed | warning
|
||||
message
|
||||
related_table
|
||||
related_record_id
|
||||
checked_at
|
||||
```
|
||||
|
||||
The backend should run the VISA 19-rule checker against a complete model version and persist results here. These records drive frontend check reports.
|
||||
|
||||
## Frontend Boundary
|
||||
|
||||
The frontend must not present the database as eight raw table pages. It should provide modeling workflows:
|
||||
|
||||
```text
|
||||
Model overview
|
||||
Agent and space designer
|
||||
Variable editor
|
||||
Behavior/function designer
|
||||
Data source manager
|
||||
Schedule builder
|
||||
Validation and check report
|
||||
Export/generate actions
|
||||
```
|
||||
|
||||
These screens are allowed to show summaries, forms, cards, graphs, and reports. They should not make the user edit `visa_agents` or `visa_variables` as database rows.
|
||||
|
||||
## Adapter Contract
|
||||
|
||||
The adapter layer should expose two families of API responses.
|
||||
|
||||
Canonical VISA:
|
||||
|
||||
```text
|
||||
GET /api/model-versions/:id/visa
|
||||
```
|
||||
|
||||
Returns a complete T1-T8 VISA JSON object for checking, export, and code generation.
|
||||
|
||||
Frontend view model:
|
||||
|
||||
```text
|
||||
GET /api/model-versions/:id/view-model
|
||||
```
|
||||
|
||||
Returns graph nodes, graph edges, grouped variables, behavior cards, schedule timeline items, and validation summaries derived from VISA records.
|
||||
|
||||
Semantic edit actions:
|
||||
|
||||
```text
|
||||
POST /api/model-versions/:id/actions/add-agent
|
||||
POST /api/model-versions/:id/actions/add-variable
|
||||
POST /api/model-versions/:id/actions/add-function
|
||||
POST /api/model-versions/:id/actions/run-check
|
||||
```
|
||||
|
||||
The backend translates these actions into normalized VISA writes and refreshes the derived view model.
|
||||
|
||||
## First Implementation Step
|
||||
|
||||
The current frontend can remain on `WorkflowModel` while a VISA adapter is introduced. The adapter converts:
|
||||
|
||||
```text
|
||||
T1 Agent -> workflow nodes
|
||||
T3 Sensing + T4 Internal Function -> view edges and behavior sections
|
||||
T5 Associated Data -> associated data rows
|
||||
T6 Input/Output -> I/O rows
|
||||
T7 Schedule -> schedule rows
|
||||
T8 Validation -> validation rows
|
||||
```
|
||||
|
||||
This keeps the existing UI functional while moving the source of truth toward the backend VISA model.
|
||||
Reference in New Issue
Block a user