v1.4.0 Design: Persistent Memory + Webhook DX

Date: 2026-02-14 Status: Approved Milestone: v1.4.0

Overview

v1.4.0 adds two features:

Persistent Memory — Agents accumulate knowledge across sessions via Markdown-backed storage, searchable through the existing RAG pipeline.
Webhook DX — Signature verification middleware (HMAC-SHA256 / JWT) with provider presets and DSL-level webhook definitions.

Both features extend existing abstractions rather than introducing new subsystems.

1. MarkdownMemoryStore

A new ContextPersistence implementation that stores agent memory as human-readable Markdown files.

File Layout

data/
  agents/
    {agent_id}/
      memory.md          # Current memory state
      logs/
        2026-02-14.md    # Daily interaction log
        2026-02-13.md

Markdown Format

memory.md:

# Agent Memory: {agent_id}
Updated: 2026-02-14T10:30:00Z

## Facts
- User prefers dark mode
- API rate limit is 100 req/min

## Procedures
- Deploy via `cargo shuttle deploy`

## Learned Patterns
- User asks about metrics after every deployment

Daily log:

# Session Log: 2026-02-14

## 10:30 — Deployment Review
- Deployed v1.3.2
- User confirmed metrics looked normal

## 14:15 — Bug Report
- Issue #42: Memory leak in context manager
- Assigned to backlog

Implementation

pub struct MarkdownMemoryStore {
    root_dir: PathBuf,
    retention: RetentionPolicy,
}

Implements ContextPersistence from crates/runtime/src/context/manager.rs:

save_context — Serializes AgentContext.memory (the HierarchicalMemory struct) into the Markdown sections (Facts, Procedures, Learned Patterns). Atomic write via tempfile::NamedTempFile + persist(). Appends session summary to today’s daily log.
load_context — Parses memory.md back into HierarchicalMemory fields. Falls back to empty context if file missing.
delete_context — Removes agent directory.
list_agent_contexts — Lists subdirectories under data/agents/.
get_storage_stats — Walks agent directories, sums file sizes.

Retention & Compaction

RetentionPolicy specifies max_age: Duration for daily logs (default: 90 days).
compact() flushes long-term memory items from daily logs into memory.md sections and deletes expired logs.
purge(agent_id) removes the agent’s entire memory directory.

Integration Point

StandardContextManager already holds persistence: Option<Arc<dyn ContextPersistence>>. The MarkdownMemoryStore is injected here via configuration — no changes to StandardContextManager itself.

2. Memory Search via RAG Pipeline

Memory items are searchable through the existing StandardRAGEngine rather than a standalone search system.

Indexing

When MarkdownMemoryStore::save_context() writes memory, it also indexes memory items as Document objects in the RAG engine:

let doc = DocumentInput {
    title: format!("Memory: {}", agent_id),
    content: memory_section_text,
    metadata: HashMap::from([
        ("source".into(), "memory".into()),
        ("agent_id".into(), agent_id.to_string()),
        ("memory_type".into(), "fact".into()),  // or "procedure", "pattern"
    ]),
};
rag_engine.ingest_documents(vec![doc]).await?;

Source-Filtered Search

Memory search uses RAGEngine::process_query() with a source filter constraint:

let request = RAGRequest {
    agent_id,
    query: search_query.to_string(),
    constraints: Some(QueryConstraints {
        source_filter: Some("memory".to_string()),
        ..Default::default()
    }),
    ..Default::default()
};

This reuses the existing retrieval → ranking pipeline without duplicating vector search logic.

Ranking

Memory results use RankingAlgorithm::Hybrid with weights:

70% vector similarity (semantic match)
30% BM25 (keyword match)

These weights are configurable via RAGConfig::ranking_config.

Index Maintenance

On save_context: Re-index changed memory items (upsert by agent_id + memory_type).
On delete_context: Remove all documents with matching agent_id source metadata.
On compact: Re-index after compaction merges daily log items into long-term memory.

3. DSL `memory` Block

A new top-level block in the tree-sitter grammar for declaring agent memory configuration.

Grammar

memory "agent-memory" {
    store    markdown
    path     "data/agents"
    retention 90d
    search {
        vector_weight  0.7
        keyword_weight 0.3
    }
}

Parsed Type

pub struct MemoryDefinition {
    pub name: String,
    pub store: MemoryStoreType,     // Markdown (only variant for now)
    pub path: PathBuf,
    pub retention: Duration,         // Parsed via humantime (e.g. "90d", "6months")
    pub search: Option<MemorySearchConfig>,
}

pub struct MemorySearchConfig {
    pub vector_weight: f64,          // Default: 0.7
    pub keyword_weight: f64,         // Default: 0.3
}

pub enum MemoryStoreType {
    Markdown,
}

Extraction

New function extract_memory_definitions(tree: &Tree, source: &str) -> Result<Vec<MemoryDefinition>, String> in the DSL crate, following the pattern of extract_schedule_definitions and extract_channel_definitions.

The retention field uses humantime::parse_duration() for human-readable durations (90d, 6months, 1y).

4. Webhook Signature Verification

Axum middleware that verifies webhook signatures before requests reach HttpInputServer handlers.

Verification Methods

HMAC-SHA256:

pub struct HmacVerifier {
    secret: Vec<u8>,
    header_name: String,       // e.g. "X-Hub-Signature-256"
    prefix: Option<String>,    // e.g. "sha256=" for GitHub
}

Reads raw request body, computes HMAC-SHA256(secret, body), compares against the signature header value using subtle::ConstantTimeEq for timing-safe comparison.

JWT:

pub struct JwtVerifier {
    public_key: DecodingKey,
    header_name: String,       // e.g. "Authorization"
    algorithms: Vec<Algorithm>,
}

Extracts token from header, validates signature and claims (exp, iss).

Provider Presets

Pre-configured verifier setups for common webhook sources:

pub enum WebhookProvider {
    GitHub,    // X-Hub-Signature-256, sha256= prefix, HMAC-SHA256
    Stripe,    // Stripe-Signature, t=...,v1=... format, HMAC-SHA256
    Slack,     // X-Slack-Signature, v0= prefix, HMAC-SHA256
    Custom,    // User-specified header + method
}

impl WebhookProvider {
    pub fn verifier(&self, secret: &[u8]) -> Box<dyn SignatureVerifier>;
}

Middleware Integration

The verifier runs as Axum middleware on HttpInputServer routes:

pub async fn webhook_signature_layer(
    verifier: Arc<dyn SignatureVerifier>,
    request: Request<Body>,
    next: Next,
) -> Response {
    // Extract signature header
    // Read body bytes (buffered for re-reading by handler)
    // Verify signature
    // 401 Unauthorized on failure, pass through on success
}

Secret Resolution

Webhook secrets support SecretStore references (e.g. secret: "secret://vault/github-webhook-secret") resolved at startup via the existing SecretStore trait in crates/runtime/src/secrets/.

5. DSL `webhook` Block

A new top-level block in the tree-sitter grammar for declaring webhook endpoints.

Grammar

webhook "github-events" {
    path     "/hooks/github"
    provider github
    secret   "secret://vault/github-webhook-secret"
    agent    code-review-agent
    filter {
        json_path "$.action"
        equals    "opened"
    }
}

Parsed Type

pub struct WebhookDefinition {
    pub name: String,
    pub path: String,
    pub provider: WebhookProvider,
    pub secret: String,              // Literal or secret:// reference
    pub agent: Option<String>,
    pub filter: Option<WebhookFilter>,
}

pub struct WebhookFilter {
    pub json_path: String,
    pub equals: Option<String>,
    pub contains: Option<String>,
}

Mapping to HttpInputConfig

Each WebhookDefinition compiles to an AgentRoutingRule in HttpInputConfig:

AgentRoutingRule {
    condition: RouteMatch::PathPrefix(webhook_def.path.clone()),
    agent: AgentId::new(&webhook_def.agent.unwrap_or_default()),
}

The signature verifier is attached as middleware on the route matching webhook_def.path.

Extraction

New function extract_webhook_definitions(tree: &Tree, source: &str) -> Result<Vec<WebhookDefinition>, String> following existing DSL extraction patterns.

6. CLI Commands

Memory Commands

symbi memory inspect <agent-id>     # Show memory.md contents
symbi memory search <agent-id> <query> [--limit N]  # Search via RAG
symbi memory compact <agent-id>     # Flush daily logs → memory.md
symbi memory purge <agent-id>       # Delete all memory for agent

inspect reads and pretty-prints data/agents/{agent_id}/memory.md.
search initializes StandardRAGEngine, calls process_query with source filter.
compact calls MarkdownMemoryStore::compact().
purge calls MarkdownMemoryStore::delete_context() with confirmation prompt.

Webhook Commands

symbi webhook add <name> --path /hooks/github --provider github --secret <secret>
symbi webhook list                   # Show configured webhooks
symbi webhook test <name> --payload '{"action":"opened"}'  # Simulate delivery
symbi webhook remove <name>          # Remove webhook definition
symbi webhook logs [--name <name>] [--tail N]  # View recent deliveries

add appends a webhook block to the agent’s .symbiont DSL file.
list parses DSL and prints webhook definitions in table format.
test sends a local HTTP request to the webhook path with the given payload, verifying signature and routing.
remove removes the named webhook block from DSL.
logs reads the audit log (when audit_enabled: true on HttpInputConfig).

Dependencies

Crate	Purpose	New?
`subtle`	Constant-time comparison for HMAC	Yes
`hmac` + `sha2`	HMAC-SHA256 computation	Yes
`humantime`	Parse duration strings (`90d`, `6months`)	Yes
`jsonwebtoken`	JWT verification	Existing (used in http_input)
`tempfile`	Atomic file writes	Existing

Files Changed

Area	Files
Memory store	`crates/runtime/src/context/markdown_memory.rs` (new)
Memory search	`crates/runtime/src/context/manager.rs` (extend)
Webhook middleware	`crates/runtime/src/http_input/webhook_verify.rs` (new)
Webhook providers	`crates/runtime/src/http_input/providers.rs` (new)
DSL grammar	`crates/dsl/grammar.js` (extend), `crates/dsl/src/lib.rs` (extend)
CLI	`crates/repl-cli/src/main.rs` or new CLI crate (extend)
Config	`crates/runtime/src/config.rs` (extend)

Non-Goals

Full memory compaction pipeline with summarization (just flush-to-markdown)
Standalone search engine (reuse RAG pipeline)
Event-driven webhook activation (extend existing HTTP input)
New HTTP server for webhooks (use existing HttpInputServer)

v1.4.0 Design: Persistent Memory + Webhook DX

Overview

1. MarkdownMemoryStore

File Layout

Markdown Format

Implementation

Retention & Compaction

Integration Point

2. Memory Search via RAG Pipeline

Indexing

Source-Filtered Search

Ranking

Index Maintenance

3. DSL memory Block

Grammar

Parsed Type

Extraction

4. Webhook Signature Verification

Verification Methods

Provider Presets

Middleware Integration

Secret Resolution

5. DSL webhook Block

Grammar

Parsed Type

Mapping to HttpInputConfig

Extraction

6. CLI Commands

Memory Commands

Webhook Commands

Dependencies

Files Changed

Non-Goals

3. DSL `memory` Block

5. DSL `webhook` Block