rsspipe/docs/004_variant_pipeline_architecture.md

RSSPipe Variant Data & Extensible Processing Pipeline Architecture

1. Overview & Vision

RSSPipe is designed as an RSS-Driven Information Gateway. Subscriptions come in many formats (full-text blogs, brief news tickers, media podcasts, academic papers). Consequently, a one-size-fits-all processing approach fails:

• Some feeds require automatic full-text extraction via Readability.
• Some feeds require LLM keyword and topic tagging.
• Specific high-value posts (filtered by keywords or feeds) require AI summarization, translation, or Text-to-Speech (TTS) audio synthesis.

To support this cleanly without database corruption, performance degradation, or rigid schemas, RSSPipe adopts a Variant Data Architecture paired with a Reactive Chained Trigger Pipeline.

---

2. Core Data Model

2.1 Separation of Immutable Source vs. Enriched Variants

1. feed_entries (Immutable Source of Truth):
   • Stores the exact metadata and content parsed from the RSS XML feed.
   • Core fields: id, feed_id, guid, url, title, author, rss_content, published_at, is_read, is_starred, created_at.
   • Rule: rss_content is NEVER overwritten by scrapers, LLMs, or TTS engines.
2. entry_variants (Extensible Key-Value Store):

   • Stores all downstream enrichments as strongly typed variant entries.

   • Schema:

     CREATE TABLE IF NOT EXISTS entry_variants (
         entry_id TEXT NOT NULL,
         variant_key TEXT NOT NULL,
         content_type TEXT NOT NULL, -- 'text', 'html', 'json', 'yaml', 'audio/mp3', 'binary'
         text_value TEXT,            -- UTF-8 text, HTML, JSON array/object, YAML string
         blob_path TEXT,             -- Relative filesystem path for large media (e.g. data/blobs/{entry_id}/{key}.mp3)
         metadata JSON DEFAULT '{}',  -- Stats: model, tokens, duration_ms, byte_size, executed_at
         created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
         updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
         PRIMARY KEY (entry_id, variant_key),
         FOREIGN KEY (entry_id) REFERENCES feed_entries(id) ON DELETE CASCADE
     );
     CREATE INDEX IF NOT EXISTS idx_entry_variants_key ON entry_variants(variant_key);
     

---

3. Supported Variant Keys & Types

Variant Key	Content Type	Storage Location	Description
readability_full_text	html	text_value	Sanitized readability full text extracted from the source web page.
keywords	json / yaml	text_value	List of extracted keywords (e.g., ["AI", "Go", "Compiler"]).
tags	json	text_value	Category and topic tags.
ai_summary	text / markdown	text_value	Bulleted or narrative AI executive summary.
translation_{lang}	html / text	text_value	Translated content (e.g. translation_zh).
tts	audio/mp3	blob_path	Synthesized speech audio file of the article.
podcast_tts	audio/mp3	blob_path	Multi-speaker podcast discussion audio file.
ai_summary_tts	audio/mp3	blob_path	Audio narration of the AI summary.

---

4. Large Binary Blob Strategy

Storing multi-megabyte audio files or binary blobs directly inside SQLite columns degrades query cache efficiency and slows down table scans.

• Text & JSON Variants are stored inline in entry_variants.text_value.
• Audio & Media Blobs are stored on disk under ./data/blobs/{entry_id}/{variant_key}.{ext}.
• entry_variants.blob_path stores the relative path.
• The API provides a streaming endpoint with HTTP Range support: GET /api/entries/{id}/variants/{key}/stream.

---

5. Reactive Chained Trigger & Watch Pipeline

The pipeline operates on typed pub/sub events with depth guards to prevent infinite loops:

graph TD
    A[Feed Ingestion / Crawler] -->|Event: on_post_created| Bus[Typed Event Bus]
    Bus --> Rules[Rules Engine Evaluator]
    
    Rules -->|Matches Rule 1: Scrape Tech| W1[Worker: Readability Scraper]
    W1 -->|Writes variant: readability_full_text| DB[(entry_variants)]
    W1 -->|Event: on_variant_updated| Bus
    
    Rules -->|Matches Rule 2: Tag AI Keywords| W2[Worker: Keyword Extractor]
    W2 -->|Writes variant: keywords| DB
    W2 -->|Event: on_variant_updated| Bus
    
    Bus -->|Matches Rule 3: Watch keywords contains 'AI'| W3[Worker: AI Summary & TTS]
    W3 -->|Writes variant: ai_summary & tts blob| DB

5.1 Concrete Chained Pipeline Example

1. Trigger 1: New post ingested from Hacker News -> Emits on_post_created.
2. Rule 1 (Auto-Enrich Keywords):
   • Condition: feed_id == "hn-feed-id"
   • Action: extract_keywords -> Saves entry_variants(key='keywords', text_value='["OpenAI", "LLM", "Agent"]').
   • Emits: on_variant_updated(key='keywords').
3. Rule 2 (Watch & Synthesize Audio):
   • Condition: variant.keywords CONTAINS "OpenAI"
   • Action: generate_tts -> Synthesizes MP3, saves to ./data/blobs/..., saves variant tts.
   • Emits: on_variant_updated(key='tts').

---

6. Reader UX & Content Switcher

In the 3-pane reader interface:

• Default View: Renders rss_content (original source).
• Enrichment Pills: If variants exist on the entry, dynamic tabs/pills appear in the reader header:
  • [📄 Original RSS]
  • [⚡ Readability Full-Text] (if readability_full_text exists)
  • [💡 AI Summary] (if ai_summary exists)
  • [🎧 Listen Audio] (if tts or podcast_tts exists, plays with embedded HTML5 player)
• Switching variants never mutates the underlying article or overwrites database fields.