# RSSPipe Variant Data & Extensible Processing Pipeline Architecture

## 1. Overview & Vision

RSSPipe is designed as an **RSS-Driven Information Gateway**. Subscriptions come in many formats (full-text blogs, brief news tickers, media podcasts, academic papers). Consequently, a one-size-fits-all processing approach fails:

- Some feeds require automatic full-text extraction via Readability.
- Some feeds require LLM keyword and topic tagging.
- Specific high-value posts (filtered by keywords or feeds) require AI summarization, translation, or Text-to-Speech (TTS) audio synthesis.

To support this cleanly without database corruption, performance degradation, or rigid schemas, RSSPipe adopts a **Variant Data Architecture** paired with a **Reactive Chained Trigger Pipeline**.

---

## 2. Core Data Model

### 2.1 Separation of Immutable Source vs. Enriched Variants

1. **`feed_entries` (Immutable Source of Truth)**:
   - Stores the exact metadata and content parsed from the RSS XML feed.
   - Core fields: `id`, `feed_id`, `guid`, `url`, `title`, `author`, `rss_content`, `published_at`, `is_read`, `is_starred`, `created_at`.
   - **Rule**: `rss_content` is NEVER overwritten by scrapers, LLMs, or TTS engines.
2. **`entry_variants` (Extensible Key-Value Store)**:
   - Stores all downstream enrichments as strongly typed variant entries.
   - Schema:

     ```sql
     CREATE TABLE IF NOT EXISTS entry_variants (
         entry_id TEXT NOT NULL,
         variant_key TEXT NOT NULL,
         content_type TEXT NOT NULL, -- 'text', 'html', 'json', 'yaml', 'audio/mp3', 'binary'
         text_value TEXT,            -- UTF-8 text, HTML, JSON array/object, YAML string
         blob_path TEXT,             -- Relative filesystem path for large media (e.g. data/blobs/{entry_id}/{key}.mp3)
         metadata JSON DEFAULT '{}',  -- Stats: model, tokens, duration_ms, byte_size, executed_at
         created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
         updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
         PRIMARY KEY (entry_id, variant_key),
         FOREIGN KEY (entry_id) REFERENCES feed_entries(id) ON DELETE CASCADE
     );
     CREATE INDEX IF NOT EXISTS idx_entry_variants_key ON entry_variants(variant_key);
     ```

---

## 3. Supported Variant Keys & Types

| Variant Key | Content Type | Storage Location | Description |
| :--- | :--- | :--- | :--- |
| `readability_full_text` | `html` | `text_value` | Sanitized readability full text extracted from the source web page. |
| `keywords` | `json` / `yaml` | `text_value` | List of extracted keywords (e.g., `["AI", "Go", "Compiler"]`). |
| `tags` | `json` | `text_value` | Category and topic tags. |
| `ai_summary` | `text` / `markdown` | `text_value` | Bulleted or narrative AI executive summary. |
| `translation_{lang}` | `html` / `text` | `text_value` | Translated content (e.g. `translation_zh`). |
| `tts` | `audio/mp3` | `blob_path` | Synthesized speech audio file of the article. |
| `podcast_tts` | `audio/mp3` | `blob_path` | Multi-speaker podcast discussion audio file. |
| `ai_summary_tts` | `audio/mp3` | `blob_path` | Audio narration of the AI summary. |

---

## 4. Large Binary Blob Strategy

Storing multi-megabyte audio files or binary blobs directly inside SQLite columns degrades query cache efficiency and slows down table scans.

- **Text & JSON Variants** are stored inline in `entry_variants.text_value`.
- **Audio & Media Blobs** are stored on disk under `./data/blobs/{entry_id}/{variant_key}.{ext}`.
- `entry_variants.blob_path` stores the relative path.
- The API provides a streaming endpoint with HTTP Range support: `GET /api/entries/{id}/variants/{key}/stream`.

---

## 5. Reactive Chained Trigger & Watch Pipeline

The pipeline operates on typed pub/sub events with depth guards to prevent infinite loops:

```mermaid
graph TD
    A[Feed Ingestion / Crawler] -->|Event: on_post_created| Bus[Typed Event Bus]
    Bus --> Rules[Rules Engine Evaluator]
    
    Rules -->|Matches Rule 1: Scrape Tech| W1[Worker: Readability Scraper]
    W1 -->|Writes variant: readability_full_text| DB[(entry_variants)]
    W1 -->|Event: on_variant_updated| Bus
    
    Rules -->|Matches Rule 2: Tag AI Keywords| W2[Worker: Keyword Extractor]
    W2 -->|Writes variant: keywords| DB
    W2 -->|Event: on_variant_updated| Bus
    
    Bus -->|Matches Rule 3: Watch keywords contains 'AI'| W3[Worker: AI Summary & TTS]
    W3 -->|Writes variant: ai_summary & tts blob| DB
```

### 5.1 Concrete Chained Pipeline Example

1. **Trigger 1**: New post ingested from Hacker News -> Emits `on_post_created`.
2. **Rule 1 (Auto-Enrich Keywords)**:
   - Condition: `feed_id == "hn-feed-id"`
   - Action: `extract_keywords` -> Saves `entry_variants(key='keywords', text_value='["OpenAI", "LLM", "Agent"]')`.
   - Emits: `on_variant_updated(key='keywords')`.
3. **Rule 2 (Watch & Synthesize Audio)**:
   - Condition: `variant.keywords CONTAINS "OpenAI"`
   - Action: `generate_tts` -> Synthesizes MP3, saves to `./data/blobs/...`, saves variant `tts`.
   - Emits: `on_variant_updated(key='tts')`.

---

## 6. Reader UX & Content Switcher

In the 3-pane reader interface:

- **Default View**: Renders `rss_content` (original source).
- **Enrichment Pills**: If variants exist on the entry, dynamic tabs/pills appear in the reader header:
  - `[📄 Original RSS]`
  - `[⚡ Readability Full-Text]` (if `readability_full_text` exists)
  - `[💡 AI Summary]` (if `ai_summary` exists)
  - `[🎧 Listen Audio]` (if `tts` or `podcast_tts` exists, plays with embedded HTML5 player)
- Switching variants never mutates the underlying article or overwrites database fields.