117 lines
5.5 KiB
Markdown
117 lines
5.5 KiB
Markdown
# RSSPipe Variant Data & Extensible Processing Pipeline Architecture
|
|
|
|
## 1. Overview & Vision
|
|
|
|
RSSPipe is designed as an **RSS-Driven Information Gateway**. Subscriptions come in many formats (full-text blogs, brief news tickers, media podcasts, academic papers). Consequently, a one-size-fits-all processing approach fails:
|
|
|
|
- Some feeds require automatic full-text extraction via Readability.
|
|
- Some feeds require LLM keyword and topic tagging.
|
|
- Specific high-value posts (filtered by keywords or feeds) require AI summarization, translation, or Text-to-Speech (TTS) audio synthesis.
|
|
|
|
To support this cleanly without database corruption, performance degradation, or rigid schemas, RSSPipe adopts a **Variant Data Architecture** paired with a **Reactive Chained Trigger Pipeline**.
|
|
|
|
---
|
|
|
|
## 2. Core Data Model
|
|
|
|
### 2.1 Separation of Immutable Source vs. Enriched Variants
|
|
|
|
1. **`feed_entries` (Immutable Source of Truth)**:
|
|
- Stores the exact metadata and content parsed from the RSS XML feed.
|
|
- Core fields: `id`, `feed_id`, `guid`, `url`, `title`, `author`, `rss_content`, `published_at`, `is_read`, `is_starred`, `created_at`.
|
|
- **Rule**: `rss_content` is NEVER overwritten by scrapers, LLMs, or TTS engines.
|
|
2. **`entry_variants` (Extensible Key-Value Store)**:
|
|
- Stores all downstream enrichments as strongly typed variant entries.
|
|
- Schema:
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS entry_variants (
|
|
entry_id TEXT NOT NULL,
|
|
variant_key TEXT NOT NULL,
|
|
content_type TEXT NOT NULL, -- 'text', 'html', 'json', 'yaml', 'audio/mp3', 'binary'
|
|
text_value TEXT, -- UTF-8 text, HTML, JSON array/object, YAML string
|
|
blob_path TEXT, -- Relative filesystem path for large media (e.g. data/blobs/{entry_id}/{key}.mp3)
|
|
metadata JSON DEFAULT '{}', -- Stats: model, tokens, duration_ms, byte_size, executed_at
|
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
|
PRIMARY KEY (entry_id, variant_key),
|
|
FOREIGN KEY (entry_id) REFERENCES feed_entries(id) ON DELETE CASCADE
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_entry_variants_key ON entry_variants(variant_key);
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Supported Variant Keys & Types
|
|
|
|
| Variant Key | Content Type | Storage Location | Description |
|
|
| :--- | :--- | :--- | :--- |
|
|
| `readability_full_text` | `html` | `text_value` | Sanitized readability full text extracted from the source web page. |
|
|
| `keywords` | `json` / `yaml` | `text_value` | List of extracted keywords (e.g., `["AI", "Go", "Compiler"]`). |
|
|
| `tags` | `json` | `text_value` | Category and topic tags. |
|
|
| `ai_summary` | `text` / `markdown` | `text_value` | Bulleted or narrative AI executive summary. |
|
|
| `translation_{lang}` | `html` / `text` | `text_value` | Translated content (e.g. `translation_zh`). |
|
|
| `tts` | `audio/mp3` | `blob_path` | Synthesized speech audio file of the article. |
|
|
| `podcast_tts` | `audio/mp3` | `blob_path` | Multi-speaker podcast discussion audio file. |
|
|
| `ai_summary_tts` | `audio/mp3` | `blob_path` | Audio narration of the AI summary. |
|
|
|
|
---
|
|
|
|
## 4. Large Binary Blob Strategy
|
|
|
|
Storing multi-megabyte audio files or binary blobs directly inside SQLite columns degrades query cache efficiency and slows down table scans.
|
|
|
|
- **Text & JSON Variants** are stored inline in `entry_variants.text_value`.
|
|
- **Audio & Media Blobs** are stored on disk under `./data/blobs/{entry_id}/{variant_key}.{ext}`.
|
|
- `entry_variants.blob_path` stores the relative path.
|
|
- The API provides a streaming endpoint with HTTP Range support: `GET /api/entries/{id}/variants/{key}/stream`.
|
|
|
|
---
|
|
|
|
## 5. Reactive Chained Trigger & Watch Pipeline
|
|
|
|
The pipeline operates on typed pub/sub events with depth guards to prevent infinite loops:
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Feed Ingestion / Crawler] -->|Event: on_post_created| Bus[Typed Event Bus]
|
|
Bus --> Rules[Rules Engine Evaluator]
|
|
|
|
Rules -->|Matches Rule 1: Scrape Tech| W1[Worker: Readability Scraper]
|
|
W1 -->|Writes variant: readability_full_text| DB[(entry_variants)]
|
|
W1 -->|Event: on_variant_updated| Bus
|
|
|
|
Rules -->|Matches Rule 2: Tag AI Keywords| W2[Worker: Keyword Extractor]
|
|
W2 -->|Writes variant: keywords| DB
|
|
W2 -->|Event: on_variant_updated| Bus
|
|
|
|
Bus -->|Matches Rule 3: Watch keywords contains 'AI'| W3[Worker: AI Summary & TTS]
|
|
W3 -->|Writes variant: ai_summary & tts blob| DB
|
|
```
|
|
|
|
### 5.1 Concrete Chained Pipeline Example
|
|
|
|
1. **Trigger 1**: New post ingested from Hacker News -> Emits `on_post_created`.
|
|
2. **Rule 1 (Auto-Enrich Keywords)**:
|
|
- Condition: `feed_id == "hn-feed-id"`
|
|
- Action: `extract_keywords` -> Saves `entry_variants(key='keywords', text_value='["OpenAI", "LLM", "Agent"]')`.
|
|
- Emits: `on_variant_updated(key='keywords')`.
|
|
3. **Rule 2 (Watch & Synthesize Audio)**:
|
|
- Condition: `variant.keywords CONTAINS "OpenAI"`
|
|
- Action: `generate_tts` -> Synthesizes MP3, saves to `./data/blobs/...`, saves variant `tts`.
|
|
- Emits: `on_variant_updated(key='tts')`.
|
|
|
|
---
|
|
|
|
## 6. Reader UX & Content Switcher
|
|
|
|
In the 3-pane reader interface:
|
|
|
|
- **Default View**: Renders `rss_content` (original source).
|
|
- **Enrichment Pills**: If variants exist on the entry, dynamic tabs/pills appear in the reader header:
|
|
- `[📄 Original RSS]`
|
|
- `[⚡ Readability Full-Text]` (if `readability_full_text` exists)
|
|
- `[💡 AI Summary]` (if `ai_summary` exists)
|
|
- `[🎧 Listen Audio]` (if `tts` or `podcast_tts` exists, plays with embedded HTML5 player)
|
|
- Switching variants never mutates the underlying article or overwrites database fields.
|