feat: implement rule engine, worker pool, event bus, and readability scraping with enhanced logging and content sanitization.
This commit is contained in:
@@ -93,3 +93,42 @@ This document records planned features, UX enhancements, and architectural exten
|
||||
- Staggered dynamic aspect ratios with prominent media cards.
|
||||
- Smooth infinite scroll pagination.
|
||||
- Interactive gesture swipe to dismiss the full-screen reader.
|
||||
|
||||
---
|
||||
|
||||
## 10. Beautiful 404 Error Page
|
||||
|
||||
- **Goal**: Provide a polished, modern, brand-consistent 404 Not Found experience.
|
||||
- **Scope**:
|
||||
- Frontend 404 route component with sleek modern aesthetics (glassmorphism/minimal illustration, clear error description, and a "Return to Gateway" action button).
|
||||
- Server-side fallback handler returning a clean, branded HTML response for unmatched routes.
|
||||
|
||||
---
|
||||
|
||||
## 11. Multi-Variant Pipeline Enrichments (AI Summary, TTS, Keywords, Audio)
|
||||
|
||||
- **Goal**: Expand the event-driven rules engine and variant storage system with automated enrichment plugins.
|
||||
- **Scope**:
|
||||
- **AI Summary Hook**: Integrate LLM actions (e.g. OpenAI / Anthropic / Gemini / Ollama) to automatically generate and store concise summaries in `entry_variants(key='ai_summary')`.
|
||||
- **Keyword & Tag Extraction**: Automated extractor storing structured YAML/JSON keyword lists in `entry_variants(key='keywords')` and `entry_variants(key='tags')` to enable rule-based chaining.
|
||||
- **Text-to-Speech (TTS)**: Synthesize article text to voice audio blobs (`.mp3` stored under `./data/blobs/{entry_id}/tts.mp3`) with HTTP 206 partial streaming.
|
||||
- **Podcast TTS & Summary TTS**: Convert articles into dual-host conversational podcasts or audio-narrated summaries streamed directly within the reader.
|
||||
|
||||
---
|
||||
|
||||
## 12. Aggregated Shared Posts Dashboard (`/shares`)
|
||||
|
||||
- **Goal**: Provide a dedicated gallery view aggregating all publicly shared articles.
|
||||
- **Scope**:
|
||||
- Centralized dashboard listing all articles with an active `share_uuid`.
|
||||
- Copy public link, preview shared HTML canvas, revoke share link, and view access stats.
|
||||
|
||||
---
|
||||
|
||||
## 13. Data Retention & Cleanup Policies (Auto-Deletion)
|
||||
|
||||
- **Goal**: Provide configurable retention rules to prune older or noisy articles and keep SQLite storage compact.
|
||||
- **Scope**:
|
||||
- Retention triggers (e.g. delete unstarred articles older than 90 days, or limit feeds to latest 500 posts).
|
||||
- Background garbage collection cleaning associated `entry_variants` and on-disk audio blobs (`./data/blobs/{entry_id}/`).
|
||||
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# RSSPipe Variant Data & Extensible Processing Pipeline Architecture
|
||||
|
||||
## 1. Overview & Vision
|
||||
|
||||
RSSPipe is designed as an **RSS-Driven Information Gateway**. Subscriptions come in many formats (full-text blogs, brief news tickers, media podcasts, academic papers). Consequently, a one-size-fits-all processing approach fails:
|
||||
|
||||
- Some feeds require automatic full-text extraction via Readability.
|
||||
- Some feeds require LLM keyword and topic tagging.
|
||||
- Specific high-value posts (filtered by keywords or feeds) require AI summarization, translation, or Text-to-Speech (TTS) audio synthesis.
|
||||
|
||||
To support this cleanly without database corruption, performance degradation, or rigid schemas, RSSPipe adopts a **Variant Data Architecture** paired with a **Reactive Chained Trigger Pipeline**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Core Data Model
|
||||
|
||||
### 2.1 Separation of Immutable Source vs. Enriched Variants
|
||||
|
||||
1. **`feed_entries` (Immutable Source of Truth)**:
|
||||
- Stores the exact metadata and content parsed from the RSS XML feed.
|
||||
- Core fields: `id`, `feed_id`, `guid`, `url`, `title`, `author`, `rss_content`, `published_at`, `is_read`, `is_starred`, `created_at`.
|
||||
- **Rule**: `rss_content` is NEVER overwritten by scrapers, LLMs, or TTS engines.
|
||||
2. **`entry_variants` (Extensible Key-Value Store)**:
|
||||
- Stores all downstream enrichments as strongly typed variant entries.
|
||||
- Schema:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS entry_variants (
|
||||
entry_id TEXT NOT NULL,
|
||||
variant_key TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL, -- 'text', 'html', 'json', 'yaml', 'audio/mp3', 'binary'
|
||||
text_value TEXT, -- UTF-8 text, HTML, JSON array/object, YAML string
|
||||
blob_path TEXT, -- Relative filesystem path for large media (e.g. data/blobs/{entry_id}/{key}.mp3)
|
||||
metadata JSON DEFAULT '{}', -- Stats: model, tokens, duration_ms, byte_size, executed_at
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (entry_id, variant_key),
|
||||
FOREIGN KEY (entry_id) REFERENCES feed_entries(id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_entry_variants_key ON entry_variants(variant_key);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Supported Variant Keys & Types
|
||||
|
||||
| Variant Key | Content Type | Storage Location | Description |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `readability_full_text` | `html` | `text_value` | Sanitized readability full text extracted from the source web page. |
|
||||
| `keywords` | `json` / `yaml` | `text_value` | List of extracted keywords (e.g., `["AI", "Go", "Compiler"]`). |
|
||||
| `tags` | `json` | `text_value` | Category and topic tags. |
|
||||
| `ai_summary` | `text` / `markdown` | `text_value` | Bulleted or narrative AI executive summary. |
|
||||
| `translation_{lang}` | `html` / `text` | `text_value` | Translated content (e.g. `translation_zh`). |
|
||||
| `tts` | `audio/mp3` | `blob_path` | Synthesized speech audio file of the article. |
|
||||
| `podcast_tts` | `audio/mp3` | `blob_path` | Multi-speaker podcast discussion audio file. |
|
||||
| `ai_summary_tts` | `audio/mp3` | `blob_path` | Audio narration of the AI summary. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Large Binary Blob Strategy
|
||||
|
||||
Storing multi-megabyte audio files or binary blobs directly inside SQLite columns degrades query cache efficiency and slows down table scans.
|
||||
|
||||
- **Text & JSON Variants** are stored inline in `entry_variants.text_value`.
|
||||
- **Audio & Media Blobs** are stored on disk under `./data/blobs/{entry_id}/{variant_key}.{ext}`.
|
||||
- `entry_variants.blob_path` stores the relative path.
|
||||
- The API provides a streaming endpoint with HTTP Range support: `GET /api/entries/{id}/variants/{key}/stream`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Reactive Chained Trigger & Watch Pipeline
|
||||
|
||||
The pipeline operates on typed pub/sub events with depth guards to prevent infinite loops:
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Feed Ingestion / Crawler] -->|Event: on_post_created| Bus[Typed Event Bus]
|
||||
Bus --> Rules[Rules Engine Evaluator]
|
||||
|
||||
Rules -->|Matches Rule 1: Scrape Tech| W1[Worker: Readability Scraper]
|
||||
W1 -->|Writes variant: readability_full_text| DB[(entry_variants)]
|
||||
W1 -->|Event: on_variant_updated| Bus
|
||||
|
||||
Rules -->|Matches Rule 2: Tag AI Keywords| W2[Worker: Keyword Extractor]
|
||||
W2 -->|Writes variant: keywords| DB
|
||||
W2 -->|Event: on_variant_updated| Bus
|
||||
|
||||
Bus -->|Matches Rule 3: Watch keywords contains 'AI'| W3[Worker: AI Summary & TTS]
|
||||
W3 -->|Writes variant: ai_summary & tts blob| DB
|
||||
```
|
||||
|
||||
### 5.1 Concrete Chained Pipeline Example
|
||||
|
||||
1. **Trigger 1**: New post ingested from Hacker News -> Emits `on_post_created`.
|
||||
2. **Rule 1 (Auto-Enrich Keywords)**:
|
||||
- Condition: `feed_id == "hn-feed-id"`
|
||||
- Action: `extract_keywords` -> Saves `entry_variants(key='keywords', text_value='["OpenAI", "LLM", "Agent"]')`.
|
||||
- Emits: `on_variant_updated(key='keywords')`.
|
||||
3. **Rule 2 (Watch & Synthesize Audio)**:
|
||||
- Condition: `variant.keywords CONTAINS "OpenAI"`
|
||||
- Action: `generate_tts` -> Synthesizes MP3, saves to `./data/blobs/...`, saves variant `tts`.
|
||||
- Emits: `on_variant_updated(key='tts')`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reader UX & Content Switcher
|
||||
|
||||
In the 3-pane reader interface:
|
||||
|
||||
- **Default View**: Renders `rss_content` (original source).
|
||||
- **Enrichment Pills**: If variants exist on the entry, dynamic tabs/pills appear in the reader header:
|
||||
- `[📄 Original RSS]`
|
||||
- `[⚡ Readability Full-Text]` (if `readability_full_text` exists)
|
||||
- `[💡 AI Summary]` (if `ai_summary` exists)
|
||||
- `[🎧 Listen Audio]` (if `tts` or `podcast_tts` exists, plays with embedded HTML5 player)
|
||||
- Switching variants never mutates the underlying article or overwrites database fields.
|
||||
Reference in New Issue
Block a user