514 lines
30 KiB
Markdown
514 lines
30 KiB
Markdown
# RSSPipe: Product Requirements Document (PRD)
|
||
|
||
> **Document Version:** 1.0.0
|
||
> **Status:** Approved / Ready for Implementation
|
||
> **Target Audience:** Engineering, Architecture, Product
|
||
> **Replaces:** Miniflux Deployment
|
||
> **Deployment Target:** k3s / Kubernetes (Single Container Architecture with Separate or Embedded DB)
|
||
|
||
---
|
||
|
||
## 1. Executive Summary & Product Vision
|
||
|
||
### 1.1 Overview
|
||
|
||
**RSSPipe** is a self-hosted, RSS-driven **Information Gateway** and intelligent reading platform built to replace traditional RSS aggregators (such as Miniflux).
|
||
|
||
Rather than serving purely as a passive feed bucket, RSSPipe combines:
|
||
|
||
1. **Google Reader & Gmail UX Philosophy**: Rapid, keyboard-first triaging (`j`/`k`, `o`, `v`, `s`, `e`), "Inbox Zero" workflow, dual view modes (expanded scrolling stream vs. compact list/split-pane), and Gmail-style search syntax.
|
||
2. **Dual-Tier Feed Hooks Engine**:
|
||
- **Single-Feed Enrichment**: Real-time readability full-text scraping, AI summarization/takeaways, relevance scoring, auto-tagging, TTS generation, and automatic public share-page publishing.
|
||
- **Multi-Feed AI Digest Synthesis**: Scheduled or threshold-based multi-document synthesis that rolls up unread items into clean "Virtual Digest Feeds" (e.g., Daily Morning Briefing, Topic Radar) with synthesized audio podcasts, automatically archiving raw individual items.
|
||
3. **Single-User Security with Public Sharing**: Single admin user model protecting the private gateway, with frictionless unauthenticated public sharing via tokenized URLs (`/share/<uuid>.html`).
|
||
4. **Offline-Capable PWA & Native Ecosystem Support**: Installable Progressive Web App (PWA) with responsive mobile/desktop interfaces, alongside a Fever / Google Reader API compatibility layer for native mobile clients (e.g., NetNewsWire, Reeder).
|
||
5. **Unified K3s-Ready Architecture**: High-performance Go backend with embedded React/Vite PWA frontend, packaged as a single lightweight Docker container targeting k3s/Kubernetes.
|
||
|
||
---
|
||
|
||
## 2. Core Architecture & System Tenets
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────┐
|
||
│ RSSPipe Web App │
|
||
│ (React + Vite PWA / Google Reader & Gmail UX) │
|
||
└───────────┬────────────────────────────────┬───────────┘
|
||
│ REST / SSE Events │ Fever / GReader API
|
||
▼ ▼
|
||
┌───────────────────────────────────────────────────────────────────────┐
|
||
│ Go Unified Server │
|
||
│ │
|
||
│ ┌────────────────────┐ ┌────────────────────┐ ┌─────────────────┐ │
|
||
│ │ Auth & Session │ │ Ingestion Pool │ │ Public Share │ │
|
||
│ │ (Single Admin) │ │ (Cron & Web Crawl) │ │ (/share/<uuid>) │ │
|
||
│ └────────────────────┘ └─────────┬──────────┘ └─────────────────┘ │
|
||
│ ▼ │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ Rules & Hook Engine (DAG) │ │
|
||
│ │ ┌───────────────────────────────┐ ┌──────────────────────────┐ │ │
|
||
│ │ │ Single-Feed Enrichment │ │ Multi-Feed Aggregation │ │ │
|
||
│ │ │ (Readability/LLM/TTS/Webhook) │ │ (Digest Synthesis/Audio) │ │ │
|
||
│ │ └───────────────────────────────┘ └──────────────────────────┘ │ │
|
||
│ └─────────────────────────────────┬───────────────────────────────┘ │
|
||
│ ▼ │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ Data Layer (PostgreSQL / SQLite WAL + FTS5 Full-Text) │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
└───────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 2.1 Design Tenets
|
||
|
||
1. **Speed over Friction**: Sub-100ms UI responses, instant keyboard navigation, and optimistic UI updates.
|
||
2. **Information High Signal-to-Noise**: Raw RSS feeds are enriched or distilled into digestible summaries and podcasts so the user stays informed without drowning in unread counts.
|
||
3. **Zero Maintenance & Single Container Simplicity**: Single binary/image containing backend server, worker pools, and embedded web assets, deployable via simple Helm chart / Kubernetes manifests on k3s.
|
||
4. **Cycle-Safe Rule Processing**: All feed transformation steps follow a strict Directed Acyclic Graph (DAG) with topological validation to prevent infinite execution loops.
|
||
|
||
---
|
||
|
||
## 3. End-to-End User Journeys
|
||
|
||
```mermaid
|
||
journey
|
||
title Core User Journeys in RSSPipe
|
||
section 1. Onboarding & Migration
|
||
Import OPML from Miniflux: 5: Admin
|
||
Auto-categorize and test feeds: 4: Admin
|
||
section 2. Daily Triaging (Google Reader / Gmail UX)
|
||
Open PWA (Desktop/Mobile): 5: Admin
|
||
Navigate feeds with 'j'/'k' and expand with 'o': 5: Admin
|
||
Quick triage with 'e' (archive/read) and 's' (star): 5: Admin
|
||
Search with operators 'is:unread tag:ai': 5: Admin
|
||
section 3. Feed Hooks & Enrichment
|
||
Ingest feed & scrape full-text via Readability: 5: System
|
||
LLM summarizes & scores relevance: 5: System
|
||
Generate TTS audio preview: 4: System
|
||
section 4. Multi-Feed Digest Generation
|
||
Accumulate > 15 topic feeds or reach 8:00 AM: 5: System
|
||
Synthesize AI Daily Digest & Audio Podcast: 5: System
|
||
Publish to Virtual Digest Feed & mark raw items read: 5: System
|
||
section 5. Knowledge Sharing
|
||
Click 'Share to Public' on article/digest: 5: Admin
|
||
External visitor views /share/<uuid>.html: 5: Public Reader
|
||
```
|
||
|
||
### Journey 1: One-Click Migration from Miniflux (Onboarding)
|
||
|
||
- **Goal**: Seamlessly transition existing feeds, categories, and folder hierarchies from Miniflux to RSSPipe in seconds.
|
||
- **Flow**:
|
||
1. User navigates to `Settings -> Migration & OPML`.
|
||
2. User uploads Miniflux `subscriptions.opml`.
|
||
3. System parses categories and feed URLs, tests connection concurrency in the background, and preserves category folder tree.
|
||
4. System prompts option to backfill recent posts (last N days) or start fresh from latest entries.
|
||
5. Dashboard immediately renders feeds categorized and organized ready for consumption.
|
||
|
||
### Journey 2: Morning Triage (Google Reader & Gmail Workflow)
|
||
|
||
- **Goal**: Rapidly process 200+ unread items in under 5 minutes using keyboard shortcuts and split views.
|
||
- **Flow**:
|
||
1. User opens RSSPipe PWA. The "Inbox Zero" view displays all unread entries grouped by priority or chronological stream.
|
||
2. User toggles view:
|
||
- **Expanded Stream (Google Reader)**: Full article body inline, scrolling continuously with `Space` or `j`/`k`.
|
||
- **Compact List (Gmail)**: Single-line item rows displaying Source, Star, Title, AI Summary badge, and Time.
|
||
3. User presses `j` to focus the next item, `o` or `Enter` to expand/collapse.
|
||
4. User presses `s` to star for deep reading later, `v` to open original link in a new tab, `m` to toggle read/unread, or `e` to archive/mark read.
|
||
5. User presses `/` to activate search and types `is:unread tag:ai score:>80` to filter high-impact items instantly.
|
||
|
||
### Journey 3: Single-Feed Automated Enrichment Hook
|
||
|
||
- **Goal**: Automatically convert truncated RSS feeds into readable, summarized, and categorized entries with voice synthesis.
|
||
- **Flow**:
|
||
1. Ingestion engine fetches a newly published feed item.
|
||
2. **Content Extraction**: System detects truncated content and runs the backend Readability scraper to fetch the full clean article markdown.
|
||
3. **AI Enrichment**: Configured rule triggers LLM prompt:
|
||
- Generates a 3-bullet executive summary.
|
||
- Assigns relevant topic tags (`#llm`, `#k8s`).
|
||
- Scores article relevance from 1 to 100 based on user interest profile.
|
||
4. **TTS Audio Hook**: High-scoring items automatically trigger Edge-TTS / OpenAI TTS generation, attaching an audio player to the feed header.
|
||
5. **Notification Hook**: If relevance `score >= 90`, sends a webhook alert to user's Telegram/Discord channel with the summary and direct link.
|
||
|
||
### Journey 4: Multi-Feed AI Digest Synthesis & "Inbox Zero" Auto-Rollup
|
||
|
||
- **Goal**: Eliminate RSS fatigue by condensing tens of high-volume feeds into a single curated morning/evening brief.
|
||
- **Flow**:
|
||
1. User sets a Multi-Feed Aggregation Rule: *"Every day at 7:30 AM (or whenever unread items in #tech exceed 20), synthesize a 5-minute Executive Digest and mark source items as read."*
|
||
2. At the scheduled time / threshold, the Multi-Feed Engine queries all eligible unread items.
|
||
3. The Engine passes all article snippets/full-text into the LLM with a structured synthesis prompt:
|
||
- Grouping into major themes / top headlines.
|
||
- Highlighting key takeaways, contradictions, or trends across sources.
|
||
- Linking back to source items.
|
||
4. An AI TTS audio briefing is synthesized and attached.
|
||
5. The resulting synthesis is injected as a new entry into a **Virtual Digest Feed** (e.g., `📰 Daily Tech Executive Briefing`).
|
||
6. The 20+ raw underlying source feeds are automatically marked as read / archived.
|
||
7. When user opens the PWA, they see 1 high-quality digest episode instead of 20 fragmented articles.
|
||
|
||
### Journey 5: Knowledge Sharing to External Visitors
|
||
|
||
- **Goal**: Share an insightful article or AI digest with friends, colleagues, or social followers without granting access to the private RSS reader.
|
||
- **Flow**:
|
||
1. In the PWA reader view, user clicks the **"Share to Public"** icon (or an automated rule publishes high-scoring digests automatically).
|
||
2. Backend generates an immutable, non-guessable UUID share route: `https://rss.domain.com/share/550e8400-e29b-41d4-a716-446655440000.html`.
|
||
3. The generated page is a standalone, beautifully styled, mobile-friendly HTML view containing:
|
||
- Article/Digest Title, Source attribution, and publication date.
|
||
- Audio player (if TTS generated).
|
||
- AI Summary & Full Markdown Body.
|
||
4. External readers can access the link without any login or authentication prompts.
|
||
|
||
### Journey 6: Multi-Device Sync & Third-Party Native App Support
|
||
|
||
- **Goal**: Read on mobile (e.g., NetNewsWire / Reeder on iOS/macOS) while traveling, and continue on desktop PWA at work.
|
||
- **Flow**:
|
||
1. User configures NetNewsWire using the built-in **Fever API** or **Google Reader API** endpoint credentials (`https://rss.domain.com/api/greader`).
|
||
2. Mobile app synchronizes unread states, starred articles, and folders in real time.
|
||
3. Read state changes on mobile reflect instantly on the web PWA via Go backend sync.
|
||
|
||
---
|
||
|
||
## 4. Functional Requirements & Feature Specifications
|
||
|
||
### 4.1 Single-User Authentication & Security
|
||
|
||
| Requirement ID | Feature | Description |
|
||
| :--- | :--- | :--- |
|
||
| **AUTH-01** | Single Admin Entry | Database stores single admin user account with hashed password (Argon2id / bcrypt). |
|
||
| **AUTH-02** | Session & JWT Auth | Secure, HTTP-only Cookie-based session for Web PWA, plus persistent API Tokens for third-party apps and CLI access. |
|
||
| **AUTH-03** | Public Share Isolation | Endpoints under `/share/*` bypass authentication entirely; all administrative, ingestion, and reader endpoints require valid admin credentials. |
|
||
| **AUTH-04** | Security Headers & Rate Limiting | Built-in rate limiting on login/API endpoints, CSRF protection, and customizable CORS headers. |
|
||
|
||
### 4.2 Ingestion & Feed Management
|
||
|
||
| Requirement ID | Feature | Description |
|
||
| :--- | :--- | :--- |
|
||
| **FEED-01** | Multi-Format Parsing | Supports RSS 0.9x/1.0/2.0, Atom 1.0, and JSON Feed 1.0/1.1 formats. |
|
||
| **FEED-02** | Concurrency Worker Pool | Background worker pool in Go with configurable concurrency, rate limits per domain, and exponential backoff retry. |
|
||
| **FEED-03** | Conditional GET / ETag | Utilizes `If-Modified-Since` and `ETag` headers to minimize network traffic and server load. |
|
||
| **FEED-04** | OPML Import / Export | Full OPML 2.0 import and export support, maintaining folder hierarchy and feed settings. |
|
||
| **FEED-05** | Category & Tag Hierarchy | Supports nesting feeds under Categories/Folders as well as applying multi-dimensional Tags. |
|
||
|
||
### 4.3 Content Processing & Scraper Engine
|
||
|
||
| Requirement ID | Feature | Description |
|
||
| :--- | :--- | :--- |
|
||
| **PROC-01** | Readability Full-Text Scraper | Built-in readability engine (extracts clean article HTML/Markdown from original URL for truncated feeds). |
|
||
| **PROC-02** | HTML Sanitization & Media Proxy | Strict XSS sanitization (DOMPurify / Bluemonday) and optional image caching/proxying to avoid mixed-content and tracking pixels. |
|
||
| **PROC-03** | Content Deduplication | SHA-256 content hashing and URL canonicalization to detect duplicate articles across feeds. |
|
||
|
||
### 4.4 Dual-Tier Rules Engine & Feed Hooks (DAG Architecture)
|
||
|
||
```mermaid
|
||
graph TD
|
||
subgraph Ingestion Phase
|
||
A[Fetch RSS / Atom] --> B[Parse XML & Normalize]
|
||
B --> C[Readability Full-Text Extraction]
|
||
end
|
||
|
||
subgraph Enrichment Phase (Single-Feed Hooks)
|
||
C --> D{Single-Feed Rules Match?}
|
||
D -->|Match| E1[LLM Summarization & Scoring]
|
||
D -->|Match| E2[Edge-TTS / Audio Generation]
|
||
D -->|Match| E3[Auto-Tag / Star / Drop]
|
||
D -->|Match| E4[Outbound Webhook Telegram/Discord]
|
||
D -->|Match| E5[Publish to /share/uuid.html]
|
||
end
|
||
|
||
subgraph Storage & Indexing
|
||
E1 & E2 & E3 & E4 & E5 & D -->|Persist| F[(Database & FTS Index)]
|
||
end
|
||
|
||
subgraph Aggregation Phase (Multi-Feed Hooks)
|
||
F --> G{Aggregation Trigger Fired?}
|
||
G -->|Threshold > N items OR Schedule Cron| H[Batch Fetch Unread Target Feeds]
|
||
H --> I[LLM Multi-Document Synthesis]
|
||
I --> J[Generate Audio Podcast]
|
||
J --> K[Inject to Virtual Digest Feed]
|
||
K --> L[Batch Mark Source Items as Read]
|
||
end
|
||
```
|
||
|
||
#### A. Single-Feed Hooks (Per-Item Pipeline)
|
||
|
||
- **Triggers**: On Ingestion (`on_feed_ingest`), On User Star (`on_item_star`), On Tag Added (`on_item_tag`).
|
||
- **Conditions**: Source Feed / Category, Title/Content Regex, Author, Word Count, Language, Metadata matching.
|
||
- **Actions**:
|
||
1. `ai_summarize`: Send article content to configured OpenAI-compatible endpoint with custom system prompt; save summary to metadata.
|
||
2. `ai_score`: Compute relevance score (0–100) and rationale.
|
||
3. `generate_tts`: Synthesize audio via Edge-TTS / OpenAI TTS / ElevenLabs and store media link.
|
||
4. `auto_tag` / `auto_star` / `mark_read` / `drop_item`: Modify feed state automatically.
|
||
5. `create_public_share`: Automatically generate a public share link `/share/<uuid>.html`.
|
||
6. `send_webhook`: Dispatch JSON payload or message formatting to Telegram, Discord, Slack, or generic HTTP webhook (n8n/Zapier).
|
||
|
||
#### B. Multi-Feed Hooks (Batch & Temporal Aggregation)
|
||
|
||
- **Triggers**:
|
||
- `schedule`: Cron expression (e.g. `30 7 * * *` for 7:30 AM daily).
|
||
- `threshold`: When unread items matching criteria $\ge N$ (e.g. $> 15$ unread articles in Category "AI & ML").
|
||
- **Conditions**: Category, Tag filter, Keyword filter, Unread status.
|
||
- **Actions**:
|
||
1. `synthesize_digest`: Execute multi-document LLM prompt to generate an editorial digest with categorized trends and citations.
|
||
2. `create_virtual_digest_feed`: Inject the synthesized article into an internal "Virtual Digest Feed" (e.g., "Daily AI Brief").
|
||
3. `generate_digest_podcast`: Synthesize speech for the full digest episode and attach audio file.
|
||
4. `batch_archive_sources`: Automatically mark the individual source articles as read/archived, maintaining Inbox Zero.
|
||
|
||
#### C. DAG Safety & Anti-Loop Validation
|
||
|
||
- All rule definitions are validated upon creation/update via Topological Sort.
|
||
- Cyclic dependencies (e.g., Rule A creates Feed Item that triggers Rule B, which triggers Rule A) are rejected at configuration time.
|
||
- Hard recursion depth cap (depth = 3) on cascading triggers.
|
||
|
||
### 4.5 Reading Experience & UI / UX Specifications
|
||
|
||
```
|
||
+-----------------------------------------------------------------------------------------------+
|
||
| RSSPipe [Logo] [ Search: is:unread tag:tech / ] [Theme] [Sync] [Admin Profile]|
|
||
+-------------------+---------------------------------------------------------------------------+
|
||
| + Add Subscription| (Inbox: 12 unread) [ Mark All Read ] [ View: Stream | Compact ] [Sort] |
|
||
| ----------------- +---------------------------------------------------------------------------+
|
||
| * Inbox (12) | [ ] * [Hacker News] Building Local-First PWA with SQLite 10m ago |
|
||
| * Starred (4) | AI Summary: Explains sync protocols and CRDTs... (Score: 92) |
|
||
| * Digests (2) | ------------------------------------------------------------------------- |
|
||
| - Morning Brief | [ ] [Ars Technica] Quantum Computing Milestone Announced 45m ago |
|
||
| - Weekly Radar | ------------------------------------------------------------------------- |
|
||
| v CATEGORIES | [ ] * [Virtual Digest] Morning AI Briefing (5 mins audio) 7:30 AM |
|
||
| > Tech (8) | [ ▶ Play Audio 05:21 ] Full synthesized summary of 18 posts... |
|
||
| > Science (4) | ------------------------------------------------------------------------- |
|
||
+-------------------+---------------------------------------------------------------------------+
|
||
```
|
||
|
||
#### Dual View Modes
|
||
|
||
1. **Google Reader Expanded Stream View**:
|
||
- Continuous scroll through full article cards.
|
||
- Header with Feed Title, Favicon, Author, Publish Date, AI Summary pill, Audio Player bar (if present), and Actions (Star, Share, Original URL).
|
||
- Clean typographical hierarchy with readable line lengths (65–80 characters max for text body), syntax-highlighted code blocks, and responsive image scaling.
|
||
2. **Gmail Compact List / Split-Pane View**:
|
||
- High-density list: Checkbox, Star icon, Source Feed badge, Article Title, 1-line snippet/AI Summary preview, Timestamp.
|
||
- Split-pane layout: Clicking an item opens the reading pane on the right or below, retaining the list for fast scanning.
|
||
|
||
#### Keyboard Shortcut Matrix (Google Reader & Gmail Paradigm)
|
||
|
||
| Key Binding | Action |
|
||
| :--- | :--- |
|
||
| `j` / `k` | Navigate to Next / Previous article |
|
||
| `Space` / `Shift + Space` | Page Down / Page Up through stream |
|
||
| `o` or `Enter` | Expand / Collapse current item in compact view |
|
||
| `v` | Open original article URL in a new browser tab |
|
||
| `s` | Star / Unstar current item |
|
||
| `m` | Toggle Read / Unread status |
|
||
| `e` or `Shift + I` | Archive / Mark as Read and advance |
|
||
| `Shift + A` | Mark all items in current view as read (with confirmation modal) |
|
||
| `/` | Focus search bar |
|
||
| `g` then `i` | Go to Inbox |
|
||
| `g` then `s` | Go to Starred |
|
||
| `g` then `d` | Go to Virtual Digests |
|
||
| `?` | Open keyboard shortcuts help modal |
|
||
|
||
#### PWA & Offline Support
|
||
|
||
- Fully installable PWA with Web App Manifest and Service Worker caching.
|
||
- IndexedDB client caching (Dexie.js) allowing offline reading of already synced unread/starred articles and summaries.
|
||
- Offline action queuing (marking read, starring while offline syncs automatically when connection is restored).
|
||
|
||
### 4.6 Search & Query Syntax (Gmail-Style Engine)
|
||
|
||
The search bar supports advanced tokenized queries backed by database Full-Text Search:
|
||
|
||
- `is:unread` / `is:read` / `is:starred`
|
||
- `tag:tech` / `category:engineering`
|
||
- `feed:"Hacker News"` / `from:author_name`
|
||
- `has:summary` / `has:audio` / `has:share`
|
||
- `score:>80` / `score:<50`
|
||
- `after:2026-08-01` / `before:2026-08-20`
|
||
- Raw keyword and phrase matching: `"generative ai" kubernetes`
|
||
|
||
### 4.7 API & Ecosystem Integrations
|
||
|
||
1. **Fever API & Google Reader API**:
|
||
- `/api/fever`: Implements standard Fever protocol endpoints (`?api`, `?items`, `?feeds`, `?groups`, `?unread_item_ids`, `?saved_item_ids`).
|
||
- `/api/greader`: Implements Google Reader API endpoints (`/reader/api/0/...`) for NetNewsWire, Reeder, and FeedMe.
|
||
2. **AI / LLM Integration**:
|
||
- Generic OpenAI-compatible client (configurable `base_url`, `api_key`, `model_name`, `temperature`, `max_tokens`).
|
||
- Supports OpenAI (GPT-4o/Mini), Anthropic (via OpenAI proxy/native), Google Gemini, DeepSeek, Ollama, and LocalAI.
|
||
3. **TTS Integration**:
|
||
- Free Microsoft Edge-TTS integration (high-quality neural voices with zero API key requirement).
|
||
- Optional OpenAI TTS (`tts-1`, `tts-1-hd`) and ElevenLabs API support.
|
||
4. **Outbound Notification Webhooks**:
|
||
- Native Telegram Bot API formatting.
|
||
- Discord / Slack incoming webhook formatting.
|
||
- Generic HTTP POST with configurable headers and JSON template.
|
||
|
||
---
|
||
|
||
## 5. Technical Architecture & Data Schema
|
||
|
||
### 5.1 Data Models (Relational & Flexible Metadata)
|
||
|
||
```sql
|
||
-- 1. Users Table (Single Admin Entry)
|
||
CREATE TABLE users (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
username VARCHAR(64) UNIQUE NOT NULL,
|
||
password_hash VARCHAR(255) NOT NULL,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
-- 2. Categories Table
|
||
CREATE TABLE categories (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
title VARCHAR(128) NOT NULL,
|
||
parent_id UUID REFERENCES categories(id) ON DELETE SET NULL,
|
||
sort_order INT DEFAULT 0,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
-- 3. Feeds Table
|
||
CREATE TABLE feeds (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
title VARCHAR(255) NOT NULL,
|
||
feed_url TEXT UNIQUE NOT NULL,
|
||
site_url TEXT,
|
||
category_id UUID REFERENCES categories(id) ON DELETE SET NULL,
|
||
is_virtual_digest BOOLEAN DEFAULT FALSE,
|
||
refresh_interval_minutes INT DEFAULT 30,
|
||
etag VARCHAR(255),
|
||
last_modified VARCHAR(255),
|
||
last_fetched_at TIMESTAMPTZ,
|
||
fetch_error TEXT,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
-- 4. Feed Entries Table
|
||
CREATE TABLE feed_entries (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
feed_id UUID NOT NULL REFERENCES feeds(id) ON DELETE CASCADE,
|
||
guid TEXT NOT NULL,
|
||
url TEXT NOT NULL,
|
||
title TEXT NOT NULL,
|
||
author VARCHAR(255),
|
||
content TEXT NOT NULL, -- Cleaned full-text / readability content
|
||
raw_content TEXT, -- Original feed snippet
|
||
published_at TIMESTAMPTZ NOT NULL,
|
||
is_read BOOLEAN DEFAULT FALSE,
|
||
is_starred BOOLEAN DEFAULT FALSE,
|
||
share_uuid UUID UNIQUE, -- Non-null if published to /share/<uuid>.html
|
||
|
||
-- Dynamic Metadata & Extensibility (JSONB)
|
||
-- Contains: { "aiSummary": "...", "score": 85, "tags": ["ai", "go"], "ttsUrl": "..." }
|
||
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
CONSTRAINT unique_feed_guid UNIQUE(feed_id, guid)
|
||
);
|
||
|
||
-- Indexes for Fast Triage and FTS
|
||
CREATE INDEX idx_entries_feed_read ON feed_entries(feed_id, is_read);
|
||
CREATE INDEX idx_entries_published ON feed_entries(published_at DESC);
|
||
CREATE INDEX idx_entries_share_uuid ON feed_entries(share_uuid) WHERE share_uuid IS NOT NULL;
|
||
CREATE INDEX idx_entries_metadata_gin ON feed_entries USING GIN (metadata);
|
||
|
||
-- 5. Rules & Hooks Table
|
||
CREATE TABLE rules (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
name VARCHAR(128) NOT NULL,
|
||
rule_type VARCHAR(32) NOT NULL, -- 'single_feed' | 'multi_feed_digest'
|
||
is_active BOOLEAN DEFAULT TRUE,
|
||
trigger_type VARCHAR(32) NOT NULL, -- 'on_ingest' | 'schedule' | 'threshold'
|
||
trigger_config JSONB NOT NULL DEFAULT '{}'::jsonb, -- { "cron": "0 8 * * *", "threshold": 15 }
|
||
conditions JSONB NOT NULL DEFAULT '[]'::jsonb, -- [{ "field": "category", "op": "eq", "value": "tech" }]
|
||
actions JSONB NOT NULL DEFAULT '[]'::jsonb, -- [{ "action": "ai_summarize" }, { "action": "generate_tts" }]
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
```
|
||
|
||
### 5.2 Deployment & Packaging Specification
|
||
|
||
- **Single Docker Container**:
|
||
- Base Image: `golang:1.23-alpine` (builder) + `alpine:latest` (runtime).
|
||
- Web Assets: Built using `vite build` and embedded into the Go binary using `//go:embed dist/*`.
|
||
- Database Support: Built-in driver for PostgreSQL (via `pgx`) and SQLite3 with WAL mode enabled.
|
||
- **K3s / Kubernetes Manifests**:
|
||
- `Deployment`: 1 replica, resource limits ~64MB RAM / 0.1 CPU idle.
|
||
- `Service` & `Ingress`: Traefik ingress exposing web routes and public `/share/*` routes.
|
||
- `PersistentVolumeClaim` (optional for SQLite storage or local audio cache).
|
||
|
||
---
|
||
|
||
## 6. Milestone Roadmap & Implementation Breakdown
|
||
|
||
```mermaid
|
||
gantt
|
||
title RSSPipe Implementation Roadmap
|
||
dateFormat YYYY-MM-DD
|
||
section M1: Core Foundation
|
||
Go Server, DB & Single-User Auth :m1_1, 2026-09-01, 7d
|
||
Ingestion Engine & OPML Migration :m1_2, after m1_1, 7d
|
||
section M2: PWA & Reading UX
|
||
React Vite PWA Setup & Theming :m2_1, after m1_2, 5d
|
||
Google Reader & Gmail Keyboards / UI :m2_2, after m2_1, 7d
|
||
Gmail Search Engine & Offline Cache :m2_3, after m2_2, 5d
|
||
section M3: Single-Feed Hooks
|
||
Readability Full-Text Scraper :m3_1, after m2_3, 4d
|
||
LLM Summarization & Scoring Hooks :m3_2, after m3_1, 5d
|
||
Public Share /share/<uuid>.html :m3_3, after m3_2, 4d
|
||
Outbound Webhooks (Telegram/Discord) :m3_4, after m3_3, 3d
|
||
section M4: Multi-Feed Digest & Audio
|
||
Multi-Feed Batch Aggregator :m4_1, after m3_4, 6d
|
||
Virtual Digest Feeds & Auto-Archive :m4_2, after m4_1, 5d
|
||
Edge-TTS & Audio Player Integration :m4_3, after m4_2, 5d
|
||
section M5: Ecosystem & Polishing
|
||
Fever / Google Reader API Layer :m5_1, after m4_3, 6d
|
||
Rules UI Builder & DAG Validation :m5_2, after m5_1, 5d
|
||
K3s Deployment & E2E Validation :m5_3, after m5_2, 4d
|
||
```
|
||
|
||
### Milestone 1: Core Foundation & Ingestion Engine
|
||
|
||
- [ ] Go backend server skeleton with Chi/Echo router and embedded static server.
|
||
- [ ] Database layer supporting PostgreSQL and SQLite (WAL mode).
|
||
- [ ] Single admin user authentication (Session Cookies, JWT, bcrypt/Argon2id).
|
||
- [ ] Multi-format RSS/Atom/JSON parser with concurrent fetch worker pool.
|
||
- [ ] OPML 2.0 import/export engine for seamless Miniflux migration.
|
||
- [ ] Single Dockerfile packaging backend + frontend ready for k3s deployment.
|
||
|
||
### Milestone 2: PWA Frontend & Google Reader / Gmail UX
|
||
|
||
- [ ] React + Vite PWA boilerplate with Tailwind/Vanilla CSS design system, dark/light modes, and responsive layout.
|
||
- [ ] Dual view modes: Google Reader continuous expanded stream + Gmail compact list/split-pane.
|
||
- [ ] Full keyboard navigation workflow (`j`, `k`, `o`, `v`, `s`, `m`, `e`, `Shift+A`, `/`).
|
||
- [ ] Service Worker and IndexedDB (Dexie.js) for local caching and offline reading.
|
||
- [ ] Gmail-style tokenized search bar (`is:unread`, `tag:xxx`, `score:>80`, etc.).
|
||
|
||
### Milestone 3: Single-Feed Hooks & Enrichment
|
||
|
||
- [ ] Backend readability scraper to extract clean full-text markdown for truncated feeds.
|
||
- [ ] Single-feed rules engine pipeline with trigger-condition-action execution.
|
||
- [ ] OpenAI-compatible LLM connector for automated 3-bullet summary, key takeaways, and relevance scoring (0–100).
|
||
- [ ] Public share generator creating standalone `/share/<uuid>.html` accessible without auth.
|
||
- [ ] Outbound webhook dispatcher (Telegram, Discord, Slack, generic HTTP POST).
|
||
|
||
### Milestone 4: Multi-Feed Digest Synthesis & Audio (TTS)
|
||
|
||
- [ ] Multi-feed batch scheduler with cron and count-threshold triggers.
|
||
- [ ] Multi-document LLM synthesis engine for creating structured daily/topic briefs.
|
||
- [ ] Virtual Digest Feeds architecture (internal native feeds receiving synthesized articles).
|
||
- [ ] Batch auto-archive / auto-mark read mechanism for processed source articles.
|
||
- [ ] Audio generation engine (Edge-TTS and OpenAI TTS) with embedded audio player in PWA and public share pages.
|
||
|
||
### Milestone 5: Ecosystem Compatibility & Final Polish
|
||
|
||
- [ ] Fever API (`/api/fever`) and Google Reader API (`/api/greader`) implementation for native mobile app sync (NetNewsWire, Reeder).
|
||
- [ ] Visual Rule Builder in Settings with DAG cycle validator (Topological Sort).
|
||
- [ ] Telemetry & Reading Analytics (dwell time, reading progress).
|
||
- [ ] Complete k3s deployment manifests (Deployment, Service, Ingress, PVC) and comprehensive documentation.
|
||
|
||
---
|
||
|
||
## 7. Success Metrics & Verification Criteria
|
||
|
||
1. **Migration Fidelity**: 100% of feeds and categories from Miniflux OPML imported cleanly without data loss.
|
||
2. **Triaging Speed**: Zero-lag keyboard navigation (`j`/`k`) with UI frame times strictly $< 16\text{ms}$ (60 FPS).
|
||
3. **Resource Efficiency**: Single container idling at $< 64\text{MB}$ RAM and $< 1\%$ CPU on k3s cluster.
|
||
4. **Hook Reliability**: 100% of ingestion hooks executed asynchronously without blocking feed fetch workers; zero DAG execution cycles.
|
||
5. **Ecosystem Compatibility**: NetNewsWire / Reeder connect and synchronize unread/starred states without errors.
|