feat: scaffold initial RSS reader project with Go backend and React frontend

2026-08-24 08:08:31 -07:00
commit 3742dbdce4
44 changed files with 9901 additions and 0 deletions
@@ -0,0 +1,103 @@
+
+# 架构设计与需求文档：RSSPipe
+
+1. 核心需求概述
+本系统旨在构建一个针对单用户场景的 高度可扩展、本地优先（Local-First）且端到端加密的 RSS 网关客户端（PWA）。它不仅具备传统 RSS 的订阅与阅读功能，更通过事件驱动的规则引擎，深度集成 AI 能力。
+关键特性要求：
+
+* 本地优先 (Local-First PWA)：核心数据存储与业务流转在客户端（浏览器环境）完成，支持离线访问。
+* 端到端加密 (E2EE)：数据在落盘或同步前必须在客户端加密，保证绝对隐私。
+* 事件驱动的规则引擎 (Rules Engine)：支持类似 Gmail Filter 的 Trigger-Condition-Action 机制。
+* 防死循环与状态隔离 (DAG & CQRS)：严格拆分 Feed 生命周期，分离单实体（Single-Entity）富化逻辑与多实体（Multi-Entity）聚合逻辑，彻底杜绝规则循环执行。
+* 无缝旁路扩展 (Plugin/RPC)：通过定义统一的“增量 Patch”通信契约，支持用 TS 编写本地 Plugin，或通过 gRPC/Connect 调用外部 Go/Python 的 AI 节点。
+* 无预设动态 Schema：Feed 的附加数据（如 AI 摘要、评分、TTS 音频等）不硬编码在数据模型中，而是通过 JSON 增量合并。
+* 遥测与反馈循环 (Telemetry)：采集阅读行为（驻留时长、可见字数比），为下游 AI 模型或动态权重系统提供微调依据。
+
+1. 核心架构视图
+2. 核心数据模型 (Data Schema)
+系统采用 Core (不可变主键) + Metadata (可变 JSONB) 的模式。
+// 核心路由与索引字段（关系型特征）
+interface FeedCore {
+  id: string;          // 唯一标识符，基于 url + timestamp 哈希
+  url: string;         // 原始链接
+  sourceId: string;    // 订阅源 ID
+  timestamp: number;   // 抓取/发布时间戳
+  title: string;
+  content: string;     // 清洗后的纯文本/Markdown
+}
+
+// 变长扩展数据（无预设 Schema）
+interface FeedMetadata {
+  isRead: boolean;
+  tags: string[];      // 标签集合
+  aiSummary?: string;  // AI 摘要
+  ttsUrl?: string;     // 语音合成链接
+  score?: number;      // 推荐分数
+  [key: string]: any;  // 其他外部 Plugin 注入的动态字段
+}
+
+// 数据库实际存储结构（明文态）
+interface LocalFeedEntry extends FeedCore {
+  metadata: FeedMetadata;
+}
+
+1. 生命周期与规则引擎设计 (DAG Rule Engine)
+为了防止死循环，Feed 的处理被严格限制在一个单向流动的流水线中。
+4.1 生命周期的四个阶段 (Phases)
+
+* Ingestion (摄入期)：从远程 XML 拉取数据，使用 rss-parser 解析，使用 @mozilla/readability 提取纯正文。
+* Enrichment (丰富期)：单 Feed 事件的主战场。在此阶段，系统调用配置好的 Plugin（本地或远程），获取摘要、打分等元数据。
+* Aggregation (聚合期)：批处理的主战场。由独立的调度器（基于 RxJS）监听数据库写入操作，当符合特定条件的 Feed 数量达到阈值，触发聚合 Action（如生成周报）。
+* Action (动作期)：数据的最终落盘加密存储，以及触发 UI 层的渲染与系统通知。
+4.2 统一 Plugin 接口契约 (Plugin Interface Protocol)
+无论是前端本地的 TypeScript 函数，还是远端的 Go 语言 gRPC 服务，都必须遵循以下契约进行通信，强制使用 JSON Patch，禁止外部服务直接覆写完整 Feed 数据。
+// 输入契约 (发送给 Plugin 的数据)
+interface PluginInput {
+  eventId: string;
+  phase: "enrichment" | "aggregation";
+  feedSnapshot: Readonly<FeedCore>;
+  feedMetadata: Readonly<FeedMetadata>;
+  triggerReason: string;
+}
+
+// 输出契约 (Plugin 返回给主系统的指令)
+interface PluginOutput {
+  action: "merge_metadata" | "drop_feed" | "none";
+  metadataPatch?: {
+    $set?: Record<string, any>;   // 覆写字段，如 { aiSummary: "..." }
+    $push?: Record<string, any[]>;  // 追加数组元素，如 { tags: ["tech"] }
+  };
+}
+
+4.3 规则防死锁机制
+在添加或修改 Rule 时，运行拓扑排序（Topological Sort）引擎，验证 Trigger 和 Action 之间是否存在循环依赖，存在则拒绝保存配置。
+5. 里程碑拆解与模块化实现方案
+Milestone 1: 本地基础设施 (Local-First Base)
+
+* 任务目标：搭建基础 PWA 框架，实现端到端加密与基础数据读写。
+* 关键技术栈：Vite + React, Dexie.js (IndexedDB), Web Crypto API。
+* 核心模块：
+  * DB 层：基于 Dexie.js 定义本地表结构，只允许应用层通过统一的 Service 读写。
+  * 加密拦截器：数据写入 IndexedDB 前进行 AES-GCM 加密，读取时解密，确保本地离线数据的安全性。
+  * 基础 Parser：实现对 RSS 源的拉取和 @mozilla/readability 的正文清洗。
+Milestone 2: 单 Feed 规则引擎与本地 Plugin (Core Engine)
+* 任务目标：实现生命周期流水线与 Enrichment 阶段的核心逻辑。
+* 关键技术栈：RxJS。
+* 核心模块：
+  * 生命周期 Event Bus：基于 RxJS Subject 创建事件总线，控制数据流从 Ingestion 向下流转。
+  * Plugin 调度中心：根据用户配置的 Rules，将数据 Snapshot 传递给对应的本地 TS 函数，获取 Patch 并执行安全合并（Lodash merge 等效逻辑）。
+  * DAG 校验器：实现并集成拓扑排序算法，验证用户配置。
+Milestone 3: 旁路扩展层 (Remote RPC & Extensibility)
+* 任务目标：实现外部 Go/Python AI 控制器的接入。
+* 关键技术栈：Connect-RPC (@connectrpc/connect-web), Protobuf。
+* 核心模块：
+  * 契约生成：使用 Protobuf 将 PluginInput 和 PluginOutput 标准化，生成两端的接口代码。
+  * RPC 适配器：在前端 Rules Engine 中编写一个特殊的 Plugin Adapter，遇到远程规则时，通过 Connect 将数据发往 Go 后端服务，并等待返回 JSON Patch。
+  * 动态 UI 渲染器：重构阅读页面，使其不再依赖硬编码字段，而是根据 Metadata 动态挂载组件（如检测到 ttsUrl 挂载播放器）。
+Milestone 4: 遥测与批量聚合处理 (Telemetry & Aggregation)
+* 任务目标：实现多状态 Filter 与用户阅读数据的反馈回路。
+* 关键技术栈：RxJS (bufferCount, bufferTime), Intersection Observer API。
+* 核心模块：
+  * 批处理调度器：针对 Aggregation 阶段，使用 RxJS 的缓冲操作符，在达到数量或时间阈值时触发特定的 RPC 聚合调用。
+  * 阅读统计探针：基于页面的滚动事件与 Intersection Observer，计算用户的实际停留时间与可见内容量，生成打分日志。
+  * 反馈推流机制：周期性地将本地存储的遥测日志通过网络发送给后端模型训练/微调服务。
@@ -0,0 +1,513 @@
+# RSSPipe: Product Requirements Document (PRD)
+
+> **Document Version:** 1.0.0  
+> **Status:** Approved / Ready for Implementation  
+> **Target Audience:** Engineering, Architecture, Product  
+> **Replaces:** Miniflux Deployment  
+> **Deployment Target:** k3s / Kubernetes (Single Container Architecture with Separate or Embedded DB)
+
+---
+
+## 1. Executive Summary & Product Vision
+
+### 1.1 Overview
+
+**RSSPipe** is a self-hosted, RSS-driven **Information Gateway** and intelligent reading platform built to replace traditional RSS aggregators (such as Miniflux).
+
+Rather than serving purely as a passive feed bucket, RSSPipe combines:
+
+1. **Google Reader & Gmail UX Philosophy**: Rapid, keyboard-first triaging (`j`/`k`, `o`, `v`, `s`, `e`), "Inbox Zero" workflow, dual view modes (expanded scrolling stream vs. compact list/split-pane), and Gmail-style search syntax.
+2. **Dual-Tier Feed Hooks Engine**:
+   - **Single-Feed Enrichment**: Real-time readability full-text scraping, AI summarization/takeaways, relevance scoring, auto-tagging, TTS generation, and automatic public share-page publishing.
+   - **Multi-Feed AI Digest Synthesis**: Scheduled or threshold-based multi-document synthesis that rolls up unread items into clean "Virtual Digest Feeds" (e.g., Daily Morning Briefing, Topic Radar) with synthesized audio podcasts, automatically archiving raw individual items.
+3. **Single-User Security with Public Sharing**: Single admin user model protecting the private gateway, with frictionless unauthenticated public sharing via tokenized URLs (`/share/<uuid>.html`).
+4. **Offline-Capable PWA & Native Ecosystem Support**: Installable Progressive Web App (PWA) with responsive mobile/desktop interfaces, alongside a Fever / Google Reader API compatibility layer for native mobile clients (e.g., NetNewsWire, Reeder).
+5. **Unified K3s-Ready Architecture**: High-performance Go backend with embedded React/Vite PWA frontend, packaged as a single lightweight Docker container targeting k3s/Kubernetes.
+
+---
+
+## 2. Core Architecture & System Tenets
+
+```
+               ┌────────────────────────────────────────────────────────┐
+               │                     RSSPipe Web App                    │
+               │   (React + Vite PWA / Google Reader & Gmail UX)        │
+               └───────────┬────────────────────────────────┬───────────┘
+                           │ REST / SSE Events              │ Fever / GReader API
+                           ▼                                ▼
+┌───────────────────────────────────────────────────────────────────────┐
+│                          Go Unified Server                            │
+│                                                                       │
+│  ┌────────────────────┐  ┌────────────────────┐  ┌─────────────────┐  │
+│  │   Auth & Session   │  │   Ingestion Pool   │  │ Public Share    │  │
+│  │  (Single Admin)    │  │ (Cron & Web Crawl) │  │ (/share/<uuid>) │  │
+│  └────────────────────┘  └─────────┬──────────┘  └─────────────────┘  │
+│                                    ▼                                  │
+│  ┌─────────────────────────────────────────────────────────────────┐  │
+│  │                     Rules & Hook Engine (DAG)                   │  │
+│  │  ┌───────────────────────────────┐ ┌──────────────────────────┐ │  │
+│  │  │   Single-Feed Enrichment      │ │   Multi-Feed Aggregation │ │  │
+│  │  │ (Readability/LLM/TTS/Webhook) │ │ (Digest Synthesis/Audio) │ │  │
+│  │  └───────────────────────────────┘ └──────────────────────────┘ │  │
+│  └─────────────────────────────────┬───────────────────────────────┘  │
+│                                    ▼                                  │
+│  ┌─────────────────────────────────────────────────────────────────┐  │
+│  │     Data Layer (PostgreSQL / SQLite WAL + FTS5 Full-Text)       │  │
+│  └─────────────────────────────────────────────────────────────────┘  │
+└───────────────────────────────────────────────────────────────────────┘
+```
+
+### 2.1 Design Tenets
+
+1. **Speed over Friction**: Sub-100ms UI responses, instant keyboard navigation, and optimistic UI updates.
+2. **Information High Signal-to-Noise**: Raw RSS feeds are enriched or distilled into digestible summaries and podcasts so the user stays informed without drowning in unread counts.
+3. **Zero Maintenance & Single Container Simplicity**: Single binary/image containing backend server, worker pools, and embedded web assets, deployable via simple Helm chart / Kubernetes manifests on k3s.
+4. **Cycle-Safe Rule Processing**: All feed transformation steps follow a strict Directed Acyclic Graph (DAG) with topological validation to prevent infinite execution loops.
+
+---
+
+## 3. End-to-End User Journeys
+
+```mermaid
+journey
+    title Core User Journeys in RSSPipe
+    section 1. Onboarding & Migration
+      Import OPML from Miniflux: 5: Admin
+      Auto-categorize and test feeds: 4: Admin
+    section 2. Daily Triaging (Google Reader / Gmail UX)
+      Open PWA (Desktop/Mobile): 5: Admin
+      Navigate feeds with 'j'/'k' and expand with 'o': 5: Admin
+      Quick triage with 'e' (archive/read) and 's' (star): 5: Admin
+      Search with operators 'is:unread tag:ai': 5: Admin
+    section 3. Feed Hooks & Enrichment
+      Ingest feed & scrape full-text via Readability: 5: System
+      LLM summarizes & scores relevance: 5: System
+      Generate TTS audio preview: 4: System
+    section 4. Multi-Feed Digest Generation
+      Accumulate > 15 topic feeds or reach 8:00 AM: 5: System
+      Synthesize AI Daily Digest & Audio Podcast: 5: System
+      Publish to Virtual Digest Feed & mark raw items read: 5: System
+    section 5. Knowledge Sharing
+      Click 'Share to Public' on article/digest: 5: Admin
+      External visitor views /share/<uuid>.html: 5: Public Reader
+```
+
+### Journey 1: One-Click Migration from Miniflux (Onboarding)
+
+- **Goal**: Seamlessly transition existing feeds, categories, and folder hierarchies from Miniflux to RSSPipe in seconds.
+- **Flow**:
+  1. User navigates to `Settings -> Migration & OPML`.
+  2. User uploads Miniflux `subscriptions.opml`.
+  3. System parses categories and feed URLs, tests connection concurrency in the background, and preserves category folder tree.
+  4. System prompts option to backfill recent posts (last N days) or start fresh from latest entries.
+  5. Dashboard immediately renders feeds categorized and organized ready for consumption.
+
+### Journey 2: Morning Triage (Google Reader & Gmail Workflow)
+
+- **Goal**: Rapidly process 200+ unread items in under 5 minutes using keyboard shortcuts and split views.
+- **Flow**:
+  1. User opens RSSPipe PWA. The "Inbox Zero" view displays all unread entries grouped by priority or chronological stream.
+  2. User toggles view:
+     - **Expanded Stream (Google Reader)**: Full article body inline, scrolling continuously with `Space` or `j`/`k`.
+     - **Compact List (Gmail)**: Single-line item rows displaying Source, Star, Title, AI Summary badge, and Time.
+  3. User presses `j` to focus the next item, `o` or `Enter` to expand/collapse.
+  4. User presses `s` to star for deep reading later, `v` to open original link in a new tab, `m` to toggle read/unread, or `e` to archive/mark read.
+  5. User presses `/` to activate search and types `is:unread tag:ai score:>80` to filter high-impact items instantly.
+
+### Journey 3: Single-Feed Automated Enrichment Hook
+
+- **Goal**: Automatically convert truncated RSS feeds into readable, summarized, and categorized entries with voice synthesis.
+- **Flow**:
+  1. Ingestion engine fetches a newly published feed item.
+  2. **Content Extraction**: System detects truncated content and runs the backend Readability scraper to fetch the full clean article markdown.
+  3. **AI Enrichment**: Configured rule triggers LLM prompt:
+     - Generates a 3-bullet executive summary.
+     - Assigns relevant topic tags (`#llm`, `#k8s`).
+     - Scores article relevance from 1 to 100 based on user interest profile.
+  4. **TTS Audio Hook**: High-scoring items automatically trigger Edge-TTS / OpenAI TTS generation, attaching an audio player to the feed header.
+  5. **Notification Hook**: If relevance `score >= 90`, sends a webhook alert to user's Telegram/Discord channel with the summary and direct link.
+
+### Journey 4: Multi-Feed AI Digest Synthesis & "Inbox Zero" Auto-Rollup
+
+- **Goal**: Eliminate RSS fatigue by condensing tens of high-volume feeds into a single curated morning/evening brief.
+- **Flow**:
+  1. User sets a Multi-Feed Aggregation Rule: *"Every day at 7:30 AM (or whenever unread items in #tech exceed 20), synthesize a 5-minute Executive Digest and mark source items as read."*
+  2. At the scheduled time / threshold, the Multi-Feed Engine queries all eligible unread items.
+  3. The Engine passes all article snippets/full-text into the LLM with a structured synthesis prompt:
+     - Grouping into major themes / top headlines.
+     - Highlighting key takeaways, contradictions, or trends across sources.
+     - Linking back to source items.
+  4. An AI TTS audio briefing is synthesized and attached.
+  5. The resulting synthesis is injected as a new entry into a **Virtual Digest Feed** (e.g., `📰 Daily Tech Executive Briefing`).
+  6. The 20+ raw underlying source feeds are automatically marked as read / archived.
+  7. When user opens the PWA, they see 1 high-quality digest episode instead of 20 fragmented articles.
+
+### Journey 5: Knowledge Sharing to External Visitors
+
+- **Goal**: Share an insightful article or AI digest with friends, colleagues, or social followers without granting access to the private RSS reader.
+- **Flow**:
+  1. In the PWA reader view, user clicks the **"Share to Public"** icon (or an automated rule publishes high-scoring digests automatically).
+  2. Backend generates an immutable, non-guessable UUID share route: `https://rss.domain.com/share/550e8400-e29b-41d4-a716-446655440000.html`.
+  3. The generated page is a standalone, beautifully styled, mobile-friendly HTML view containing:
+     - Article/Digest Title, Source attribution, and publication date.
+     - Audio player (if TTS generated).
+     - AI Summary & Full Markdown Body.
+  4. External readers can access the link without any login or authentication prompts.
+
+### Journey 6: Multi-Device Sync & Third-Party Native App Support
+
+- **Goal**: Read on mobile (e.g., NetNewsWire / Reeder on iOS/macOS) while traveling, and continue on desktop PWA at work.
+- **Flow**:
+  1. User configures NetNewsWire using the built-in **Fever API** or **Google Reader API** endpoint credentials (`https://rss.domain.com/api/greader`).
+  2. Mobile app synchronizes unread states, starred articles, and folders in real time.
+  3. Read state changes on mobile reflect instantly on the web PWA via Go backend sync.
+
+---
+
+## 4. Functional Requirements & Feature Specifications
+
+### 4.1 Single-User Authentication & Security
+
+| Requirement ID | Feature | Description |
+| :--- | :--- | :--- |
+| **AUTH-01** | Single Admin Entry | Database stores single admin user account with hashed password (Argon2id / bcrypt). |
+| **AUTH-02** | Session & JWT Auth | Secure, HTTP-only Cookie-based session for Web PWA, plus persistent API Tokens for third-party apps and CLI access. |
+| **AUTH-03** | Public Share Isolation | Endpoints under `/share/*` bypass authentication entirely; all administrative, ingestion, and reader endpoints require valid admin credentials. |
+| **AUTH-04** | Security Headers & Rate Limiting | Built-in rate limiting on login/API endpoints, CSRF protection, and customizable CORS headers. |
+
+### 4.2 Ingestion & Feed Management
+
+| Requirement ID | Feature | Description |
+| :--- | :--- | :--- |
+| **FEED-01** | Multi-Format Parsing | Supports RSS 0.9x/1.0/2.0, Atom 1.0, and JSON Feed 1.0/1.1 formats. |
+| **FEED-02** | Concurrency Worker Pool | Background worker pool in Go with configurable concurrency, rate limits per domain, and exponential backoff retry. |
+| **FEED-03** | Conditional GET / ETag | Utilizes `If-Modified-Since` and `ETag` headers to minimize network traffic and server load. |
+| **FEED-04** | OPML Import / Export | Full OPML 2.0 import and export support, maintaining folder hierarchy and feed settings. |
+| **FEED-05** | Category & Tag Hierarchy | Supports nesting feeds under Categories/Folders as well as applying multi-dimensional Tags. |
+
+### 4.3 Content Processing & Scraper Engine
+
+| Requirement ID | Feature | Description |
+| :--- | :--- | :--- |
+| **PROC-01** | Readability Full-Text Scraper | Built-in readability engine (extracts clean article HTML/Markdown from original URL for truncated feeds). |
+| **PROC-02** | HTML Sanitization & Media Proxy | Strict XSS sanitization (DOMPurify / Bluemonday) and optional image caching/proxying to avoid mixed-content and tracking pixels. |
+| **PROC-03** | Content Deduplication | SHA-256 content hashing and URL canonicalization to detect duplicate articles across feeds. |
+
+### 4.4 Dual-Tier Rules Engine & Feed Hooks (DAG Architecture)
+
+```mermaid
+graph TD
+    subgraph Ingestion Phase
+        A[Fetch RSS / Atom] --> B[Parse XML & Normalize]
+        B --> C[Readability Full-Text Extraction]
+    end
+
+    subgraph Enrichment Phase (Single-Feed Hooks)
+        C --> D{Single-Feed Rules Match?}
+        D -->|Match| E1[LLM Summarization & Scoring]
+        D -->|Match| E2[Edge-TTS / Audio Generation]
+        D -->|Match| E3[Auto-Tag / Star / Drop]
+        D -->|Match| E4[Outbound Webhook Telegram/Discord]
+        D -->|Match| E5[Publish to /share/uuid.html]
+    end
+
+    subgraph Storage & Indexing
+        E1 & E2 & E3 & E4 & E5 & D -->|Persist| F[(Database & FTS Index)]
+    end
+
+    subgraph Aggregation Phase (Multi-Feed Hooks)
+        F --> G{Aggregation Trigger Fired?}
+        G -->|Threshold > N items OR Schedule Cron| H[Batch Fetch Unread Target Feeds]
+        H --> I[LLM Multi-Document Synthesis]
+        I --> J[Generate Audio Podcast]
+        J --> K[Inject to Virtual Digest Feed]
+        K --> L[Batch Mark Source Items as Read]
+    end
+```
+
+#### A. Single-Feed Hooks (Per-Item Pipeline)
+
+- **Triggers**: On Ingestion (`on_feed_ingest`), On User Star (`on_item_star`), On Tag Added (`on_item_tag`).
+- **Conditions**: Source Feed / Category, Title/Content Regex, Author, Word Count, Language, Metadata matching.
+- **Actions**:
+  1. `ai_summarize`: Send article content to configured OpenAI-compatible endpoint with custom system prompt; save summary to metadata.
+  2. `ai_score`: Compute relevance score (0–100) and rationale.
+  3. `generate_tts`: Synthesize audio via Edge-TTS / OpenAI TTS / ElevenLabs and store media link.
+  4. `auto_tag` / `auto_star` / `mark_read` / `drop_item`: Modify feed state automatically.
+  5. `create_public_share`: Automatically generate a public share link `/share/<uuid>.html`.
+  6. `send_webhook`: Dispatch JSON payload or message formatting to Telegram, Discord, Slack, or generic HTTP webhook (n8n/Zapier).
+
+#### B. Multi-Feed Hooks (Batch & Temporal Aggregation)
+
+- **Triggers**:
+  - `schedule`: Cron expression (e.g. `30 7 * * *` for 7:30 AM daily).
+  - `threshold`: When unread items matching criteria $\ge N$ (e.g. $> 15$ unread articles in Category "AI & ML").
+- **Conditions**: Category, Tag filter, Keyword filter, Unread status.
+- **Actions**:
+  1. `synthesize_digest`: Execute multi-document LLM prompt to generate an editorial digest with categorized trends and citations.
+  2. `create_virtual_digest_feed`: Inject the synthesized article into an internal "Virtual Digest Feed" (e.g., "Daily AI Brief").
+  3. `generate_digest_podcast`: Synthesize speech for the full digest episode and attach audio file.
+  4. `batch_archive_sources`: Automatically mark the individual source articles as read/archived, maintaining Inbox Zero.
+
+#### C. DAG Safety & Anti-Loop Validation
+
+- All rule definitions are validated upon creation/update via Topological Sort.
+- Cyclic dependencies (e.g., Rule A creates Feed Item that triggers Rule B, which triggers Rule A) are rejected at configuration time.
+- Hard recursion depth cap (depth = 3) on cascading triggers.
+
+### 4.5 Reading Experience & UI / UX Specifications
+
+```
+-----------------------------------------------------------------------------------------------+
+| RSSPipe [Logo]       [ Search: is:unread tag:tech / ]           [Theme] [Sync] [Admin Profile]|
+-------------------+---------------------------------------------------------------------------+
+| + Add Subscription| (Inbox: 12 unread)  [ Mark All Read ]  [ View: Stream | Compact ]  [Sort] |
+| ----------------- +---------------------------------------------------------------------------+
+| * Inbox (12)      | [ ] * [Hacker News] Building Local-First PWA with SQLite        10m ago   |
+| * Starred (4)     |     AI Summary: Explains sync protocols and CRDTs... (Score: 92)          |
+| * Digests (2)     | ------------------------------------------------------------------------- |
+|   - Morning Brief | [ ]   [Ars Technica] Quantum Computing Milestone Announced      45m ago   |
+|   - Weekly Radar  | ------------------------------------------------------------------------- |
+| v CATEGORIES      | [ ] * [Virtual Digest] Morning AI Briefing (5 mins audio)       7:30 AM   |
+|   > Tech (8)      |     [ ▶ Play Audio 05:21 ] Full synthesized summary of 18 posts...        |
+|   > Science (4)   | ------------------------------------------------------------------------- |
+-------------------+---------------------------------------------------------------------------+
+```
+
+#### Dual View Modes
+
+1. **Google Reader Expanded Stream View**:
+   - Continuous scroll through full article cards.
+   - Header with Feed Title, Favicon, Author, Publish Date, AI Summary pill, Audio Player bar (if present), and Actions (Star, Share, Original URL).
+   - Clean typographical hierarchy with readable line lengths (65–80 characters max for text body), syntax-highlighted code blocks, and responsive image scaling.
+2. **Gmail Compact List / Split-Pane View**:
+   - High-density list: Checkbox, Star icon, Source Feed badge, Article Title, 1-line snippet/AI Summary preview, Timestamp.
+   - Split-pane layout: Clicking an item opens the reading pane on the right or below, retaining the list for fast scanning.
+
+#### Keyboard Shortcut Matrix (Google Reader & Gmail Paradigm)
+
+| Key Binding | Action |
+| :--- | :--- |
+| `j` / `k` | Navigate to Next / Previous article |
+| `Space` / `Shift + Space` | Page Down / Page Up through stream |
+| `o` or `Enter` | Expand / Collapse current item in compact view |
+| `v` | Open original article URL in a new browser tab |
+| `s` | Star / Unstar current item |
+| `m` | Toggle Read / Unread status |
+| `e` or `Shift + I` | Archive / Mark as Read and advance |
+| `Shift + A` | Mark all items in current view as read (with confirmation modal) |
+| `/` | Focus search bar |
+| `g` then `i` | Go to Inbox |
+| `g` then `s` | Go to Starred |
+| `g` then `d` | Go to Virtual Digests |
+| `?` | Open keyboard shortcuts help modal |
+
+#### PWA & Offline Support
+
+- Fully installable PWA with Web App Manifest and Service Worker caching.
+- IndexedDB client caching (Dexie.js) allowing offline reading of already synced unread/starred articles and summaries.
+- Offline action queuing (marking read, starring while offline syncs automatically when connection is restored).
+
+### 4.6 Search & Query Syntax (Gmail-Style Engine)
+
+The search bar supports advanced tokenized queries backed by database Full-Text Search:
+
+- `is:unread` / `is:read` / `is:starred`
+- `tag:tech` / `category:engineering`
+- `feed:"Hacker News"` / `from:author_name`
+- `has:summary` / `has:audio` / `has:share`
+- `score:>80` / `score:<50`
+- `after:2026-08-01` / `before:2026-08-20`
+- Raw keyword and phrase matching: `"generative ai" kubernetes`
+
+### 4.7 API & Ecosystem Integrations
+
+1. **Fever API & Google Reader API**:
+   - `/api/fever`: Implements standard Fever protocol endpoints (`?api`, `?items`, `?feeds`, `?groups`, `?unread_item_ids`, `?saved_item_ids`).
+   - `/api/greader`: Implements Google Reader API endpoints (`/reader/api/0/...`) for NetNewsWire, Reeder, and FeedMe.
+2. **AI / LLM Integration**:
+   - Generic OpenAI-compatible client (configurable `base_url`, `api_key`, `model_name`, `temperature`, `max_tokens`).
+   - Supports OpenAI (GPT-4o/Mini), Anthropic (via OpenAI proxy/native), Google Gemini, DeepSeek, Ollama, and LocalAI.
+3. **TTS Integration**:
+   - Free Microsoft Edge-TTS integration (high-quality neural voices with zero API key requirement).
+   - Optional OpenAI TTS (`tts-1`, `tts-1-hd`) and ElevenLabs API support.
+4. **Outbound Notification Webhooks**:
+   - Native Telegram Bot API formatting.
+   - Discord / Slack incoming webhook formatting.
+   - Generic HTTP POST with configurable headers and JSON template.
+
+---
+
+## 5. Technical Architecture & Data Schema
+
+### 5.1 Data Models (Relational & Flexible Metadata)
+
+```sql
+-- 1. Users Table (Single Admin Entry)
+CREATE TABLE users (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    username VARCHAR(64) UNIQUE NOT NULL,
+    password_hash VARCHAR(255) NOT NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+-- 2. Categories Table
+CREATE TABLE categories (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    title VARCHAR(128) NOT NULL,
+    parent_id UUID REFERENCES categories(id) ON DELETE SET NULL,
+    sort_order INT DEFAULT 0,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+-- 3. Feeds Table
+CREATE TABLE feeds (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    title VARCHAR(255) NOT NULL,
+    feed_url TEXT UNIQUE NOT NULL,
+    site_url TEXT,
+    category_id UUID REFERENCES categories(id) ON DELETE SET NULL,
+    is_virtual_digest BOOLEAN DEFAULT FALSE,
+    refresh_interval_minutes INT DEFAULT 30,
+    etag VARCHAR(255),
+    last_modified VARCHAR(255),
+    last_fetched_at TIMESTAMPTZ,
+    fetch_error TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+-- 4. Feed Entries Table
+CREATE TABLE feed_entries (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    feed_id UUID NOT NULL REFERENCES feeds(id) ON DELETE CASCADE,
+    guid TEXT NOT NULL,
+    url TEXT NOT NULL,
+    title TEXT NOT NULL,
+    author VARCHAR(255),
+    content TEXT NOT NULL,          -- Cleaned full-text / readability content
+    raw_content TEXT,              -- Original feed snippet
+    published_at TIMESTAMPTZ NOT NULL,
+    is_read BOOLEAN DEFAULT FALSE,
+    is_starred BOOLEAN DEFAULT FALSE,
+    share_uuid UUID UNIQUE,        -- Non-null if published to /share/<uuid>.html
+    
+    -- Dynamic Metadata & Extensibility (JSONB)
+    -- Contains: { "aiSummary": "...", "score": 85, "tags": ["ai", "go"], "ttsUrl": "..." }
+    metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
+    
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+    CONSTRAINT unique_feed_guid UNIQUE(feed_id, guid)
+);
+
+-- Indexes for Fast Triage and FTS
+CREATE INDEX idx_entries_feed_read ON feed_entries(feed_id, is_read);
+CREATE INDEX idx_entries_published ON feed_entries(published_at DESC);
+CREATE INDEX idx_entries_share_uuid ON feed_entries(share_uuid) WHERE share_uuid IS NOT NULL;
+CREATE INDEX idx_entries_metadata_gin ON feed_entries USING GIN (metadata);
+
+-- 5. Rules & Hooks Table
+CREATE TABLE rules (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    name VARCHAR(128) NOT NULL,
+    rule_type VARCHAR(32) NOT NULL, -- 'single_feed' | 'multi_feed_digest'
+    is_active BOOLEAN DEFAULT TRUE,
+    trigger_type VARCHAR(32) NOT NULL, -- 'on_ingest' | 'schedule' | 'threshold'
+    trigger_config JSONB NOT NULL DEFAULT '{}'::jsonb, -- { "cron": "0 8 * * *", "threshold": 15 }
+    conditions JSONB NOT NULL DEFAULT '[]'::jsonb,     -- [{ "field": "category", "op": "eq", "value": "tech" }]
+    actions JSONB NOT NULL DEFAULT '[]'::jsonb,        -- [{ "action": "ai_summarize" }, { "action": "generate_tts" }]
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+```
+
+### 5.2 Deployment & Packaging Specification
+
+- **Single Docker Container**:
+  - Base Image: `golang:1.23-alpine` (builder) + `alpine:latest` (runtime).
+  - Web Assets: Built using `vite build` and embedded into the Go binary using `//go:embed dist/*`.
+  - Database Support: Built-in driver for PostgreSQL (via `pgx`) and SQLite3 with WAL mode enabled.
+- **K3s / Kubernetes Manifests**:
+  - `Deployment`: 1 replica, resource limits ~64MB RAM / 0.1 CPU idle.
+  - `Service` & `Ingress`: Traefik ingress exposing web routes and public `/share/*` routes.
+  - `PersistentVolumeClaim` (optional for SQLite storage or local audio cache).
+
+---
+
+## 6. Milestone Roadmap & Implementation Breakdown
+
+```mermaid
+gantt
+    title RSSPipe Implementation Roadmap
+    dateFormat  YYYY-MM-DD
+    section M1: Core Foundation
+    Go Server, DB & Single-User Auth      :m1_1, 2026-09-01, 7d
+    Ingestion Engine & OPML Migration     :m1_2, after m1_1, 7d
+    section M2: PWA & Reading UX
+    React Vite PWA Setup & Theming        :m2_1, after m1_2, 5d
+    Google Reader & Gmail Keyboards / UI  :m2_2, after m2_1, 7d
+    Gmail Search Engine & Offline Cache   :m2_3, after m2_2, 5d
+    section M3: Single-Feed Hooks
+    Readability Full-Text Scraper         :m3_1, after m2_3, 4d
+    LLM Summarization & Scoring Hooks     :m3_2, after m3_1, 5d
+    Public Share /share/<uuid>.html       :m3_3, after m3_2, 4d
+    Outbound Webhooks (Telegram/Discord)  :m3_4, after m3_3, 3d
+    section M4: Multi-Feed Digest & Audio
+    Multi-Feed Batch Aggregator           :m4_1, after m3_4, 6d
+    Virtual Digest Feeds & Auto-Archive   :m4_2, after m4_1, 5d
+    Edge-TTS & Audio Player Integration   :m4_3, after m4_2, 5d
+    section M5: Ecosystem & Polishing
+    Fever / Google Reader API Layer       :m5_1, after m4_3, 6d
+    Rules UI Builder & DAG Validation     :m5_2, after m5_1, 5d
+    K3s Deployment & E2E Validation       :m5_3, after m5_2, 4d
+```
+
+### Milestone 1: Core Foundation & Ingestion Engine
+
+- [ ] Go backend server skeleton with Chi/Echo router and embedded static server.
+- [ ] Database layer supporting PostgreSQL and SQLite (WAL mode).
+- [ ] Single admin user authentication (Session Cookies, JWT, bcrypt/Argon2id).
+- [ ] Multi-format RSS/Atom/JSON parser with concurrent fetch worker pool.
+- [ ] OPML 2.0 import/export engine for seamless Miniflux migration.
+- [ ] Single Dockerfile packaging backend + frontend ready for k3s deployment.
+
+### Milestone 2: PWA Frontend & Google Reader / Gmail UX
+
+- [ ] React + Vite PWA boilerplate with Tailwind/Vanilla CSS design system, dark/light modes, and responsive layout.
+- [ ] Dual view modes: Google Reader continuous expanded stream + Gmail compact list/split-pane.
+- [ ] Full keyboard navigation workflow (`j`, `k`, `o`, `v`, `s`, `m`, `e`, `Shift+A`, `/`).
+- [ ] Service Worker and IndexedDB (Dexie.js) for local caching and offline reading.
+- [ ] Gmail-style tokenized search bar (`is:unread`, `tag:xxx`, `score:>80`, etc.).
+
+### Milestone 3: Single-Feed Hooks & Enrichment
+
+- [ ] Backend readability scraper to extract clean full-text markdown for truncated feeds.
+- [ ] Single-feed rules engine pipeline with trigger-condition-action execution.
+- [ ] OpenAI-compatible LLM connector for automated 3-bullet summary, key takeaways, and relevance scoring (0–100).
+- [ ] Public share generator creating standalone `/share/<uuid>.html` accessible without auth.
+- [ ] Outbound webhook dispatcher (Telegram, Discord, Slack, generic HTTP POST).
+
+### Milestone 4: Multi-Feed Digest Synthesis & Audio (TTS)
+
+- [ ] Multi-feed batch scheduler with cron and count-threshold triggers.
+- [ ] Multi-document LLM synthesis engine for creating structured daily/topic briefs.
+- [ ] Virtual Digest Feeds architecture (internal native feeds receiving synthesized articles).
+- [ ] Batch auto-archive / auto-mark read mechanism for processed source articles.
+- [ ] Audio generation engine (Edge-TTS and OpenAI TTS) with embedded audio player in PWA and public share pages.
+
+### Milestone 5: Ecosystem Compatibility & Final Polish
+
+- [ ] Fever API (`/api/fever`) and Google Reader API (`/api/greader`) implementation for native mobile app sync (NetNewsWire, Reeder).
+- [ ] Visual Rule Builder in Settings with DAG cycle validator (Topological Sort).
+- [ ] Telemetry & Reading Analytics (dwell time, reading progress).
+- [ ] Complete k3s deployment manifests (Deployment, Service, Ingress, PVC) and comprehensive documentation.
+
+---
+
+## 7. Success Metrics & Verification Criteria
+
+1. **Migration Fidelity**: 100% of feeds and categories from Miniflux OPML imported cleanly without data loss.
+2. **Triaging Speed**: Zero-lag keyboard navigation (`j`/`k`) with UI frame times strictly $< 16\text{ms}$ (60 FPS).
+3. **Resource Efficiency**: Single container idling at $< 64\text{MB}$ RAM and $< 1\%$ CPU on k3s cluster.
+4. **Hook Reliability**: 100% of ingestion hooks executed asynchronously without blocking feed fetch workers; zero DAG execution cycles.
+5. **Ecosystem Compatibility**: NetNewsWire / Reeder connect and synchronize unread/starred states without errors.