rsspipe/docs/009_worklog.md

RSSPipe Worklog & Architectural Refactoring (009_worklog.md)

This document records the comprehensive architectural overhaul, bug fixes, and engineering enhancements implemented across the RSSPipe frontend and backend.

---

1. Overview & Objectives

The primary goal of this milestone was to transform RSSPipe into a resilient, reactive, and easily extensible event-driven platform capable of executing asynchronous pipelines (e.g. readability extraction, AI summarization, tagging, text-to-speech) without concurrency bottlenecks or UI state drift.

Core Problems Addressed

1. Monolithic API Structure: A single 1,140-line handlers.go held all business logic, making feature expansion error-prone.
2. Missing State Transitions: Internal background updates (e.g., crawler writes, variant additions) lacked declarative event propagation.
3. Task Concurrency & Recovery: Long-running or rate-limited tasks needed persistent queuing, exponential backoff retries, and crash recovery.
4. SQLite Concurrency & Lock Collisions: Concurrent worker writes threw SQLITE_BUSY (5) due to pool-level PRAGMA isolation.
5. Cross-Tab State Synchronization: Updating read/star status in one session did not reflect in other sessions or tabs.
6. Mobile UX on iOS: Fixed bottom action bars overlapping with the iOS virtual home indicator bar.

---

2. Architectural Changes

                  ┌───────────────────────────────────────────────────────────┐
                  │                      HTTP API / Worker                    │
                  └─────────────────────────────┬─────────────────────────────┘
                                                │
                                                ▼
                               ┌──────────────────────────────────┐
                               │   EntryService.PatchEntry()      │
                               │  (Atomic DB Update + EntryDiff)  │
                               └────────────────┬─────────────────┘
                                                │
                                                ▼
                               ┌──────────────────────────────────┐
                               │   EventBus.Publish(Transition)   │
                               └────────┬─────────────────┬───────┘
                                        │                 │
                   ┌────────────────────┘                 └────────────────────┐
                   ▼                                                           ▼
┌──────────────────────────────────────┐                   ┌──────────────────────────────────────┐
│       Server-Sent Events (SSE)       │                   │             RulesEngine              │
│       GET /api/events/stream         │                   │    (Evaluate Old vs New State)       │
└──────────────────┬───────────────────┘                   └──────────────────┬───────────────────┘
                   │                                                           │
                   ▼                                                           ▼
┌──────────────────────────────────────┐                   ┌──────────────────────────────────────┐
│  Client Tabs (Real-Time Reactive UI) │                   │       JobEngine.Enqueue(Job)         │
└──────────────────────────────────────┘                   └──────────────────┬───────────────────┘
                                                                               │
                                                                               ▼
                                                           ┌──────────────────────────────────────┐
                                                           │   Persistent SQLite `jobs` Table     │
                                                           │  (Workers, Retries, Idempotency)     │
                                                           └──────────────────────────────────────┘

2.1 Atomic State Transitions & Event Bus

• Implemented PatchEntry(ctx, id, patch) in internal/services/entry_service.go.
• Computes atomic EntryDiff comparing pre-update and post-update state:
  • Boolean flag changes (is_read, is_starred).
  • Added or updated variants (e.g. readability_full_text, ai_summary, tts_audio).
• Publishes EventEntryTransition(OldState, NewState, Diff) onto internal/events/bus.go.
• Enables declarative rules evaluation on every state change without tight coupling.

2.2 Persistent SQLite Job Queue (JobEngine)

• Created jobs table in SQLite schema with lifecycle states (pending, running, completed, failed, dead).
• Added worker pool in internal/worker/engine.go with:
  • Concurrency limits (default 5 concurrent workers).
  • Exponential backoff retries (run_at scheduling).
  • Idempotency key deduplication (prevents duplicate execution of identical tasks).
  • Crash recovery on engine startup (resets zombie running jobs back to pending).
• Registered handlers for:
  • readability_scrape (Readability full-text extraction).
  • ai_summary / ai_enrich (LLM summarization and variant generation).
  • mark_read / mark_starred (Status modifications).
  • set_variant (Structured tag and variant persistence).

2.3 Modular Backend HTTP Handlers

Decomposed the monolithic handlers.go into domain-specific files under internal/api/:

• auth_handler.go: Authentication, session verification, login, and logout.
• feeds_handler.go: Feed CRUD, OPML import/export, manual refresh, and favicon resolution.
• entries_handler.go: Entry queries, filtering, atomic patch, scrape, and AI actions.
• jobs_handler.go: Job queue querying, execution status, and asynchronous dispatch.
• settings_handler.go: User preferences and AI provider configuration.
• rules_handler.go: Automation rule CRUD and test execution.
• sse_handler.go: Server-Sent Events real-time event streaming.

2.4 Real-Time Reactive Synchronization (SSE)

• Added GET /api/events/stream endpoint authenticated via Bearer token / session cookie.
• Connected event stream in web/src/App.tsx via api.connectEventStream().
• Automatically syncs read status, star status, and variant additions in real-time across multiple open browser tabs or devices.

---

3. Critical Bug Fixes

3.1 SQLite SQLITE_BUSY (5) Database Lock Fix

• Root Cause: In Go's database/sql, executing PRAGMA busy_timeout = 5000; on startup only applied to a single pooled connection. Subsequent pool connections defaulted to 0ms timeout, immediately throwing SQLITE_BUSY when concurrent writes occurred.

• Solution: Configured PRAGMAs directly in the SQLite DSN connection string in internal/db/db.go:

  dsn := fmt.Sprintf("%s?_pragma=busy_timeout(10000)&_pragma=journal_mode(WAL)&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(ON)&_pragma=cache_size(-20000)", dbPath)
  

  Every connection checked out of the pool now inherits a 10-second busy timeout and WAL mode.

3.2 AI Settings & Prompt Template Unification

• Root Cause: SettingsService saved AI configuration as a unified JSON object under key "ai_settings", but LoadConfig in the AI actions was reading non-existent legacy individual keys (ai_enabled, ai_api_key). This caused prompt templates to be ignored and disabled states to be bypassed.
• Solution: Refactored LoadConfig in internal/actions/ai/enrich.go to unmarshal "ai_settings" JSON. Enforced strict enabled checks and wired custom prompt templates (e.g. prompt "a": "aaa").

3.3 Undo Share State Reset

• Root Cause: models.FeedEntry had omitempty on ShareUUID *string, which omitted share_uuid from JSON responses when set to nil rather than outputting null.
• Solution: Removed omitempty from ShareUUID in internal/models/models.go and updated DesktopReader.tsx and MobileArticleSheet.tsx to explicitly reset share_uuid: res.share_uuid ?? null.

3.4 Middle Bar Unread / All Count Discrepancy

• Root Cause: The middle article list header count displayed in-memory array lengths (entries.length) rather than true database statistics from stats and feed/category metadata.
• Solution: Updated DesktopArticleList.tsx to compute true database counts from stats and feed totals, and reverted switch buttons to clean Unread and All labels.

3.5 Mobile iOS Safe Area Insets & Home Indicator

• Root Cause: viewport-fit=cover was missing in web/index.html, causing iOS Safari to evaluate env(safe-area-inset-bottom) to 0px.
• Solution: Added viewport-fit=cover and calibrated bottom navigation height (50px + env(safe-area-inset-bottom, 0px)) and article bottom toolbar height (48px + env(safe-area-inset-bottom, 0px)).

---

4. Work Items & Backlog for Future Milestones

The following items have been identified for future implementation:

1. SSE Auto-Reconnect & Connection State Recovery:

   • Exponential backoff with jitter on EventSource disconnection in web/src/api.ts.
   • Snapshot reconciliation (/api/stats and /api/entries?since=...) upon reconnect.

2. Background Job Queue & Dead-Letter Management in Admin Settings:

   • Dedicated "Job Queue" tab in Settings > Account & System.
   • Live table of tasks (pending, running, completed, failed, dead) with error logs and manual retry/cancel actions.

3. Reader UI Prompt Template Selector:

   • Dropdown selector next to "Generate AI Summary" button in Desktop and Mobile reader bars to choose between configured prompt templates on the fly.

4. Local Ollama Model Autodiscovery & Health Check:

   • "Discover Models" button in Settings > AI & Integrations querying http://localhost:11434/api/tags to populate the model list automatically.

5. Desktop 3-Pane Middle List Infinite Scrolling:

   • Windowed offset pagination (limit: 50, offset: entries.length) with scroll threshold trigger on .desktop-articles-scroll.