# RSSPipe Worklog & Architectural Refactoring (`009_worklog.md`) This document records the comprehensive architectural overhaul, bug fixes, and engineering enhancements implemented across the RSSPipe frontend and backend. --- ## 1. Overview & Objectives The primary goal of this milestone was to transform RSSPipe into a resilient, reactive, and easily extensible event-driven platform capable of executing asynchronous pipelines (e.g. readability extraction, AI summarization, tagging, text-to-speech) without concurrency bottlenecks or UI state drift. ### Core Problems Addressed 1. **Monolithic API Structure**: A single 1,140-line `handlers.go` held all business logic, making feature expansion error-prone. 2. **Missing State Transitions**: Internal background updates (e.g., crawler writes, variant additions) lacked declarative event propagation. 3. **Task Concurrency & Recovery**: Long-running or rate-limited tasks needed persistent queuing, exponential backoff retries, and crash recovery. 4. **SQLite Concurrency & Lock Collisions**: Concurrent worker writes threw `SQLITE_BUSY (5)` due to pool-level PRAGMA isolation. 5. **Cross-Tab State Synchronization**: Updating read/star status in one session did not reflect in other sessions or tabs. 6. **Mobile UX on iOS**: Fixed bottom action bars overlapping with the iOS virtual home indicator bar. --- ## 2. Architectural Changes ``` ┌───────────────────────────────────────────────────────────┐ │ HTTP API / Worker │ └─────────────────────────────┬─────────────────────────────┘ │ ▼ ┌──────────────────────────────────┐ │ EntryService.PatchEntry() │ │ (Atomic DB Update + EntryDiff) │ └────────────────┬─────────────────┘ │ ▼ ┌──────────────────────────────────┐ │ EventBus.Publish(Transition) │ └────────┬─────────────────┬───────┘ │ │ ┌────────────────────┘ └────────────────────┐ ▼ ▼ ┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐ │ Server-Sent Events (SSE) │ │ RulesEngine │ │ GET /api/events/stream │ │ (Evaluate Old vs New State) │ └──────────────────┬───────────────────┘ └──────────────────┬───────────────────┘ │ │ ▼ ▼ ┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐ │ Client Tabs (Real-Time Reactive UI) │ │ JobEngine.Enqueue(Job) │ └──────────────────────────────────────┘ └──────────────────┬───────────────────┘ │ ▼ ┌──────────────────────────────────────┐ │ Persistent SQLite `jobs` Table │ │ (Workers, Retries, Idempotency) │ └──────────────────────────────────────┘ ``` ### 2.1 Atomic State Transitions & Event Bus - Implemented `PatchEntry(ctx, id, patch)` in `internal/services/entry_service.go`. - Computes atomic `EntryDiff` comparing pre-update and post-update state: - Boolean flag changes (`is_read`, `is_starred`). - Added or updated variants (e.g. `readability_full_text`, `ai_summary`, `tts_audio`). - Publishes `EventEntryTransition(OldState, NewState, Diff)` onto `internal/events/bus.go`. - Enables declarative rules evaluation on every state change without tight coupling. ### 2.2 Persistent SQLite Job Queue (`JobEngine`) - Created `jobs` table in SQLite schema with lifecycle states (`pending`, `running`, `completed`, `failed`, `dead`). - Added worker pool in `internal/worker/engine.go` with: - Concurrency limits (default 5 concurrent workers). - Exponential backoff retries (`run_at` scheduling). - Idempotency key deduplication (prevents duplicate execution of identical tasks). - Crash recovery on engine startup (resets zombie `running` jobs back to `pending`). - Registered handlers for: - `readability_scrape` (Readability full-text extraction). - `ai_summary` / `ai_enrich` (LLM summarization and variant generation). - `mark_read` / `mark_starred` (Status modifications). - `set_variant` (Structured tag and variant persistence). ### 2.3 Modular Backend HTTP Handlers Decomposed the monolithic `handlers.go` into domain-specific files under `internal/api/`: - `auth_handler.go`: Authentication, session verification, login, and logout. - `feeds_handler.go`: Feed CRUD, OPML import/export, manual refresh, and favicon resolution. - `entries_handler.go`: Entry queries, filtering, atomic patch, scrape, and AI actions. - `jobs_handler.go`: Job queue querying, execution status, and asynchronous dispatch. - `settings_handler.go`: User preferences and AI provider configuration. - `rules_handler.go`: Automation rule CRUD and test execution. - `sse_handler.go`: Server-Sent Events real-time event streaming. ### 2.4 Real-Time Reactive Synchronization (SSE) - Added `GET /api/events/stream` endpoint authenticated via Bearer token / session cookie. - Connected event stream in `web/src/App.tsx` via `api.connectEventStream()`. - Automatically syncs read status, star status, and variant additions in real-time across multiple open browser tabs or devices. --- ## 3. Critical Bug Fixes ### 3.1 SQLite `SQLITE_BUSY (5)` Database Lock Fix - **Root Cause**: In Go's `database/sql`, executing `PRAGMA busy_timeout = 5000;` on startup only applied to a single pooled connection. Subsequent pool connections defaulted to `0ms` timeout, immediately throwing `SQLITE_BUSY` when concurrent writes occurred. - **Solution**: Configured PRAGMAs directly in the SQLite DSN connection string in `internal/db/db.go`: ```go dsn := fmt.Sprintf("%s?_pragma=busy_timeout(10000)&_pragma=journal_mode(WAL)&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(ON)&_pragma=cache_size(-20000)", dbPath) ``` Every connection checked out of the pool now inherits a 10-second busy timeout and WAL mode. ### 3.2 AI Settings & Prompt Template Unification - **Root Cause**: `SettingsService` saved AI configuration as a unified JSON object under key `"ai_settings"`, but `LoadConfig` in the AI actions was reading non-existent legacy individual keys (`ai_enabled`, `ai_api_key`). This caused prompt templates to be ignored and disabled states to be bypassed. - **Solution**: Refactored `LoadConfig` in `internal/actions/ai/enrich.go` to unmarshal `"ai_settings"` JSON. Enforced strict enabled checks and wired custom prompt templates (e.g. prompt `"a": "aaa"`). ### 3.3 Undo Share State Reset - **Root Cause**: `models.FeedEntry` had `omitempty` on `ShareUUID *string`, which omitted `share_uuid` from JSON responses when set to `nil` rather than outputting `null`. - **Solution**: Removed `omitempty` from `ShareUUID` in `internal/models/models.go` and updated `DesktopReader.tsx` and `MobileArticleSheet.tsx` to explicitly reset `share_uuid: res.share_uuid ?? null`. ### 3.4 Middle Bar Unread / All Count Discrepancy - **Root Cause**: The middle article list header count displayed in-memory array lengths (`entries.length`) rather than true database statistics from `stats` and feed/category metadata. - **Solution**: Updated `DesktopArticleList.tsx` to compute true database counts from `stats` and feed totals, and reverted switch buttons to clean `Unread` and `All` labels. ### 3.5 Mobile iOS Safe Area Insets & Home Indicator - **Root Cause**: `viewport-fit=cover` was missing in `web/index.html`, causing iOS Safari to evaluate `env(safe-area-inset-bottom)` to `0px`. - **Solution**: Added `viewport-fit=cover` and calibrated bottom navigation height (`50px + env(safe-area-inset-bottom, 0px)`) and article bottom toolbar height (`48px + env(safe-area-inset-bottom, 0px)`). --- ## 4. Work Items & Backlog for Future Milestones The following items have been identified for future implementation: 1. **SSE Auto-Reconnect & Connection State Recovery**: - Exponential backoff with jitter on EventSource disconnection in `web/src/api.ts`. - Snapshot reconciliation (`/api/stats` and `/api/entries?since=...`) upon reconnect. 2. **Background Job Queue & Dead-Letter Management in Admin Settings**: - Dedicated "Job Queue" tab in `Settings > Account & System`. - Live table of tasks (`pending`, `running`, `completed`, `failed`, `dead`) with error logs and manual retry/cancel actions. 3. **Reader UI Prompt Template Selector**: - Dropdown selector next to "Generate AI Summary" button in Desktop and Mobile reader bars to choose between configured prompt templates on the fly. 4. **Local Ollama Model Autodiscovery & Health Check**: - "Discover Models" button in `Settings > AI & Integrations` querying `http://localhost:11434/api/tags` to populate the model list automatically. 5. **Desktop 3-Pane Middle List Infinite Scrolling**: - Windowed offset pagination (`limit: 50`, `offset: entries.length`) with scroll threshold trigger on `.desktop-articles-scroll`.