haopengzhan
ebcaa5d72a
- Implement atomic state transitions: add EntryService.PatchEntry() with EntryDiff computation and EventEntryTransition event emission. - Add persistent SQLite task queue (JobEngine): add jobs table with crash recovery, exponential backoff retries, concurrency control, and idempotency key deduplication. - Decompose monolithic HTTP handlers: split handlers.go into domain handlers (auth, feeds, entries, jobs, settings, rules, sse). - Implement real-time Server-Sent Events (/api/events/stream) for multi-tab/device state synchronization. - Fix SQLite concurrency: apply busy_timeout(10000) and journal_mode(WAL) via DSN query parameters to resolve SQLITE_BUSY (5). - Fix AI settings & prompt loading: unmarshal JSON from ai_settings, apply configured prompt templates, and enforce strict enabled/API key validations. - Fix undo share state: remove omitempty from ShareUUID to serialize null for state resets. - Fix middle list counter: compute true unread/total stats from DB and clean switcher button labels. - Adapt mobile UI for iOS: add viewport-fit=cover in index.html and calibrate safe-area insets for the bottom navigation and article reader toolbar. - Add architectural documentation and worklog in docs/009_worklog.md.
157 lines
11 KiB
Markdown
157 lines
11 KiB
Markdown
# RSSPipe Worklog & Architectural Refactoring (`009_worklog.md`)
|
|
|
|
This document records the comprehensive architectural overhaul, bug fixes, and engineering enhancements implemented across the RSSPipe frontend and backend.
|
|
|
|
---
|
|
|
|
## 1. Overview & Objectives
|
|
|
|
The primary goal of this milestone was to transform RSSPipe into a resilient, reactive, and easily extensible event-driven platform capable of executing asynchronous pipelines (e.g. readability extraction, AI summarization, tagging, text-to-speech) without concurrency bottlenecks or UI state drift.
|
|
|
|
### Core Problems Addressed
|
|
|
|
1. **Monolithic API Structure**: A single 1,140-line `handlers.go` held all business logic, making feature expansion error-prone.
|
|
2. **Missing State Transitions**: Internal background updates (e.g., crawler writes, variant additions) lacked declarative event propagation.
|
|
3. **Task Concurrency & Recovery**: Long-running or rate-limited tasks needed persistent queuing, exponential backoff retries, and crash recovery.
|
|
4. **SQLite Concurrency & Lock Collisions**: Concurrent worker writes threw `SQLITE_BUSY (5)` due to pool-level PRAGMA isolation.
|
|
5. **Cross-Tab State Synchronization**: Updating read/star status in one session did not reflect in other sessions or tabs.
|
|
6. **Mobile UX on iOS**: Fixed bottom action bars overlapping with the iOS virtual home indicator bar.
|
|
|
|
---
|
|
|
|
## 2. Architectural Changes
|
|
|
|
```
|
|
┌───────────────────────────────────────────────────────────┐
|
|
│ HTTP API / Worker │
|
|
└─────────────────────────────┬─────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────┐
|
|
│ EntryService.PatchEntry() │
|
|
│ (Atomic DB Update + EntryDiff) │
|
|
└────────────────┬─────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────┐
|
|
│ EventBus.Publish(Transition) │
|
|
└────────┬─────────────────┬───────┘
|
|
│ │
|
|
┌────────────────────┘ └────────────────────┐
|
|
▼ ▼
|
|
┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐
|
|
│ Server-Sent Events (SSE) │ │ RulesEngine │
|
|
│ GET /api/events/stream │ │ (Evaluate Old vs New State) │
|
|
└──────────────────┬───────────────────┘ └──────────────────┬───────────────────┘
|
|
│ │
|
|
▼ ▼
|
|
┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐
|
|
│ Client Tabs (Real-Time Reactive UI) │ │ JobEngine.Enqueue(Job) │
|
|
└──────────────────────────────────────┘ └──────────────────┬───────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────┐
|
|
│ Persistent SQLite `jobs` Table │
|
|
│ (Workers, Retries, Idempotency) │
|
|
└──────────────────────────────────────┘
|
|
```
|
|
|
|
### 2.1 Atomic State Transitions & Event Bus
|
|
|
|
- Implemented `PatchEntry(ctx, id, patch)` in `internal/services/entry_service.go`.
|
|
- Computes atomic `EntryDiff` comparing pre-update and post-update state:
|
|
- Boolean flag changes (`is_read`, `is_starred`).
|
|
- Added or updated variants (e.g. `readability_full_text`, `ai_summary`, `tts_audio`).
|
|
- Publishes `EventEntryTransition(OldState, NewState, Diff)` onto `internal/events/bus.go`.
|
|
- Enables declarative rules evaluation on every state change without tight coupling.
|
|
|
|
### 2.2 Persistent SQLite Job Queue (`JobEngine`)
|
|
|
|
- Created `jobs` table in SQLite schema with lifecycle states (`pending`, `running`, `completed`, `failed`, `dead`).
|
|
- Added worker pool in `internal/worker/engine.go` with:
|
|
- Concurrency limits (default 5 concurrent workers).
|
|
- Exponential backoff retries (`run_at` scheduling).
|
|
- Idempotency key deduplication (prevents duplicate execution of identical tasks).
|
|
- Crash recovery on engine startup (resets zombie `running` jobs back to `pending`).
|
|
- Registered handlers for:
|
|
- `readability_scrape` (Readability full-text extraction).
|
|
- `ai_summary` / `ai_enrich` (LLM summarization and variant generation).
|
|
- `mark_read` / `mark_starred` (Status modifications).
|
|
- `set_variant` (Structured tag and variant persistence).
|
|
|
|
### 2.3 Modular Backend HTTP Handlers
|
|
|
|
Decomposed the monolithic `handlers.go` into domain-specific files under `internal/api/`:
|
|
|
|
- `auth_handler.go`: Authentication, session verification, login, and logout.
|
|
- `feeds_handler.go`: Feed CRUD, OPML import/export, manual refresh, and favicon resolution.
|
|
- `entries_handler.go`: Entry queries, filtering, atomic patch, scrape, and AI actions.
|
|
- `jobs_handler.go`: Job queue querying, execution status, and asynchronous dispatch.
|
|
- `settings_handler.go`: User preferences and AI provider configuration.
|
|
- `rules_handler.go`: Automation rule CRUD and test execution.
|
|
- `sse_handler.go`: Server-Sent Events real-time event streaming.
|
|
|
|
### 2.4 Real-Time Reactive Synchronization (SSE)
|
|
|
|
- Added `GET /api/events/stream` endpoint authenticated via Bearer token / session cookie.
|
|
- Connected event stream in `web/src/App.tsx` via `api.connectEventStream()`.
|
|
- Automatically syncs read status, star status, and variant additions in real-time across multiple open browser tabs or devices.
|
|
|
|
---
|
|
|
|
## 3. Critical Bug Fixes
|
|
|
|
### 3.1 SQLite `SQLITE_BUSY (5)` Database Lock Fix
|
|
|
|
- **Root Cause**: In Go's `database/sql`, executing `PRAGMA busy_timeout = 5000;` on startup only applied to a single pooled connection. Subsequent pool connections defaulted to `0ms` timeout, immediately throwing `SQLITE_BUSY` when concurrent writes occurred.
|
|
- **Solution**: Configured PRAGMAs directly in the SQLite DSN connection string in `internal/db/db.go`:
|
|
|
|
```go
|
|
dsn := fmt.Sprintf("%s?_pragma=busy_timeout(10000)&_pragma=journal_mode(WAL)&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(ON)&_pragma=cache_size(-20000)", dbPath)
|
|
```
|
|
|
|
Every connection checked out of the pool now inherits a 10-second busy timeout and WAL mode.
|
|
|
|
### 3.2 AI Settings & Prompt Template Unification
|
|
|
|
- **Root Cause**: `SettingsService` saved AI configuration as a unified JSON object under key `"ai_settings"`, but `LoadConfig` in the AI actions was reading non-existent legacy individual keys (`ai_enabled`, `ai_api_key`). This caused prompt templates to be ignored and disabled states to be bypassed.
|
|
- **Solution**: Refactored `LoadConfig` in `internal/actions/ai/enrich.go` to unmarshal `"ai_settings"` JSON. Enforced strict enabled checks and wired custom prompt templates (e.g. prompt `"a": "aaa"`).
|
|
|
|
### 3.3 Undo Share State Reset
|
|
|
|
- **Root Cause**: `models.FeedEntry` had `omitempty` on `ShareUUID *string`, which omitted `share_uuid` from JSON responses when set to `nil` rather than outputting `null`.
|
|
- **Solution**: Removed `omitempty` from `ShareUUID` in `internal/models/models.go` and updated `DesktopReader.tsx` and `MobileArticleSheet.tsx` to explicitly reset `share_uuid: res.share_uuid ?? null`.
|
|
|
|
### 3.4 Middle Bar Unread / All Count Discrepancy
|
|
|
|
- **Root Cause**: The middle article list header count displayed in-memory array lengths (`entries.length`) rather than true database statistics from `stats` and feed/category metadata.
|
|
- **Solution**: Updated `DesktopArticleList.tsx` to compute true database counts from `stats` and feed totals, and reverted switch buttons to clean `Unread` and `All` labels.
|
|
|
|
### 3.5 Mobile iOS Safe Area Insets & Home Indicator
|
|
|
|
- **Root Cause**: `viewport-fit=cover` was missing in `web/index.html`, causing iOS Safari to evaluate `env(safe-area-inset-bottom)` to `0px`.
|
|
- **Solution**: Added `viewport-fit=cover` and calibrated bottom navigation height (`50px + env(safe-area-inset-bottom, 0px)`) and article bottom toolbar height (`48px + env(safe-area-inset-bottom, 0px)`).
|
|
|
|
---
|
|
|
|
## 4. Work Items & Backlog for Future Milestones
|
|
|
|
The following items have been identified for future implementation:
|
|
|
|
1. **SSE Auto-Reconnect & Connection State Recovery**:
|
|
- Exponential backoff with jitter on EventSource disconnection in `web/src/api.ts`.
|
|
- Snapshot reconciliation (`/api/stats` and `/api/entries?since=...`) upon reconnect.
|
|
|
|
2. **Background Job Queue & Dead-Letter Management in Admin Settings**:
|
|
- Dedicated "Job Queue" tab in `Settings > Account & System`.
|
|
- Live table of tasks (`pending`, `running`, `completed`, `failed`, `dead`) with error logs and manual retry/cancel actions.
|
|
|
|
3. **Reader UI Prompt Template Selector**:
|
|
- Dropdown selector next to "Generate AI Summary" button in Desktop and Mobile reader bars to choose between configured prompt templates on the fly.
|
|
|
|
4. **Local Ollama Model Autodiscovery & Health Check**:
|
|
- "Discover Models" button in `Settings > AI & Integrations` querying `http://localhost:11434/api/tags` to populate the model list automatically.
|
|
|
|
5. **Desktop 3-Pane Middle List Infinite Scrolling**:
|
|
- Windowed offset pagination (`limit: 50`, `offset: entries.length`) with scroll threshold trigger on `.desktop-articles-scroll`.
|