refactor(core): overhaul state transition architecture, persistent job queue, and API layer
CI Quality Checks & Tests / Gitleaks Secret Detection (push) Failing after 1m59s
CI Quality Checks & Tests / Go Lint & Unit Tests (push) Failing after 2m26s
CI Quality Checks & Tests / Frontend Typecheck & Build (push) Successful in 2m37s

- Implement atomic state transitions: add EntryService.PatchEntry() with EntryDiff computation and EventEntryTransition event emission.
- Add persistent SQLite task queue (JobEngine): add jobs table with crash recovery, exponential backoff retries, concurrency control, and idempotency key deduplication.
- Decompose monolithic HTTP handlers: split handlers.go into domain handlers (auth, feeds, entries, jobs, settings, rules, sse).
- Implement real-time Server-Sent Events (/api/events/stream) for multi-tab/device state synchronization.
- Fix SQLite concurrency: apply busy_timeout(10000) and journal_mode(WAL) via DSN query parameters to resolve SQLITE_BUSY (5).
- Fix AI settings & prompt loading: unmarshal JSON from ai_settings, apply configured prompt templates, and enforce strict enabled/API key validations.
- Fix undo share state: remove omitempty from ShareUUID to serialize null for state resets.
- Fix middle list counter: compute true unread/total stats from DB and clean switcher button labels.
- Adapt mobile UI for iOS: add viewport-fit=cover in index.html and calibrate safe-area insets for the bottom navigation and article reader toolbar.
- Add architectural documentation and worklog in docs/009_worklog.md.
This commit is contained in:
2026-09-01 00:41:05 -07:00
parent 271a542196
commit ebcaa5d72a
38 changed files with 3748 additions and 1202 deletions
+28 -13
View File
@@ -17,6 +17,7 @@ import (
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/logger"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/rules"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/services"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/worker"
"go.uber.org/zap"
)
@@ -46,24 +47,37 @@ func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// 3. Initialize EventBus, WorkerPool, and Action Handlers
// 3. Initialize EventBus
bus := events.NewEventBus()
pool := worker.NewPool(database, bus, 5, 1000)
// Register Core Action Handlers (readability, status, variants, AI enrich & triage)
actions.RegisterDefaultHandlers(pool, database, log)
pool.Start(ctx)
// 4. Initialize Domain Services
entryService := services.NewEntryService(database, bus)
jobService := services.NewJobService(database, bus)
settingsService := services.NewSettingsService(database, bus)
// 4. Initialize Rules Engine
engine := rules.NewEngine(database, bus, pool)
engine.Start(ctx)
// 5. Start Background Crawler Pool
// 5. Start Background Crawler
cr := crawler.NewCrawler(database, bus, cfg.CrawlerIntervalMinutes, cfg.CrawlerWorkers)
cr.Start(ctx)
// 6. Mount API Routes
handler := api.NewHandler(database, authService, cr, pool)
feedService := services.NewFeedService(database, entryService, cr)
// 6. Initialize Persistent SQLite Task Queue Engine
jobEngine := worker.NewJobEngine(database, bus, entryService, 10)
actions.RegisterJobEngineHandlers(jobEngine, database, log)
jobEngine.Start(ctx)
// 7. Legacy WorkerPool (for backward compatibility if needed)
pool := worker.NewPool(database, bus, 5, 1000)
actions.RegisterDefaultHandlers(pool, database, log)
pool.Start(ctx)
// 8. Initialize Rules Engine (Transition-Aware Evaluator)
rulesEngine := rules.NewEngine(database, bus, pool)
rulesEngine.SetEnqueuer(jobService)
rulesEngine.Start(ctx)
// 9. Mount API Routes
handler := api.NewHandler(database, authService, bus, entryService, feedService, jobService, settingsService, jobEngine)
router := api.NewRouter(handler, authService, "./web/dist")
server := &http.Server{
@@ -74,7 +88,7 @@ func main() {
IdleTimeout: 120 * time.Second,
}
// 7. Graceful shutdown handler
// 10. Graceful shutdown handler
go func() {
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
@@ -82,6 +96,7 @@ func main() {
log.Info("Shutting down RSSPipe gracefully...")
cancel()
jobEngine.Stop()
pool.Stop()
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
+156
View File
@@ -0,0 +1,156 @@
# RSSPipe Worklog & Architectural Refactoring (`009_worklog.md`)
This document records the comprehensive architectural overhaul, bug fixes, and engineering enhancements implemented across the RSSPipe frontend and backend.
---
## 1. Overview & Objectives
The primary goal of this milestone was to transform RSSPipe into a resilient, reactive, and easily extensible event-driven platform capable of executing asynchronous pipelines (e.g. readability extraction, AI summarization, tagging, text-to-speech) without concurrency bottlenecks or UI state drift.
### Core Problems Addressed
1. **Monolithic API Structure**: A single 1,140-line `handlers.go` held all business logic, making feature expansion error-prone.
2. **Missing State Transitions**: Internal background updates (e.g., crawler writes, variant additions) lacked declarative event propagation.
3. **Task Concurrency & Recovery**: Long-running or rate-limited tasks needed persistent queuing, exponential backoff retries, and crash recovery.
4. **SQLite Concurrency & Lock Collisions**: Concurrent worker writes threw `SQLITE_BUSY (5)` due to pool-level PRAGMA isolation.
5. **Cross-Tab State Synchronization**: Updating read/star status in one session did not reflect in other sessions or tabs.
6. **Mobile UX on iOS**: Fixed bottom action bars overlapping with the iOS virtual home indicator bar.
---
## 2. Architectural Changes
```
┌───────────────────────────────────────────────────────────┐
│ HTTP API / Worker │
└─────────────────────────────┬─────────────────────────────┘
│
▼
┌──────────────────────────────────┐
│ EntryService.PatchEntry() │
│ (Atomic DB Update + EntryDiff) │
└────────────────┬─────────────────┘
│
▼
┌──────────────────────────────────┐
│ EventBus.Publish(Transition) │
└────────┬─────────────────┬───────┘
│ │
┌────────────────────┘ └────────────────────┐
▼ ▼
┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐
│ Server-Sent Events (SSE) │ │ RulesEngine │
│ GET /api/events/stream │ │ (Evaluate Old vs New State) │
└──────────────────┬───────────────────┘ └──────────────────┬───────────────────┘
│ │
▼ ▼
┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐
│ Client Tabs (Real-Time Reactive UI) │ │ JobEngine.Enqueue(Job) │
└──────────────────────────────────────┘ └──────────────────┬───────────────────┘
│
▼
┌──────────────────────────────────────┐
│ Persistent SQLite `jobs` Table │
│ (Workers, Retries, Idempotency) │
└──────────────────────────────────────┘
```
### 2.1 Atomic State Transitions & Event Bus
- Implemented `PatchEntry(ctx, id, patch)` in `internal/services/entry_service.go`.
- Computes atomic `EntryDiff` comparing pre-update and post-update state:
- Boolean flag changes (`is_read`, `is_starred`).
- Added or updated variants (e.g. `readability_full_text`, `ai_summary`, `tts_audio`).
- Publishes `EventEntryTransition(OldState, NewState, Diff)` onto `internal/events/bus.go`.
- Enables declarative rules evaluation on every state change without tight coupling.
### 2.2 Persistent SQLite Job Queue (`JobEngine`)
- Created `jobs` table in SQLite schema with lifecycle states (`pending`, `running`, `completed`, `failed`, `dead`).
- Added worker pool in `internal/worker/engine.go` with:
- Concurrency limits (default 5 concurrent workers).
- Exponential backoff retries (`run_at` scheduling).
- Idempotency key deduplication (prevents duplicate execution of identical tasks).
- Crash recovery on engine startup (resets zombie `running` jobs back to `pending`).
- Registered handlers for:
- `readability_scrape` (Readability full-text extraction).
- `ai_summary` / `ai_enrich` (LLM summarization and variant generation).
- `mark_read` / `mark_starred` (Status modifications).
- `set_variant` (Structured tag and variant persistence).
### 2.3 Modular Backend HTTP Handlers
Decomposed the monolithic `handlers.go` into domain-specific files under `internal/api/`:
- `auth_handler.go`: Authentication, session verification, login, and logout.
- `feeds_handler.go`: Feed CRUD, OPML import/export, manual refresh, and favicon resolution.
- `entries_handler.go`: Entry queries, filtering, atomic patch, scrape, and AI actions.
- `jobs_handler.go`: Job queue querying, execution status, and asynchronous dispatch.
- `settings_handler.go`: User preferences and AI provider configuration.
- `rules_handler.go`: Automation rule CRUD and test execution.
- `sse_handler.go`: Server-Sent Events real-time event streaming.
### 2.4 Real-Time Reactive Synchronization (SSE)
- Added `GET /api/events/stream` endpoint authenticated via Bearer token / session cookie.
- Connected event stream in `web/src/App.tsx` via `api.connectEventStream()`.
- Automatically syncs read status, star status, and variant additions in real-time across multiple open browser tabs or devices.
---
## 3. Critical Bug Fixes
### 3.1 SQLite `SQLITE_BUSY (5)` Database Lock Fix
- **Root Cause**: In Go's `database/sql`, executing `PRAGMA busy_timeout = 5000;` on startup only applied to a single pooled connection. Subsequent pool connections defaulted to `0ms` timeout, immediately throwing `SQLITE_BUSY` when concurrent writes occurred.
- **Solution**: Configured PRAGMAs directly in the SQLite DSN connection string in `internal/db/db.go`:
```go
dsn := fmt.Sprintf("%s?_pragma=busy_timeout(10000)&_pragma=journal_mode(WAL)&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(ON)&_pragma=cache_size(-20000)", dbPath)
```
Every connection checked out of the pool now inherits a 10-second busy timeout and WAL mode.
### 3.2 AI Settings & Prompt Template Unification
- **Root Cause**: `SettingsService` saved AI configuration as a unified JSON object under key `"ai_settings"`, but `LoadConfig` in the AI actions was reading non-existent legacy individual keys (`ai_enabled`, `ai_api_key`). This caused prompt templates to be ignored and disabled states to be bypassed.
- **Solution**: Refactored `LoadConfig` in `internal/actions/ai/enrich.go` to unmarshal `"ai_settings"` JSON. Enforced strict enabled checks and wired custom prompt templates (e.g. prompt `"a": "aaa"`).
### 3.3 Undo Share State Reset
- **Root Cause**: `models.FeedEntry` had `omitempty` on `ShareUUID *string`, which omitted `share_uuid` from JSON responses when set to `nil` rather than outputting `null`.
- **Solution**: Removed `omitempty` from `ShareUUID` in `internal/models/models.go` and updated `DesktopReader.tsx` and `MobileArticleSheet.tsx` to explicitly reset `share_uuid: res.share_uuid ?? null`.
### 3.4 Middle Bar Unread / All Count Discrepancy
- **Root Cause**: The middle article list header count displayed in-memory array lengths (`entries.length`) rather than true database statistics from `stats` and feed/category metadata.
- **Solution**: Updated `DesktopArticleList.tsx` to compute true database counts from `stats` and feed totals, and reverted switch buttons to clean `Unread` and `All` labels.
### 3.5 Mobile iOS Safe Area Insets & Home Indicator
- **Root Cause**: `viewport-fit=cover` was missing in `web/index.html`, causing iOS Safari to evaluate `env(safe-area-inset-bottom)` to `0px`.
- **Solution**: Added `viewport-fit=cover` and calibrated bottom navigation height (`50px + env(safe-area-inset-bottom, 0px)`) and article bottom toolbar height (`48px + env(safe-area-inset-bottom, 0px)`).
---
## 4. Work Items & Backlog for Future Milestones
The following items have been identified for future implementation:
1. **SSE Auto-Reconnect & Connection State Recovery**:
- Exponential backoff with jitter on EventSource disconnection in `web/src/api.ts`.
- Snapshot reconciliation (`/api/stats` and `/api/entries?since=...`) upon reconnect.
2. **Background Job Queue & Dead-Letter Management in Admin Settings**:
- Dedicated "Job Queue" tab in `Settings > Account & System`.
- Live table of tasks (`pending`, `running`, `completed`, `failed`, `dead`) with error logs and manual retry/cancel actions.
3. **Reader UI Prompt Template Selector**:
- Dropdown selector next to "Generate AI Summary" button in Desktop and Mobile reader bars to choose between configured prompt templates on the fly.
4. **Local Ollama Model Autodiscovery & Health Check**:
- "Discover Models" button in `Settings > AI & Integrations` querying `http://localhost:11434/api/tags` to populate the model list automatically.
5. **Desktop 3-Pane Middle List Infinite Scrolling**:
- Windowed offset pagination (`limit: 50`, `offset: entries.length`) with scroll threshold trigger on `.desktop-articles-scroll`.
+49 -3
View File
@@ -2,6 +2,7 @@ package ai
import (
"context"
"encoding/json"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/ai"
@@ -12,7 +13,35 @@ import (
)
// LoadConfig fetches current AI provider configuration from system settings.
func LoadConfig(database *db.DB) ai.Config {
func LoadConfig(database *db.DB) (ai.Config, []models.PromptTemplate) {
raw, err := database.GetSetting("ai_settings")
if err == nil && raw != "" {
var s models.AISettings
if err := json.Unmarshal([]byte(raw), &s); err == nil {
provider := s.Provider
if provider == "" {
provider = "openai"
}
baseURL := s.BaseURL
if baseURL == "" {
baseURL = "https://api.openai.com/v1"
}
model := s.Model
if model == "" {
model = "gpt-4o-mini"
}
return ai.Config{
Enabled: s.Enabled,
Provider: provider,
BaseURL: baseURL,
APIKey: s.APIKey,
Model: model,
Temperature: s.Temperature,
MaxTokens: s.MaxTokens,
}, s.Prompts
}
}
settings, _ := database.GetSettingsMap()
enabled := settings["ai_enabled"] == "true" || settings["ai_enabled"] == "1"
provider := settings["ai_provider"]
@@ -33,13 +62,19 @@ func LoadConfig(database *db.DB) ai.Config {
BaseURL: baseURL,
APIKey: settings["ai_api_key"],
Model: model,
}
}, nil
}
// NewEnrichmentHandler creates a worker action handler for AI variant enrichment.
func NewEnrichmentHandler(database *db.DB, client *ai.Client, log *zap.Logger) worker.ActionHandler {
return func(ctx context.Context, entry models.FeedEntry, params map[string]interface{}) (*worker.ActionResult, error) {
cfg := LoadConfig(database)
cfg, prompts := LoadConfig(database)
if !cfg.Enabled {
log.Warn("AI action skipped: AI features are disabled in Settings",
zap.String("entry_id", entry.ID),
)
return nil, nil
}
if cfg.APIKey == "" && cfg.Provider != "ollama" {
log.Warn("AI action skipped: API key is not configured in Settings > AI & Integrations",
zap.String("entry_id", entry.ID),
@@ -52,6 +87,17 @@ func NewEnrichmentHandler(database *db.DB, client *ai.Client, log *zap.Logger) w
if prompt == "" {
prompt, _ = params["custom_prompt"].(string)
}
if prompt == "" {
for _, p := range prompts {
if p.IsDefault {
prompt = p.Prompt
break
}
}
if prompt == "" && len(prompts) > 0 {
prompt = prompts[0].Prompt
}
}
vKey, _ := params["variant_key"].(string)
if vKey == "" {
vKey = "ai_summary"
+5 -1
View File
@@ -15,7 +15,11 @@ import (
// NewTriageHandler creates a worker action handler for AI autonomous triage decisions.
func NewTriageHandler(database *db.DB, client *ai.Client, pool *worker.Pool, log *zap.Logger) worker.ActionHandler {
return func(ctx context.Context, entry models.FeedEntry, params map[string]interface{}) (*worker.ActionResult, error) {
cfg := LoadConfig(database)
cfg, _ := LoadConfig(database)
if !cfg.Enabled {
log.Warn("AI triage skipped: AI features are disabled in Settings", zap.String("entry_id", entry.ID))
return nil, nil
}
if cfg.APIKey == "" && cfg.Provider != "ollama" {
log.Warn("AI triage action skipped: API key is not configured in Settings > AI & Integrations",
zap.String("entry_id", entry.ID),
+191 -1
View File
@@ -1,6 +1,9 @@
package actions
import (
"context"
"fmt"
"strings"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/actions/ai"
@@ -9,11 +12,12 @@ import (
"git.pengzhan.dev/haopengzhan/rsspipe/internal/actions/variants"
aiClientPkg "git.pengzhan.dev/haopengzhan/rsspipe/internal/ai"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/worker"
"go.uber.org/zap"
)
// RegisterDefaultHandlers registers all core action handlers with the worker pool.
// RegisterDefaultHandlers registers legacy action handlers with worker.Pool
func RegisterDefaultHandlers(pool *worker.Pool, database *db.DB, log *zap.Logger) {
// 1. Readability Full-Text Scraper
scraper := readability.NewScraper(15 * time.Second)
@@ -34,3 +38,189 @@ func RegisterDefaultHandlers(pool *worker.Pool, database *db.DB, log *zap.Logger
pool.RegisterHandler("ai_agent_action", ai.NewTriageHandler(database, aiClient, pool, log))
}
// RegisterJobEngineHandlers registers async task handlers with the persistent JobEngine
func RegisterJobEngineHandlers(engine *worker.JobEngine, database *db.DB, log *zap.Logger) {
scraper := readability.NewScraper(20 * time.Second)
aiClient := aiClientPkg.NewClient(60 * time.Second)
// 1. Readability Scrape Handler
engine.RegisterHandler("readability_scrape", func(ctx context.Context, job models.Job) (*models.PatchEntryRequest, map[string]interface{}, error) {
entryID, _ := job.Payload["entry_id"].(string)
if entryID == "" {
return nil, nil, fmt.Errorf("missing entry_id in job payload")
}
entry, err := database.GetEntry(entryID)
if err != nil {
return nil, nil, fmt.Errorf("entry not found: %w", err)
}
res, err := scraper.Handler()(ctx, *entry, job.Payload)
if err != nil {
return nil, nil, err
}
if res == nil || res.ResultVariant == nil {
return nil, nil, nil
}
patch := &models.PatchEntryRequest{
Variants: map[string]*models.EntryVariant{
"readability_full_text": res.ResultVariant,
},
}
return patch, map[string]interface{}{
"variant_key": "readability_full_text",
"length": len(res.ResultVariant.TextValue),
}, nil
})
// 2. AI Summary & AI Enrich Handler
aiHandler := func(ctx context.Context, job models.Job) (*models.PatchEntryRequest, map[string]interface{}, error) {
entryID, _ := job.Payload["entry_id"].(string)
if entryID == "" {
return nil, nil, fmt.Errorf("missing entry_id in job payload")
}
entry, err := database.GetEntry(entryID)
if err != nil {
return nil, nil, fmt.Errorf("entry not found: %w", err)
}
cfg, prompts := ai.LoadConfig(database)
if !cfg.Enabled {
return nil, nil, fmt.Errorf("AI features are disabled. Please enable AI in Settings > AI & Integrations")
}
if cfg.Provider != "ollama" && strings.TrimSpace(cfg.APIKey) == "" {
return nil, nil, fmt.Errorf("AI API key is missing. Please configure an API key in Settings > AI & Integrations")
}
prompt, _ := job.Payload["prompt"].(string)
if prompt == "" {
prompt, _ = job.Payload["custom_prompt"].(string)
}
if prompt == "" {
for _, p := range prompts {
if p.IsDefault {
prompt = p.Prompt
break
}
}
if prompt == "" && len(prompts) > 0 {
prompt = prompts[0].Prompt
}
}
if prompt == "" {
prompt = "You are an expert technical editor. Summarize the following article in exactly 3 concise bullet points, followed by a bold 'Key Takeaway:' sentence.\n\nTitle: {{title}}\nFeed: {{feed_title}}\nContent:\n{{content}}"
}
vKey, _ := job.Payload["variant_key"].(string)
if vKey == "" {
vKey = "ai_summary"
}
cType, _ := job.Payload["content_type"].(string)
if cType == "" {
cType = "text"
}
resultText, err := aiClient.GenerateEnrichment(ctx, cfg, prompt, *entry)
if err != nil {
return nil, nil, fmt.Errorf("ai generation failed: %w", err)
}
variant := models.EntryVariant{
EntryID: entry.ID,
VariantKey: vKey,
ContentType: cType,
TextValue: resultText,
Metadata: map[string]interface{}{
"model": cfg.Model,
"provider": cfg.Provider,
"prompt_used": prompt,
},
CreatedAt: time.Now().UTC(),
UpdatedAt: time.Now().UTC(),
}
patch := &models.PatchEntryRequest{
Variants: map[string]*models.EntryVariant{
vKey: &variant,
},
}
return patch, map[string]interface{}{
"variant_key": vKey,
"model": cfg.Model,
"word_count": len(strings.Fields(resultText)),
}, nil
}
engine.RegisterHandler("ai_summary", aiHandler)
engine.RegisterHandler("ai_enrich", aiHandler)
// 3. Add Tags Handler
engine.RegisterHandler("add_tags", func(ctx context.Context, job models.Job) (*models.PatchEntryRequest, map[string]interface{}, error) {
entryID, _ := job.Payload["entry_id"].(string)
if entryID == "" {
return nil, nil, fmt.Errorf("missing entry_id in job payload")
}
var tagsToAdd []string
if rawTags, ok := job.Payload["tags"].([]interface{}); ok {
for _, t := range rawTags {
tagsToAdd = append(tagsToAdd, fmt.Sprintf("%v", t))
}
} else if rawTagsStr, ok := job.Payload["tags"].(string); ok {
for _, t := range strings.Split(rawTagsStr, ",") {
if tr := strings.TrimSpace(t); tr != "" {
tagsToAdd = append(tagsToAdd, tr)
}
}
}
entry, err := database.GetEntry(entryID)
if err != nil {
return nil, nil, fmt.Errorf("entry not found: %w", err)
}
existingTags := make(map[string]bool)
if entry.Metadata != nil {
if curr, ok := entry.Metadata["tags"].([]interface{}); ok {
for _, t := range curr {
existingTags[fmt.Sprintf("%v", t)] = true
}
}
}
for _, t := range tagsToAdd {
existingTags[t] = true
}
var merged []string
for t := range existingTags {
merged = append(merged, t)
}
patch := &models.PatchEntryRequest{
Metadata: map[string]interface{}{
"tags": merged,
},
}
return patch, map[string]interface{}{"tags": merged}, nil
})
// 4. Mark Read Handler
engine.RegisterHandler("mark_read", func(ctx context.Context, job models.Job) (*models.PatchEntryRequest, map[string]interface{}, error) {
read := true
return &models.PatchEntryRequest{IsRead: &read}, map[string]interface{}{"is_read": true}, nil
})
// 5. Mark Starred Handler
engine.RegisterHandler("mark_starred", func(ctx context.Context, job models.Job) (*models.PatchEntryRequest, map[string]interface{}, error) {
starred := true
return &models.PatchEntryRequest{IsStarred: &starred}, map[string]interface{}{"is_starred": true}, nil
})
}
+146
View File
@@ -0,0 +1,146 @@
package api
import (
"encoding/json"
"net/http"
"strings"
"sync"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/auth"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
)
// RateLimiter tracks login attempts per IP to prevent brute-force attacks
type RateLimiter struct {
mu sync.Mutex
attempts map[string][]time.Time
limit int
window time.Duration
}
func NewRateLimiter(limit int, window time.Duration) *RateLimiter {
return &RateLimiter{
attempts: make(map[string][]time.Time),
limit: limit,
window: window,
}
}
func (rl *RateLimiter) Allow(key string) bool {
rl.mu.Lock()
defer rl.mu.Unlock()
now := time.Now()
cutoff := now.Add(-rl.window)
// Filter out expired timestamps
valid := make([]time.Time, 0)
for _, t := range rl.attempts[key] {
if t.After(cutoff) {
valid = append(valid, t)
}
}
if len(valid) >= rl.limit {
rl.attempts[key] = valid
return false
}
valid = append(valid, now)
rl.attempts[key] = valid
return true
}
func getClientIP(r *http.Request) string {
if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
parts := strings.Split(xff, ",")
return strings.TrimSpace(parts[0])
}
if xrip := r.Header.Get("X-Real-IP"); xrip != "" {
return strings.TrimSpace(xrip)
}
return r.RemoteAddr
}
// Auth Handlers
func (h *Handler) Login(w http.ResponseWriter, r *http.Request) {
ip := getClientIP(r)
if !h.loginLimiter.Allow(ip) {
h.writeError(w, http.StatusTooManyRequests, "too many login attempts, please try again in a minute")
return
}
var req models.LoginRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid request body")
return
}
token, err := h.auth.Authenticate(req.Username, req.Password)
if err != nil {
h.writeError(w, http.StatusUnauthorized, "invalid username or password")
return
}
http.SetCookie(w, &http.Cookie{
Name: "rsspipe_token",
Value: token,
Path: "/",
Expires: time.Now().Add(30 * 24 * time.Hour),
HttpOnly: true,
SameSite: http.SameSiteLaxMode,
})
user, _ := h.db.GetUserByUsername(req.Username)
h.writeJSON(w, http.StatusOK, models.LoginResponse{
Token: token,
User: *user,
})
}
func (h *Handler) Logout(w http.ResponseWriter, r *http.Request) {
http.SetCookie(w, &http.Cookie{
Name: "rsspipe_token",
Value: "",
Path: "/",
Expires: time.Unix(0, 0),
HttpOnly: true,
})
h.writeJSON(w, http.StatusOK, map[string]string{"message": "logged out"})
}
func (h *Handler) GetMe(w http.ResponseWriter, r *http.Request) {
claims, ok := r.Context().Value(auth.UserContextKey).(*auth.Claims)
if !ok {
h.writeError(w, http.StatusUnauthorized, "unauthorized")
return
}
user, err := h.db.GetUserByUsername(claims.Username)
if err != nil {
h.writeError(w, http.StatusNotFound, "user not found")
return
}
h.writeJSON(w, http.StatusOK, user)
}
func (h *Handler) UpdatePassword(w http.ResponseWriter, r *http.Request) {
claims, ok := r.Context().Value(auth.UserContextKey).(*auth.Claims)
if !ok {
h.writeError(w, http.StatusUnauthorized, "unauthorized")
return
}
var req models.ChangePasswordRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid request payload")
return
}
if err := h.auth.UpdatePassword(claims.UserID, req.OldPassword, req.NewPassword); err != nil {
h.writeError(w, http.StatusBadRequest, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "password updated successfully"})
}
+360
View File
@@ -0,0 +1,360 @@
package api
import (
"database/sql"
"encoding/json"
"fmt"
"net/http"
"strconv"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/services"
"github.com/go-chi/chi/v5"
)
// Entry Handlers
func (h *Handler) ListEntries(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
limit, _ := strconv.Atoi(q.Get("limit"))
offset, _ := strconv.Atoi(q.Get("offset"))
filter := db.EntryFilter{
View: q.Get("view"),
FeedID: q.Get("feed_id"),
CategoryID: q.Get("category_id"),
Search: q.Get("search"),
Limit: limit,
Offset: offset,
}
entries, total, err := h.entryService.ListEntries(r.Context(), filter)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if entries == nil {
entries = []models.FeedEntry{}
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"entries": entries,
"total": total,
"limit": limit,
"offset": offset,
})
}
func (h *Handler) GetEntry(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
entry, err := h.entryService.GetEntry(r.Context(), id)
if err != nil {
if err == sql.ErrNoRows {
h.writeError(w, http.StatusNotFound, "entry not found")
return
}
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, entry)
}
// PatchEntry handles atomic partial updates to an entry and returns the updated entry + transition diff
func (h *Handler) PatchEntry(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
var req models.PatchEntryRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid request body")
return
}
updated, diff, err := h.entryService.PatchEntry(r.Context(), id, req)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"entry": updated,
"diff": diff,
})
}
func (h *Handler) UpdateEntryRead(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
var body models.UpdateEntryReadRequest
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid body")
return
}
updated, _, err := h.entryService.PatchEntry(r.Context(), id, models.PatchEntryRequest{
IsRead: &body.IsRead,
})
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{"id": id, "is_read": updated.IsRead})
}
func (h *Handler) UpdateEntryStar(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
var body models.UpdateEntryStarRequest
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid body")
return
}
updated, _, err := h.entryService.PatchEntry(r.Context(), id, models.PatchEntryRequest{
IsStarred: &body.IsStarred,
})
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{"id": id, "is_starred": updated.IsStarred})
}
func (h *Handler) MarkAllRead(w http.ResponseWriter, r *http.Request) {
var body models.MarkAllReadRequest
_ = json.NewDecoder(r.Body).Decode(&body)
if err := h.entryService.MarkAllRead(r.Context(), body.FeedID, body.CategoryID); err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "marked all as read"})
}
// ScrapeEntry triggers readability full text extraction (immediate response for UI, async if ?async=true)
func (h *Handler) ScrapeEntry(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
entry, err := h.entryService.GetEntry(r.Context(), id)
if err != nil {
h.writeError(w, http.StatusNotFound, "entry not found")
return
}
idempotencyKey := services.GenerateIdempotencyKey("readability_scrape", id, "manual", entry.CreatedAt.Format("20060102150405"))
job, err := h.jobService.Enqueue(r.Context(), "readability_scrape", map[string]interface{}{
"entry_id": id,
"url": entry.URL,
}, idempotencyKey)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if r.URL.Query().Get("async") == "true" {
h.writeJSON(w, http.StatusAccepted, map[string]interface{}{
"job_id": job.ID,
"status": job.Status,
"message": "Scrape task queued asynchronously",
})
return
}
if h.jobEngine == nil {
h.writeError(w, http.StatusInternalServerError, "job engine not initialized")
return
}
patch, result, err := h.jobEngine.ExecuteDirect(r.Context(), "readability_scrape", job.Payload)
if err != nil {
h.writeError(w, http.StatusInternalServerError, fmt.Sprintf("scraping failed: %v", err))
return
}
if patch != nil {
updated, _, patchErr := h.entryService.PatchEntry(r.Context(), id, *patch)
if patchErr != nil {
h.writeError(w, http.StatusInternalServerError, patchErr.Error())
return
}
_ = h.db.UpdateJob(models.Job{
ID: job.ID,
Status: "completed",
Progress: 100,
Result: result,
})
h.writeJSON(w, http.StatusOK, updated)
return
}
updated, _ := h.entryService.GetEntry(r.Context(), id)
h.writeJSON(w, http.StatusOK, updated)
}
// GenerateAISummary generates an AI summary (immediate response for UI, async if ?async=true)
func (h *Handler) GenerateAISummary(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
entry, err := h.entryService.GetEntry(r.Context(), id)
if err != nil {
h.writeError(w, http.StatusNotFound, "entry not found")
return
}
var body struct {
Prompt string `json:"prompt,omitempty"`
}
_ = json.NewDecoder(r.Body).Decode(&body)
idempotencyKey := services.GenerateIdempotencyKey("ai_summary", id, "manual", entry.CreatedAt.Format("20060102150405"))
job, err := h.jobService.Enqueue(r.Context(), "ai_summary", map[string]interface{}{
"entry_id": id,
"prompt": body.Prompt,
}, idempotencyKey)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if r.URL.Query().Get("async") == "true" {
h.writeJSON(w, http.StatusAccepted, map[string]interface{}{
"job_id": job.ID,
"status": job.Status,
"message": "AI summary task queued asynchronously",
})
return
}
if h.jobEngine == nil {
h.writeError(w, http.StatusInternalServerError, "job engine not initialized")
return
}
patch, result, err := h.jobEngine.ExecuteDirect(r.Context(), "ai_summary", job.Payload)
if err != nil {
h.writeError(w, http.StatusInternalServerError, fmt.Sprintf("AI summary generation failed: %v", err))
return
}
if patch != nil {
updated, _, patchErr := h.entryService.PatchEntry(r.Context(), id, *patch)
if patchErr != nil {
h.writeError(w, http.StatusInternalServerError, patchErr.Error())
return
}
_ = h.db.UpdateJob(models.Job{
ID: job.ID,
Status: "completed",
Progress: 100,
Result: result,
})
h.writeJSON(w, http.StatusOK, updated)
return
}
updated, _ := h.entryService.GetEntry(r.Context(), id)
h.writeJSON(w, http.StatusOK, updated)
}
func (h *Handler) ToggleShareEntry(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
shareUUID, err := h.entryService.ToggleShare(r.Context(), id)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
entry, _ := h.entryService.GetEntry(r.Context(), id)
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"shared": shareUUID != nil,
"share_uuid": shareUUID,
"entry": entry,
})
}
func (h *Handler) ListEntryVariants(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
variants, err := h.db.ListVariantsForEntry(id)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, variants)
}
func (h *Handler) GetVariantBlobStream(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
key := chi.URLParam(r, "key")
variant, err := h.db.GetVariant(id, key)
if err != nil {
h.writeError(w, http.StatusNotFound, "variant not found")
return
}
if variant.BlobPath == "" {
h.writeError(w, http.StatusNotFound, "variant has no binary media blob")
return
}
http.ServeFile(w, r, variant.BlobPath)
}
// ViewSharedEntry renders a styled standalone HTML page for an explicitly shared entry (strictly checked)
func (h *Handler) ViewSharedEntry(w http.ResponseWriter, r *http.Request) {
uuidStr := chi.URLParam(r, "uuid")
entry, err := h.entryService.GetSharedEntry(r.Context(), uuidStr)
if err != nil {
http.Error(w, "Shared article not found", http.StatusNotFound)
return
}
// Render standalone styled HTML
html := fmt.Sprintf(`<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>%s - RSSPipe</title>
<style>
:root { --bg: #0f172a; --card: #1e293b; --text: #f8fafc; --muted: #94a3b8; --accent: #38bdf8; }
@media (prefers-color-scheme: light) {
:root { --bg: #f8fafc; --card: #ffffff; --text: #0f172a; --muted: #64748b; --accent: #0284c7; }
}
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; background: var(--bg); color: var(--text); line-height: 1.7; margin: 0; padding: 2rem 1rem; display: flex; justify-content: center; }
.container { max-width: 760px; width: 100%%; background: var(--card); border-radius: 12px; padding: 2.5rem; box-shadow: 0 10px 25px rgba(0,0,0,0.1); }
.meta { color: var(--muted); font-size: 0.9rem; margin-bottom: 1.5rem; display: flex; flex-wrap: wrap; gap: 1rem; align-items: center; border-bottom: 1px solid rgba(148, 163, 184, 0.2); padding-bottom: 1rem; }
h1 { margin-top: 0; font-size: 2rem; line-height: 1.3; }
h1 a { color: inherit; text-decoration: none; }
h1 a:hover { color: var(--accent); }
.content img { max-width: 100%%; height: auto; border-radius: 8px; }
.content pre { background: rgba(0,0,0,0.2); padding: 1rem; border-radius: 8px; overflow-x: auto; }
.footer { margin-top: 2.5rem; padding-top: 1.5rem; border-top: 1px solid rgba(148, 163, 184, 0.2); font-size: 0.85rem; color: var(--muted); text-align: center; }
</style>
</head>
<body>
<div class="container">
<div class="meta">
<span><strong>%s</strong></span>
<span>•</span>
<span>%s</span>
%s
</div>
<h1><a href="%s" target="_blank" rel="noopener noreferrer">%s ↗</a></h1>
<div class="content">
%s
</div>
<div class="footer">
Shared via <a href="/" style="color: var(--accent); text-decoration: none;">RSSPipe Information Gateway</a>
</div>
</div>
</body>
</html>`,
entry.Title,
entry.FeedTitle,
entry.PublishedAt.Format("Jan 02, 2006 15:04"),
func() string {
if entry.Author != "" {
return "<span>•</span><span>By " + entry.Author + "</span>"
}
return ""
}(),
entry.URL,
entry.Title,
entry.Content,
)
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write([]byte(html))
}
+176
View File
@@ -0,0 +1,176 @@
package api
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"github.com/go-chi/chi/v5"
)
// Category Handlers
func (h *Handler) ListCategories(w http.ResponseWriter, r *http.Request) {
cats, err := h.feedService.ListCategories(r.Context())
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if cats == nil {
cats = []models.Category{}
}
h.writeJSON(w, http.StatusOK, cats)
}
func (h *Handler) CreateCategory(w http.ResponseWriter, r *http.Request) {
var body struct {
Title string `json:"title"`
ParentID *string `json:"parent_id,omitempty"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || strings.TrimSpace(body.Title) == "" {
h.writeError(w, http.StatusBadRequest, "valid category title is required")
return
}
cat, err := h.feedService.CreateCategory(r.Context(), strings.TrimSpace(body.Title), body.ParentID)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusCreated, cat)
}
func (h *Handler) DeleteCategory(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if err := h.feedService.DeleteCategory(r.Context(), id); err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "category deleted"})
}
// Feed Handlers
func (h *Handler) ListFeeds(w http.ResponseWriter, r *http.Request) {
feeds, err := h.feedService.ListFeeds(r.Context())
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if feeds == nil {
feeds = []models.Feed{}
}
h.writeJSON(w, http.StatusOK, feeds)
}
func (h *Handler) CreateFeed(w http.ResponseWriter, r *http.Request) {
var req models.CreateFeedRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil || strings.TrimSpace(req.FeedURL) == "" {
h.writeError(w, http.StatusBadRequest, "valid feed_url is required")
return
}
feed, err := h.feedService.CreateFeed(r.Context(), req)
if err != nil {
h.writeError(w, http.StatusBadRequest, err.Error())
return
}
h.writeJSON(w, http.StatusCreated, feed)
}
func (h *Handler) UpdateFeed(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
var req models.UpdateFeedRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid request body")
return
}
feed, err := h.feedService.UpdateFeed(r.Context(), id, req)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, feed)
}
func (h *Handler) DeleteFeed(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if err := h.feedService.DeleteFeed(r.Context(), id); err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "feed deleted"})
}
func (h *Handler) RefreshFeed(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
f, err := h.feedService.RefreshFeed(r.Context(), id)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, f)
}
func (h *Handler) RefreshAllFeeds(w http.ResponseWriter, r *http.Request) {
h.feedService.RefreshAllFeeds(r.Context())
h.writeJSON(w, http.StatusAccepted, map[string]string{"message": "refreshing all feeds in background"})
}
func (h *Handler) GetStats(w http.ResponseWriter, r *http.Request) {
stats, err := h.feedService.GetStats(r.Context())
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, stats)
}
// OPML Handlers
func (h *Handler) ImportOPML(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(10 << 20); err != nil {
h.writeError(w, http.StatusBadRequest, "failed to parse multipart form")
return
}
file, _, err := r.FormFile("file")
if err != nil {
h.writeError(w, http.StatusBadRequest, "file field is required")
return
}
defer func() {
_ = file.Close()
}()
data, err := io.ReadAll(file)
if err != nil {
h.writeError(w, http.StatusBadRequest, "failed to read file")
return
}
createdCount, totalParsed, err := h.feedService.ImportOPML(r.Context(), data)
if err != nil {
h.writeError(w, http.StatusBadRequest, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"imported_count": createdCount,
"total_parsed": totalParsed,
})
}
func (h *Handler) ExportOPML(w http.ResponseWriter, r *http.Request) {
xmlData, err := h.feedService.ExportOPML(r.Context())
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.Header().Set("Content-Disposition", fmt.Sprintf(`attachment; filename="rsspipe-subscriptions.opml"`))
w.WriteHeader(http.StatusOK)
_, _ = w.Write(xmlData)
}
+70 -1104
View File
File diff suppressed because it is too large Load Diff
+70
View File
@@ -0,0 +1,70 @@
package api
import (
"database/sql"
"net/http"
"strconv"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"github.com/go-chi/chi/v5"
)
func (h *Handler) ListJobs(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
limit, _ := strconv.Atoi(q.Get("limit"))
offset, _ := strconv.Atoi(q.Get("offset"))
filter := models.JobFilter{
Type: q.Get("type"),
Status: q.Get("status"),
Limit: limit,
Offset: offset,
}
jobs, err := h.jobService.ListJobs(r.Context(), filter)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if jobs == nil {
jobs = []models.Job{}
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"jobs": jobs,
"limit": limit,
"offset": offset,
})
}
func (h *Handler) GetJob(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
job, err := h.jobService.GetJob(r.Context(), id)
if err != nil {
if err == sql.ErrNoRows {
h.writeError(w, http.StatusNotFound, "job not found")
return
}
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, job)
}
func (h *Handler) CancelJob(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if err := h.jobService.CancelJob(r.Context(), id); err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "job cancelled"})
}
func (h *Handler) RetryJob(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if err := h.jobService.RetryJob(r.Context(), id); err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "job queued for retry"})
}
+131
View File
@@ -0,0 +1,131 @@
package api
import (
"encoding/json"
"net/http"
"strings"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/rules"
"github.com/go-chi/chi/v5"
)
// Rules API Handlers
func (h *Handler) ListRules(w http.ResponseWriter, r *http.Request) {
rulesList, err := h.db.ListRules()
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if rulesList == nil {
rulesList = []models.Rule{}
}
h.writeJSON(w, http.StatusOK, rulesList)
}
func (h *Handler) CreateRule(w http.ResponseWriter, r *http.Request) {
var body models.Rule
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid rule payload")
return
}
body.Name = strings.TrimSpace(body.Name)
if body.Name == "" {
h.writeError(w, http.StatusBadRequest, "rule name is required")
return
}
if body.Trigger == "" {
body.Trigger = "on_update"
}
if body.MatchMode == "" {
body.MatchMode = "all"
}
if err := rules.ValidateRuleSafety(body); err != nil {
h.writeError(w, http.StatusBadRequest, err.Error())
return
}
rule, err := h.db.CreateRule(body.Name, body.Trigger, body.MatchMode, body.Conditions, body.Actions)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusCreated, rule)
}
func (h *Handler) UpdateRule(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
var body models.Rule
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid rule payload")
return
}
body.Name = strings.TrimSpace(body.Name)
if body.Name == "" {
h.writeError(w, http.StatusBadRequest, "rule name is required")
return
}
if err := rules.ValidateRuleSafety(body); err != nil {
h.writeError(w, http.StatusBadRequest, err.Error())
return
}
rule, err := h.db.UpdateRule(id, body.Name, body.Trigger, body.MatchMode, body.Conditions, body.Actions, body.IsActive)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, rule)
}
func (h *Handler) DeleteRule(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if err := h.db.DeleteRule(id); err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "rule deleted"})
}
func (h *Handler) ApplyRule(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
rule, err := h.db.GetRule(id)
if err != nil {
h.writeError(w, http.StatusNotFound, "rule not found")
return
}
entries, _, err := h.db.ListEntries(db.EntryFilter{Limit: 200})
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
eval := rules.NewEvaluator()
matchedCount := 0
for _, entry := range entries {
if eval.Evaluate(*rule, entry) {
matchedCount++
if h.bus != nil {
h.bus.Publish(r.Context(), events.Event{
Type: events.EventManualTrigger,
Entry: entry,
Depth: 0,
})
}
}
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "rule applied to matching articles",
"matched_count": matchedCount,
"queued_jobs": matchedCount * len(rule.Actions),
})
}
+15 -4
View File
@@ -20,10 +20,10 @@ func NewRouter(h *Handler, authService *auth.AuthService, staticDir string) http
r.Use(middleware.Recoverer)
r.Use(middleware.Compress(5))
// CORS configuration for local development & PWA
// CORS configuration: Restricted to localhost dev ports and same-origin
r.Use(cors.Handler(cors.Options{
AllowedOrigins: []string{"http://localhost:5173", "http://localhost:3000", "http://localhost:8080", "*"},
AllowedMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTIONS"},
AllowedOrigins: []string{"http://localhost:5173", "http://localhost:3000", "http://localhost:8080"},
AllowedMethods: []string{"GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"},
AllowedHeaders: []string{"Accept", "Authorization", "Content-Type", "X-CSRF-Token"},
ExposedHeaders: []string{"Link"},
AllowCredentials: true,
@@ -43,6 +43,10 @@ func NewRouter(h *Handler, authService *auth.AuthService, staticDir string) http
r.Group(func(r chi.Router) {
r.Use(authService.Middleware)
// Real-time Event Stream (K8s Watch pattern via SSE)
r.Get("/events/stream", h.StreamEvents)
// Auth & Profile
r.Get("/auth/me", h.GetMe)
r.Put("/auth/password", h.UpdatePassword)
r.Get("/user/preferences", h.GetUserPreferences)
@@ -65,15 +69,22 @@ func NewRouter(h *Handler, authService *auth.AuthService, staticDir string) http
// Entries
r.Get("/entries", h.ListEntries)
r.Get("/entries/{id}", h.GetEntry)
r.Patch("/entries/{id}", h.PatchEntry)
r.Put("/entries/{id}/read", h.UpdateEntryRead)
r.Put("/entries/{id}/star", h.UpdateEntryStar)
r.Post("/entries/{id}/share", h.ToggleShareEntry)
r.Post("/entries/{id}/ai-summary", h.GenerateAISummary)
r.Post("/entries/mark-all-read", h.MarkAllRead)
r.Post("/entries/{id}/scrape", h.ScrapeEntry)
r.Post("/entries/mark-all-read", h.MarkAllRead)
r.Get("/entries/{id}/variants", h.ListEntryVariants)
r.Get("/entries/{id}/variants/{key}/stream", h.GetVariantBlobStream)
// Task Queue / Background Jobs
r.Get("/jobs", h.ListJobs)
r.Get("/jobs/{id}", h.GetJob)
r.Post("/jobs/{id}/cancel", h.CancelJob)
r.Post("/jobs/{id}/retry", h.RetryJob)
// Rules
r.Get("/rules", h.ListRules)
r.Post("/rules", h.CreateRule)
+159
View File
@@ -0,0 +1,159 @@
package api
import (
"encoding/json"
"net/http"
"strconv"
"strings"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/ai"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/auth"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
)
// User Preferences Handlers
func (h *Handler) GetUserPreferences(w http.ResponseWriter, r *http.Request) {
claims, ok := r.Context().Value(auth.UserContextKey).(*auth.Claims)
if !ok {
h.writeError(w, http.StatusUnauthorized, "unauthorized")
return
}
prefs, err := h.settingsService.GetPreferences(r.Context(), claims.Username)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, prefs)
}
func (h *Handler) UpdateUserPreferences(w http.ResponseWriter, r *http.Request) {
claims, ok := r.Context().Value(auth.UserContextKey).(*auth.Claims)
if !ok {
h.writeError(w, http.StatusUnauthorized, "unauthorized")
return
}
var newPrefs map[string]interface{}
if err := json.NewDecoder(r.Body).Decode(&newPrefs); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid request payload")
return
}
updated, err := h.settingsService.UpdatePreferences(r.Context(), claims.Username, newPrefs)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, updated)
}
// AI Settings Handlers
func (h *Handler) GetAISettings(w http.ResponseWriter, r *http.Request) {
cfg, err := h.settingsService.GetAISettings(r.Context())
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, cfg)
}
func (h *Handler) UpdateAISettings(w http.ResponseWriter, r *http.Request) {
var body models.AISettings
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
h.writeError(w, http.StatusBadRequest, "invalid body")
return
}
updated, err := h.settingsService.UpdateAISettings(r.Context(), body)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, updated)
}
func (h *Handler) TestAIConnection(w http.ResponseWriter, r *http.Request) {
var body models.AISettings
_ = json.NewDecoder(r.Body).Decode(&body)
cfg := ai.Config{
Enabled: body.Enabled,
Provider: body.Provider,
BaseURL: body.BaseURL,
APIKey: body.APIKey,
Model: body.Model,
}
if cfg.Provider == "" {
cfg.Provider = "openai"
}
if cfg.BaseURL == "" {
cfg.BaseURL = "https://api.openai.com/v1"
}
start := time.Now()
testEntry := models.FeedEntry{
Title: "RSSPipe Connection Test",
Content: "This is a quick test to verify connection to your AI provider.",
}
res, err := h.aiClient.GenerateEnrichment(r.Context(), cfg, "Respond with 'Connection successful' only.", testEntry)
latency := time.Since(start).Milliseconds()
if err != nil {
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"success": false,
"message": err.Error(),
"latency_ms": latency,
"model": cfg.Model,
})
return
}
h.writeJSON(w, http.StatusOK, map[string]interface{}{
"success": true,
"message": strings.TrimSpace(res),
"latency_ms": latency,
"model": cfg.Model,
})
}
// Search History Handlers
func (h *Handler) GetSearchHistory(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query().Get("q")
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
items, err := h.db.GetSearchHistory(q, limit)
if err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
if items == nil {
items = []models.SearchHistoryItem{}
}
h.writeJSON(w, http.StatusOK, items)
}
func (h *Handler) RecordSearch(w http.ResponseWriter, r *http.Request) {
var body struct {
Keyword string `json:"keyword"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || strings.TrimSpace(body.Keyword) == "" {
h.writeError(w, http.StatusBadRequest, "keyword is required")
return
}
if err := h.db.RecordSearch(body.Keyword); err != nil {
h.writeError(w, http.StatusInternalServerError, err.Error())
return
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "recorded"})
}
func (h *Handler) DeleteSearchHistory(w http.ResponseWriter, r *http.Request) {
kw := r.URL.Query().Get("keyword")
if kw != "" {
_ = h.db.DeleteSearchHistory(kw)
} else {
_ = h.db.ClearSearchHistory()
}
h.writeJSON(w, http.StatusOK, map[string]string{"message": "deleted"})
}
+104
View File
@@ -0,0 +1,104 @@
package api
import (
"encoding/json"
"fmt"
"net/http"
"sync"
"time"
)
type SSEBroadcaster struct {
mu sync.RWMutex
clients map[chan []byte]struct{}
}
func NewSSEBroadcaster() *SSEBroadcaster {
return &SSEBroadcaster{
clients: make(map[chan []byte]struct{}),
}
}
func (b *SSEBroadcaster) Subscribe() chan []byte {
b.mu.Lock()
defer b.mu.Unlock()
ch := make(chan []byte, 32)
b.clients[ch] = struct{}{}
return ch
}
func (b *SSEBroadcaster) Unsubscribe(ch chan []byte) {
b.mu.Lock()
defer b.mu.Unlock()
if _, ok := b.clients[ch]; ok {
delete(b.clients, ch)
close(ch)
}
}
func (b *SSEBroadcaster) Broadcast(event string, data interface{}) {
payload, err := json.Marshal(data)
if err != nil {
return
}
msg := fmt.Sprintf("event: %s\ndata: %s\n\n", event, string(payload))
msgBytes := []byte(msg)
b.mu.RLock()
defer b.mu.RUnlock()
for ch := range b.clients {
select {
case ch <- msgBytes:
default:
// Non-blocking drop if client buffer is full
}
}
}
// StreamEvents handles GET /api/events/stream
func (h *Handler) StreamEvents(w http.ResponseWriter, r *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "Streaming unsupported", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.Header().Set("X-Accel-Buffering", "no")
w.WriteHeader(http.StatusOK)
// Send initial connection packet
initialMsg := fmt.Sprintf("event: connected\ndata: {\"status\":\"connected\",\"time\":\"%s\"}\n\n", time.Now().UTC().Format(time.RFC3339))
_, _ = w.Write([]byte(initialMsg))
flusher.Flush()
clientCh := h.broadcaster.Subscribe()
defer h.broadcaster.Unsubscribe(clientCh)
heartbeat := time.NewTicker(25 * time.Second)
defer heartbeat.Stop()
for {
select {
case <-r.Context().Done():
return
case <-heartbeat.C:
pingMsg := fmt.Sprintf("event: ping\ndata: {\"time\":\"%s\"}\n\n", time.Now().UTC().Format(time.RFC3339))
if _, err := w.Write([]byte(pingMsg)); err != nil {
return
}
flusher.Flush()
case msg, ok := <-clientCh:
if !ok {
return
}
if _, err := w.Write(msg); err != nil {
return
}
flusher.Flush()
}
}
}
+13 -1
View File
@@ -202,14 +202,26 @@ func (c *Crawler) fetchSingleFeed(ctx context.Context, f models.Feed) error {
zap.Int("new_entries", len(newEntries)),
)
// Publish on_post_created events for newly inserted entries
// Publish on_post_created and on_entry_transition events for newly inserted entries
if c.bus != nil {
for _, entry := range newEntries {
entry.FeedTitle = f.Title
entry.FeedSiteURL = f.SiteURL
diff := &models.EntryDiff{
FieldsChanged: []string{"id", "title", "content", "url"},
}
c.bus.Publish(ctx, events.Event{
Type: events.EventPostCreated,
Entry: entry,
NewEntry: &entry,
Diff: diff,
Depth: 0,
})
c.bus.Publish(ctx, events.Event{
Type: events.EventEntryTransition,
Entry: entry,
NewEntry: &entry,
Diff: diff,
Depth: 0,
})
}
+338 -13
View File
@@ -25,7 +25,8 @@ func InitDB(dbPath string) (*DB, error) {
return nil, fmt.Errorf("failed to create db directory: %w", err)
}
db, err := sql.Open("sqlite", dbPath)
dsn := fmt.Sprintf("%s?_pragma=busy_timeout(10000)&_pragma=journal_mode(WAL)&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(ON)&_pragma=cache_size(-20000)", dbPath)
db, err := sql.Open("sqlite", dsn)
if err != nil {
return nil, fmt.Errorf("failed to open sqlite db: %w", err)
}
@@ -34,18 +35,6 @@ func InitDB(dbPath string) (*DB, error) {
db.SetMaxIdleConns(5)
db.SetConnMaxLifetime(time.Hour)
pragmas := []string{
"PRAGMA journal_mode = WAL;",
"PRAGMA busy_timeout = 5000;",
"PRAGMA synchronous = NORMAL;",
"PRAGMA foreign_keys = ON;",
}
for _, p := range pragmas {
if _, err := db.Exec(p); err != nil {
return nil, fmt.Errorf("failed to set pragma %s: %w", p, err)
}
}
database := &DB{DB: db}
if err := database.migrate(); err != nil {
return nil, fmt.Errorf("failed to run migrations: %w", err)
@@ -153,6 +142,24 @@ func (d *DB) migrate() error {
search_count INTEGER NOT NULL DEFAULT 1
);
CREATE INDEX IF NOT EXISTS idx_search_history_time ON search_history(last_searched_at DESC);
CREATE TABLE IF NOT EXISTS jobs (
id TEXT PRIMARY KEY,
type TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL,
progress INTEGER DEFAULT 0,
result TEXT DEFAULT '{}',
error_message TEXT DEFAULT '',
attempts INTEGER DEFAULT 0,
max_attempts INTEGER DEFAULT 3,
idempotency_key TEXT UNIQUE,
run_at DATETIME NOT NULL,
created_at DATETIME NOT NULL,
updated_at DATETIME NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_jobs_status_run_at ON jobs(status, run_at);
CREATE INDEX IF NOT EXISTS idx_jobs_idempotency ON jobs(idempotency_key);
`
if _, err := d.Exec(schema); err != nil {
return err
@@ -848,6 +855,15 @@ func (d *DB) UpdateEntryStarred(id string, isStarred bool) error {
return err
}
func (d *DB) UpdateEntryMetadata(id string, metadata map[string]interface{}) error {
metaBytes, err := json.Marshal(metadata)
if err != nil {
return err
}
_, err = d.Exec("UPDATE feed_entries SET metadata = ? WHERE id = ?", string(metaBytes), id)
return err
}
func (d *DB) ToggleShareEntry(id string) (*string, error) {
entry, err := d.GetEntry(id)
if err != nil {
@@ -1291,3 +1307,312 @@ func (d *DB) ClearSearchHistory() error {
_, err := d.Exec("DELETE FROM search_history")
return err
}
// Job Queue Operations
func (d *DB) CreateJob(job models.Job) (*models.Job, error) {
if job.ID == "" {
job.ID = uuid.New().String()
}
now := time.Now().UTC()
if job.CreatedAt.IsZero() {
job.CreatedAt = now
}
job.UpdatedAt = now
if job.RunAt.IsZero() {
job.RunAt = now
}
if job.Status == "" {
job.Status = "pending"
}
if job.MaxAttempts <= 0 {
job.MaxAttempts = 3
}
payloadBytes, err := json.Marshal(job.Payload)
if err != nil {
return nil, fmt.Errorf("failed to marshal job payload: %w", err)
}
resultBytes, _ := json.Marshal(job.Result)
if len(resultBytes) == 0 {
resultBytes = []byte("{}")
}
var idemKey sql.NullString
if job.IdempotencyKey != "" {
idemKey = sql.NullString{String: job.IdempotencyKey, Valid: true}
}
query := `
INSERT INTO jobs (
id, type, payload, status, progress, result, error_message, attempts, max_attempts, idempotency_key, run_at, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`
_, err = d.Exec(
query,
job.ID, job.Type, string(payloadBytes), job.Status, job.Progress,
string(resultBytes), job.ErrorMessage, job.Attempts, job.MaxAttempts,
idemKey, job.RunAt, job.CreatedAt, job.UpdatedAt,
)
if err != nil {
return nil, err
}
return &job, nil
}
func (d *DB) GetJob(id string) (*models.Job, error) {
query := `
SELECT id, type, payload, status, progress, result, error_message, attempts, max_attempts, idempotency_key, run_at, created_at, updated_at
FROM jobs
WHERE id = ?
`
var j models.Job
var payloadStr, resultStr, errorMsg string
var idemKey sql.NullString
err := d.QueryRow(query, id).Scan(
&j.ID, &j.Type, &payloadStr, &j.Status, &j.Progress, &resultStr,
&errorMsg, &j.Attempts, &j.MaxAttempts, &idemKey, &j.RunAt, &j.CreatedAt, &j.UpdatedAt,
)
if err != nil {
return nil, err
}
_ = json.Unmarshal([]byte(payloadStr), &j.Payload)
_ = json.Unmarshal([]byte(resultStr), &j.Result)
j.ErrorMessage = errorMsg
if idemKey.Valid {
j.IdempotencyKey = idemKey.String
}
return &j, nil
}
func (d *DB) GetJobByIdempotencyKey(key string) (*models.Job, error) {
if key == "" {
return nil, sql.ErrNoRows
}
query := `
SELECT id, type, payload, status, progress, result, error_message, attempts, max_attempts, idempotency_key, run_at, created_at, updated_at
FROM jobs
WHERE idempotency_key = ?
`
var j models.Job
var payloadStr, resultStr, errorMsg string
var idemKey sql.NullString
err := d.QueryRow(query, key).Scan(
&j.ID, &j.Type, &payloadStr, &j.Status, &j.Progress, &resultStr,
&errorMsg, &j.Attempts, &j.MaxAttempts, &idemKey, &j.RunAt, &j.CreatedAt, &j.UpdatedAt,
)
if err != nil {
return nil, err
}
_ = json.Unmarshal([]byte(payloadStr), &j.Payload)
_ = json.Unmarshal([]byte(resultStr), &j.Result)
j.ErrorMessage = errorMsg
if idemKey.Valid {
j.IdempotencyKey = idemKey.String
}
return &j, nil
}
func (d *DB) ListJobs(filter models.JobFilter) ([]models.Job, error) {
if filter.Limit <= 0 {
filter.Limit = 50
}
var conditions []string
var args []interface{}
if filter.Type != "" {
conditions = append(conditions, "type = ?")
args = append(args, filter.Type)
}
if filter.Status != "" {
conditions = append(conditions, "status = ?")
args = append(args, filter.Status)
}
whereClause := ""
if len(conditions) > 0 {
whereClause = "WHERE " + strings.Join(conditions, " AND ")
}
query := fmt.Sprintf(`
SELECT id, type, payload, status, progress, result, error_message, attempts, max_attempts, idempotency_key, run_at, created_at, updated_at
FROM jobs
%s
ORDER BY created_at DESC
LIMIT ? OFFSET ?
`, whereClause)
args = append(args, filter.Limit, filter.Offset)
rows, err := d.Query(query, args...)
if err != nil {
return nil, err
}
defer func() {
_ = rows.Close()
}()
var jobs []models.Job
for rows.Next() {
var j models.Job
var payloadStr, resultStr, errorMsg string
var idemKey sql.NullString
err := rows.Scan(
&j.ID, &j.Type, &payloadStr, &j.Status, &j.Progress, &resultStr,
&errorMsg, &j.Attempts, &j.MaxAttempts, &idemKey, &j.RunAt, &j.CreatedAt, &j.UpdatedAt,
)
if err != nil {
return nil, err
}
_ = json.Unmarshal([]byte(payloadStr), &j.Payload)
_ = json.Unmarshal([]byte(resultStr), &j.Result)
j.ErrorMessage = errorMsg
if idemKey.Valid {
j.IdempotencyKey = idemKey.String
}
jobs = append(jobs, j)
}
return jobs, nil
}
func (d *DB) UpdateJob(job models.Job) error {
job.UpdatedAt = time.Now().UTC()
payloadBytes, _ := json.Marshal(job.Payload)
resultBytes, _ := json.Marshal(job.Result)
if len(resultBytes) == 0 {
resultBytes = []byte("{}")
}
query := `
UPDATE jobs SET
status = ?, progress = ?, result = ?, error_message = ?,
attempts = ?, run_at = ?, updated_at = ?
WHERE id = ?
`
_, err := d.Exec(
query,
job.Status, job.Progress, string(resultBytes), job.ErrorMessage,
job.Attempts, job.RunAt, job.UpdatedAt, job.ID,
)
_ = payloadBytes
return err
}
func (d *DB) ClaimNextJob(types []string) (*models.Job, error) {
now := time.Now().UTC()
var typeFilter string
var args []interface{}
args = append(args, now)
if len(types) > 0 {
placeholders := make([]string, len(types))
for i, t := range types {
placeholders[i] = "?"
args = append(args, t)
}
typeFilter = fmt.Sprintf("AND type IN (%s)", strings.Join(placeholders, ","))
}
query := fmt.Sprintf(`
SELECT id, type, payload, status, progress, result, error_message, attempts, max_attempts, idempotency_key, run_at, created_at, updated_at
FROM jobs
WHERE status = 'pending' AND run_at <= ? %s
ORDER BY run_at ASC, created_at ASC
LIMIT 1
`, typeFilter)
var j models.Job
var payloadStr, resultStr, errorMsg string
var idemKey sql.NullString
err := d.QueryRow(query, args...).Scan(
&j.ID, &j.Type, &payloadStr, &j.Status, &j.Progress, &resultStr,
&errorMsg, &j.Attempts, &j.MaxAttempts, &idemKey, &j.RunAt, &j.CreatedAt, &j.UpdatedAt,
)
if err != nil {
return nil, err
}
// Atomically mark as running
updateRes, err := d.Exec("UPDATE jobs SET status = 'running', updated_at = ? WHERE id = ? AND status = 'pending'", now, j.ID)
if err != nil {
return nil, err
}
rowsAffected, _ := updateRes.RowsAffected()
if rowsAffected == 0 {
return nil, sql.ErrNoRows
}
_ = json.Unmarshal([]byte(payloadStr), &j.Payload)
_ = json.Unmarshal([]byte(resultStr), &j.Result)
j.ErrorMessage = errorMsg
j.Status = "running"
if idemKey.Valid {
j.IdempotencyKey = idemKey.String
}
return &j, nil
}
func (d *DB) ResetRunningJobsToPending() error {
now := time.Now().UTC()
_, err := d.Exec("UPDATE jobs SET status = 'pending', updated_at = ? WHERE status = 'running'", now)
return err
}
func (d *DB) CancelJob(id string) error {
now := time.Now().UTC()
_, err := d.Exec("UPDATE jobs SET status = 'cancelled', updated_at = ? WHERE id = ? AND status IN ('pending', 'running')", now, id)
return err
}
// GetSharedEntry strictly retrieves an entry by share_uuid only (prevents data leak of unshared articles)
func (d *DB) GetSharedEntry(shareUUID string) (*models.FeedEntry, error) {
if shareUUID == "" {
return nil, sql.ErrNoRows
}
var e models.FeedEntry
var fTitle, fSiteURL, author, rawContent, metaStr, sUUID sql.NullString
query := `
SELECT
e.id, e.feed_id, f.title as feed_title, COALESCE(f.site_url, '') as feed_site_url,
e.guid, e.url, e.title, e.author, e.content, e.raw_content,
e.published_at, e.is_read, e.is_starred, e.share_uuid, e.metadata, e.created_at
FROM feed_entries e
JOIN feeds f ON f.id = e.feed_id
WHERE e.share_uuid IS NOT NULL AND e.share_uuid != '' AND e.share_uuid = ?
`
err := d.QueryRow(query, shareUUID).Scan(
&e.ID, &e.FeedID, &fTitle, &fSiteURL,
&e.GUID, &e.URL, &e.Title, &author, &e.Content, &rawContent,
&e.PublishedAt, &e.IsRead, &e.IsStarred, &sUUID, &metaStr, &e.CreatedAt,
)
if err != nil {
return nil, err
}
if fTitle.Valid {
e.FeedTitle = fTitle.String
}
if fSiteURL.Valid {
e.FeedSiteURL = fSiteURL.String
}
if author.Valid {
e.Author = author.String
}
if sUUID.Valid {
e.ShareUUID = &sUUID.String
}
variants, _ := d.ListVariantsForEntry(e.ID)
if len(variants) > 0 {
e.Variants = make(map[string]models.EntryVariant)
for _, v := range variants {
e.Variants[v.VariantKey] = v
}
}
return &e, nil
}
+9
View File
@@ -15,12 +15,21 @@ const (
EventPostCreated EventType = "on_post_created"
EventPostUpdated EventType = "on_post_updated"
EventVariantUpdated EventType = "on_variant_updated"
EventEntryTransition EventType = "on_entry_transition"
EventManualTrigger EventType = "manual"
EventJobProgress EventType = "job_progress"
EventJobCompleted EventType = "job_completed"
EventJobFailed EventType = "job_failed"
EventPreferencesSync EventType = "preferences_updated"
)
type Event struct {
Type EventType
Entry models.FeedEntry
OldEntry *models.FeedEntry
NewEntry *models.FeedEntry
Diff *models.EntryDiff
Job *models.Job
VariantKey string // populated when Type == EventVariantUpdated
Depth int
DirectAttrs map[string]interface{}
+94 -1
View File
@@ -67,7 +67,7 @@ type FeedEntry struct {
PublishedAt time.Time `json:"published_at"`
IsRead bool `json:"is_read"`
IsStarred bool `json:"is_starred"`
ShareUUID *string `json:"share_uuid,omitempty"`
ShareUUID *string `json:"share_uuid"`
Variants map[string]EntryVariant `json:"variants,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
CreatedAt time.Time `json:"created_at"`
@@ -175,3 +175,96 @@ type ChangePasswordRequest struct {
OldPassword string `json:"old_password"`
NewPassword string `json:"new_password"`
}
// Job represents a background task in the persistent job queue
type Job struct {
ID string `json:"id"`
Type string `json:"type"`
Payload map[string]interface{} `json:"payload"`
Status string `json:"status"` // 'pending', 'running', 'completed', 'failed', 'cancelled'
Progress int `json:"progress"`
Result map[string]interface{} `json:"result,omitempty"`
ErrorMessage string `json:"error_message,omitempty"`
Attempts int `json:"attempts"`
MaxAttempts int `json:"max_attempts"`
IdempotencyKey string `json:"idempotency_key,omitempty"`
RunAt time.Time `json:"run_at"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// JobFilter is used to filter job listings
type JobFilter struct {
Type string
Status string
Limit int
Offset int
}
// PatchEntryRequest is the body for PATCH /api/entries/{id}
type PatchEntryRequest struct {
IsRead *bool `json:"is_read,omitempty"`
IsStarred *bool `json:"is_starred,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
Variants map[string]*EntryVariant `json:"variants,omitempty"`
}
// EntryDiff represents the delta between OldState and NewState of an Entry
type EntryDiff struct {
FieldsChanged []string `json:"fields_changed"` // e.g. ["is_read", "is_starred", "metadata.tags"]
VariantsAdded []string `json:"variants_added"` // e.g. ["readability_full_text"]
VariantsChanged []string `json:"variants_changed"` // e.g. ["ai_summary"]
VariantsRemoved []string `json:"variants_removed"`
}
// HasVariantAddition checks if a specific variant key was newly added in this transition
func (d EntryDiff) HasVariantAddition(variantKey string) bool {
for _, k := range d.VariantsAdded {
if k == variantKey {
return true
}
}
return false
}
// HasVariantChange checks if a specific variant key was modified in this transition
func (d EntryDiff) HasVariantChange(variantKey string) bool {
for _, k := range d.VariantsChanged {
if k == variantKey {
return true
}
}
return false
}
// HasFieldChange checks if a specific top-level or metadata field changed
func (d EntryDiff) HasFieldChange(fieldName string) bool {
for _, f := range d.FieldsChanged {
if f == fieldName {
return true
}
}
return false
}
// PromptTemplate represents an AI prompt template
type PromptTemplate struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description,omitempty"`
Prompt string `json:"prompt"`
IsDefault bool `json:"is_default,omitempty"`
}
// AISettings represents AI provider configuration
type AISettings struct {
Enabled bool `json:"enabled"`
Provider string `json:"provider"`
BaseURL string `json:"base_url"`
APIKey string `json:"api_key"`
Model string `json:"model"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
Prompts []PromptTemplate `json:"prompts"`
}
+87 -7
View File
@@ -2,6 +2,9 @@ package rules
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
@@ -14,11 +17,16 @@ type WorkerDispatcher interface {
Dispatch(ctx context.Context, action models.RuleAction, entry models.FeedEntry, depth int)
}
type JobEnqueuer interface {
Enqueue(ctx context.Context, jobType string, payload map[string]interface{}, idempotencyKey string) (*models.Job, error)
}
type Engine struct {
db *db.DB
bus *events.EventBus
evaluator *Evaluator
dispatcher WorkerDispatcher
enqueuer JobEnqueuer
log *zap.Logger
}
@@ -32,8 +40,13 @@ func NewEngine(database *db.DB, bus *events.EventBus, dispatcher WorkerDispatche
}
}
func (e *Engine) SetEnqueuer(enqueuer JobEnqueuer) {
e.enqueuer = enqueuer
}
// Start registers event bus subscriptions to evaluate rules reactively.
func (e *Engine) Start(ctx context.Context) {
e.bus.Subscribe(events.EventEntryTransition, e.handleTransitionEvent)
e.bus.Subscribe(events.EventPostCreated, e.handleEvent)
e.bus.Subscribe(events.EventPostUpdated, e.handleEvent)
e.bus.Subscribe(events.EventVariantUpdated, e.handleEvent)
@@ -41,6 +54,44 @@ func (e *Engine) Start(ctx context.Context) {
e.log.Info("Rules engine started, listening for events")
}
func (e *Engine) handleTransitionEvent(ctx context.Context, evt events.Event) error {
rules, err := e.db.ListRules()
if err != nil {
e.log.Error("Failed to fetch rules for transition", zap.Error(err))
return err
}
newEntry := evt.NewEntry
if newEntry == nil {
newEntry = &evt.Entry
}
for _, r := range rules {
if !r.IsActive {
continue
}
// Check trigger type compatibility
trigger := r.Trigger
if trigger != "on_update" && trigger != "on_entry_transition" && trigger != "on_variant_updated" && trigger != "on_post_updated" && trigger != "on_post_created" && trigger != "variant_change" {
continue
}
if e.evaluator.EvaluateTransition(r, evt.OldEntry, newEntry, evt.Diff) {
e.log.Info("Transition rule matched",
zap.String("rule_id", r.ID),
zap.String("rule_name", r.Name),
zap.String("entry_id", newEntry.ID),
zap.Int("actions_count", len(r.Actions)),
)
e.executeRuleActions(ctx, r, *newEntry, evt.Depth)
}
}
return nil
}
func (e *Engine) handleEvent(ctx context.Context, evt events.Event) error {
trigger := string(evt.Type)
rules, err := e.db.GetActiveRules(trigger)
@@ -53,7 +104,6 @@ func (e *Engine) handleEvent(ctx context.Context, evt events.Event) error {
return nil
}
// Fetch full entry with latest variants if entry ID is present
entry := evt.Entry
if entry.ID != "" {
if full, err := e.db.GetEntry(entry.ID); err == nil && full != nil {
@@ -62,7 +112,7 @@ func (e *Engine) handleEvent(ctx context.Context, evt events.Event) error {
}
for _, r := range rules {
if e.evaluator.Evaluate(r, entry) {
if e.evaluator.EvaluateTransition(r, evt.OldEntry, &entry, evt.Diff) {
e.log.Info("Rule matched",
zap.String("rule_id", r.ID),
zap.String("rule_name", r.Name),
@@ -70,13 +120,43 @@ func (e *Engine) handleEvent(ctx context.Context, evt events.Event) error {
zap.Int("actions_count", len(r.Actions)),
)
for _, action := range r.Actions {
if e.dispatcher != nil {
e.dispatcher.Dispatch(ctx, action, entry, evt.Depth+1)
}
}
e.executeRuleActions(ctx, r, entry, evt.Depth)
}
}
return nil
}
func (e *Engine) executeRuleActions(ctx context.Context, r models.Rule, entry models.FeedEntry, depth int) {
for _, action := range r.Actions {
// Prefer enqueuing async jobs to the persistent task queue if available
if e.enqueuer != nil {
payload := make(map[string]interface{})
if action.Params != nil {
for k, v := range action.Params {
payload[k] = v
}
}
payload["entry_id"] = entry.ID
payload["feed_id"] = entry.FeedID
payload["rule_id"] = r.ID
payload["rule_name"] = r.Name
// Generate deterministic idempotency key
rawKey := fmt.Sprintf("%s:%s:%s:%d:%d", r.ID, entry.ID, action.Type, entry.CreatedAt.Unix(), entry.PublishedAt.Unix())
h := sha256.Sum256([]byte(rawKey))
idempotencyKey := hex.EncodeToString(h[:16])
_, err := e.enqueuer.Enqueue(ctx, action.Type, payload, idempotencyKey)
if err != nil {
e.log.Error("Failed to enqueue job for rule action",
zap.String("rule_id", r.ID),
zap.String("action", action.Type),
zap.Error(err),
)
}
} else if e.dispatcher != nil {
e.dispatcher.Dispatch(ctx, action, entry, depth+1)
}
}
}
+201 -27
View File
@@ -2,6 +2,7 @@ package rules
import (
"fmt"
"reflect"
"regexp"
"strconv"
"strings"
@@ -16,8 +17,16 @@ func NewEvaluator() *Evaluator {
return &Evaluator{}
}
// Evaluate checks whether a FeedEntry satisfies a Rule's conditions.
// Evaluate checks whether a single FeedEntry satisfies a Rule's conditions (legacy / static evaluation).
func (e *Evaluator) Evaluate(rule models.Rule, entry models.FeedEntry) bool {
return e.EvaluateTransition(rule, nil, &entry, nil)
}
// EvaluateTransition evaluates a rule against the (OldState, NewState, EntryDiff) transition delta.
func (e *Evaluator) EvaluateTransition(rule models.Rule, oldEntry *models.FeedEntry, newEntry *models.FeedEntry, diff *models.EntryDiff) bool {
if newEntry == nil {
return false
}
if len(rule.Conditions) == 0 {
return true
}
@@ -25,7 +34,7 @@ func (e *Evaluator) Evaluate(rule models.Rule, entry models.FeedEntry) bool {
matchMode := strings.ToLower(rule.MatchMode)
if matchMode == "" || matchMode == "all" {
for _, cond := range rule.Conditions {
if !e.evaluateCondition(cond, entry) {
if !e.evaluateTransitionCondition(cond, oldEntry, newEntry, diff) {
return false
}
}
@@ -34,20 +43,109 @@ func (e *Evaluator) Evaluate(rule models.Rule, entry models.FeedEntry) bool {
// matchMode == "any"
for _, cond := range rule.Conditions {
if e.evaluateCondition(cond, entry) {
if e.evaluateTransitionCondition(cond, oldEntry, newEntry, diff) {
return true
}
}
return false
}
func (e *Evaluator) evaluateCondition(cond models.RuleCondition, entry models.FeedEntry) bool {
actualValue := e.resolveField(cond.Field, entry)
return e.compare(actualValue, cond.Operator, cond.Value)
func (e *Evaluator) evaluateTransitionCondition(cond models.RuleCondition, oldEntry *models.FeedEntry, newEntry *models.FeedEntry, diff *models.EntryDiff) bool {
opLower := strings.ToLower(strings.TrimSpace(cond.Operator))
// 1. Transition predicates: became_present, changed, removed
switch opLower {
case "became_present", "added":
return e.checkBecamePresent(cond.Field, oldEntry, newEntry, diff)
case "changed", "updated":
return e.checkFieldChanged(cond.Field, oldEntry, newEntry, diff)
case "removed", "deleted":
return e.checkFieldRemoved(cond.Field, oldEntry, newEntry, diff)
}
// 2. Standard state guards & value comparisons evaluated on NewState
actualValue := e.resolveField(cond.Field, newEntry)
return e.compare(actualValue, opLower, cond.Value)
}
func (e *Evaluator) resolveField(field string, entry models.FeedEntry) interface{} {
switch strings.ToLower(field) {
func (e *Evaluator) checkBecamePresent(field string, oldEntry *models.FeedEntry, newEntry *models.FeedEntry, diff *models.EntryDiff) bool {
if newEntry == nil {
return false
}
// If diff is available, check direct addition
if strings.HasPrefix(field, "variants.") {
vKey := strings.TrimPrefix(field, "variants.")
if strings.Contains(vKey, ".") {
vKey = strings.Split(vKey, ".")[0]
}
if diff != nil && diff.HasVariantAddition(vKey) {
return true
}
// Fallback comparison between oldEntry and newEntry
oldHas := oldEntry != nil && oldEntry.Variants != nil && oldEntry.Variants[vKey].TextValue != ""
newHas := newEntry.Variants != nil && newEntry.Variants[vKey].TextValue != ""
return !oldHas && newHas
}
if field == "metadata.tags" || field == "tags" {
oldTags := getTagsSlice(oldEntry)
newTags := getTagsSlice(newEntry)
return len(oldTags) == 0 && len(newTags) > 0
}
oldVal := e.resolveField(field, oldEntry)
newVal := e.resolveField(field, newEntry)
return (oldVal == nil || fmt.Sprintf("%v", oldVal) == "") && (newVal != nil && fmt.Sprintf("%v", newVal) != "")
}
func (e *Evaluator) checkFieldChanged(field string, oldEntry *models.FeedEntry, newEntry *models.FeedEntry, diff *models.EntryDiff) bool {
if oldEntry == nil || newEntry == nil {
return false
}
if strings.HasPrefix(field, "variants.") {
vKey := strings.TrimPrefix(field, "variants.")
if strings.Contains(vKey, ".") {
vKey = strings.Split(vKey, ".")[0]
}
if diff != nil && (diff.HasVariantAddition(vKey) || diff.HasVariantChange(vKey)) {
return true
}
}
oldVal := e.resolveField(field, oldEntry)
newVal := e.resolveField(field, newEntry)
return !reflect.DeepEqual(oldVal, newVal)
}
func (e *Evaluator) checkFieldRemoved(field string, oldEntry *models.FeedEntry, newEntry *models.FeedEntry, diff *models.EntryDiff) bool {
if oldEntry == nil {
return false
}
if strings.HasPrefix(field, "variants.") {
vKey := strings.TrimPrefix(field, "variants.")
if diff != nil {
for _, r := range diff.VariantsRemoved {
if r == vKey {
return true
}
}
}
oldHas := oldEntry.Variants != nil && oldEntry.Variants[vKey].TextValue != ""
newHas := newEntry != nil && newEntry.Variants != nil && newEntry.Variants[vKey].TextValue != ""
return oldHas && !newHas
}
oldVal := e.resolveField(field, oldEntry)
newVal := e.resolveField(field, newEntry)
return (oldVal != nil && fmt.Sprintf("%v", oldVal) != "") && (newVal == nil || fmt.Sprintf("%v", newVal) == "")
}
func (e *Evaluator) resolveField(field string, entry *models.FeedEntry) interface{} {
if entry == nil {
return nil
}
fieldLower := strings.ToLower(strings.TrimSpace(field))
switch fieldLower {
case "feed_id":
return entry.FeedID
case "feed_title":
@@ -83,16 +181,34 @@ func (e *Evaluator) resolveField(field string, entry models.FeedEntry) interface
return entry.PublishedAt
case "created_at":
return entry.CreatedAt
case "tags", "metadata.tags":
return getTagsSlice(entry)
}
// Variant field resolution: "variants.keywords", "variants.readability_full_text", "variants.tags"
if strings.HasPrefix(field, "variants.") {
variantKey := strings.TrimPrefix(field, "variants.")
// Variant resolution: "variants.ai_summary.word_count", "variants.ai_summary", "variants.readability_full_text"
if strings.HasPrefix(fieldLower, "variants.") {
path := strings.TrimPrefix(fieldLower, "variants.")
parts := strings.Split(path, ".")
vKey := parts[0]
if entry.Variants != nil {
if v, ok := entry.Variants[variantKey]; ok {
if v, ok := entry.Variants[vKey]; ok {
if len(parts) > 1 {
subProp := parts[1]
if subProp == "word_count" {
return float64(len(strings.Fields(v.TextValue)))
}
if subProp == "char_count" || subProp == "length" {
return float64(len(v.TextValue))
}
if v.Metadata != nil {
return v.Metadata[subProp]
}
}
return v.TextValue
}
}
return nil
}
return nil
@@ -101,35 +217,85 @@ func (e *Evaluator) resolveField(field string, entry models.FeedEntry) interface
func (e *Evaluator) compare(actual interface{}, op string, expected interface{}) bool {
opLower := strings.ToLower(strings.TrimSpace(op))
// Handle exists / not_exists operators
if opLower == "exists" {
if actual == nil {
return false
}
if s, ok := actual.(string); ok && s == "" {
return false
}
return true
}
if opLower == "not_exists" {
// Handle is_empty / is_present / exists / not_exists
if opLower == "is_empty" || opLower == "not_exists" {
if actual == nil {
return true
}
if s, ok := actual.(string); ok && s == "" {
if s, ok := actual.(string); ok && strings.TrimSpace(s) == "" {
return true
}
if slice, ok := actual.([]string); ok && len(slice) == 0 {
return true
}
return false
}
if opLower == "is_present" || opLower == "exists" {
if actual == nil {
return false
}
if s, ok := actual.(string); ok && strings.TrimSpace(s) == "" {
return false
}
if slice, ok := actual.([]string); ok && len(slice) == 0 {
return false
}
return true
}
if actual == nil {
return opLower == "neq" || opLower == "!=" || opLower == "not_in" || opLower == "not_contains"
}
// Handle Time/Timestamp comparisons
// Handle numeric comparisons (word_count >= 500, etc.)
if actualNum, ok := toFloat64(actual); ok == nil {
if expNum, okExp := toFloat64(expected); okExp == nil {
switch opLower {
case ">=":
return actualNum >= expNum
case ">":
return actualNum > expNum
case "<=":
return actualNum <= expNum
case "<":
return actualNum < expNum
case "eq", "==":
return actualNum == expNum
case "neq", "!=":
return actualNum != expNum
}
}
}
// Handle Time comparisons
if actualTime, ok := actual.(time.Time); ok {
return e.compareTime(actualTime, opLower, expected)
}
// Handle slice containment (e.g. tags contains "Tech")
if actualSlice, ok := actual.([]string); ok {
expectedStr := fmt.Sprintf("%v", expected)
switch opLower {
case "contains", "in":
for _, item := range actualSlice {
if strings.EqualFold(item, expectedStr) {
return true
}
}
return false
case "contains_any":
tokens := parseStringList(expected)
for _, item := range actualSlice {
for _, t := range tokens {
if strings.EqualFold(item, t) {
return true
}
}
}
return false
}
}
actualStr := fmt.Sprintf("%v", actual)
expectedStr := fmt.Sprintf("%v", expected)
@@ -218,6 +384,16 @@ func (e *Evaluator) compareTime(t time.Time, op string, expected interface{}) bo
return false
}
func getTagsSlice(entry *models.FeedEntry) []string {
if entry == nil || entry.Metadata == nil {
return nil
}
if raw, ok := entry.Metadata["tags"]; ok {
return parseStringList(raw)
}
return nil
}
func parseStringList(val interface{}) []string {
var result []string
clean := func(s string) string {
@@ -279,7 +455,6 @@ func parseTime(val interface{}) (time.Time, error) {
func ValidateRuleSafety(rule models.Rule) error {
trigger := strings.ToLower(rule.Trigger)
// If trigger is on_variant_updated / variant_change, check for recursive set_variant on same key
if trigger == "on_variant_updated" || trigger == "variant_change" {
for _, action := range rule.Actions {
if action.Type == "set_variant" {
@@ -288,7 +463,6 @@ func ValidateRuleSafety(rule models.Rule) error {
targetKey, _ = action.Params["key"].(string)
}
// Check if trigger targets this exact key without safe conditions
for _, cond := range rule.Conditions {
if cond.Field == "variants."+targetKey && (cond.Operator == "exists" || cond.Operator == "eq") {
return fmt.Errorf("cyclic rule detected: rule triggers on variant '%s' and unconditionally writes to '%s'", targetKey, targetKey)
+70
View File
@@ -116,3 +116,73 @@ func TestRuleEvaluator(t *testing.T) {
t.Error("expected ValidateRuleSafety to detect cyclic loop")
}
}
func TestTransitionEvaluation(t *testing.T) {
eval := NewEvaluator()
oldEntry := models.FeedEntry{
ID: "entry-2",
FeedID: "feed-1",
FeedTitle: "Tech Blog",
Title: "Deep Dive into Go Scheduler",
Content: "Short intro...",
Variants: map[string]models.EntryVariant{},
}
newEntryWithFullText := models.FeedEntry{
ID: "entry-2",
FeedID: "feed-1",
FeedTitle: "Tech Blog",
Title: "Deep Dive into Go Scheduler",
Content: "Short intro...",
Variants: map[string]models.EntryVariant{
"readability_full_text": {
EntryID: "entry-2",
VariantKey: "readability_full_text",
TextValue: "Full article content here with extensive details...",
},
},
}
diff := &models.EntryDiff{
VariantsAdded: []string{"readability_full_text"},
}
// Rule: trigger AI summary when readability_full_text became_present AND ai_summary is empty
aiRule := models.Rule{
ID: "r-ai",
Name: "AI summary on full text added",
MatchMode: "all",
Conditions: []models.RuleCondition{
{Field: "variants.readability_full_text", Operator: "became_present"},
{Field: "variants.ai_summary", Operator: "is_empty"},
},
}
// Should match when full text was added
if !eval.EvaluateTransition(aiRule, &oldEntry, &newEntryWithFullText, diff) {
t.Error("expected aiRule to match when readability_full_text became_present")
}
// Should NOT match if tag was added instead of full text
diffOnlyTags := &models.EntryDiff{
FieldsChanged: []string{"metadata.tags"},
}
if eval.EvaluateTransition(aiRule, &oldEntry, &oldEntry, diffOnlyTags) {
t.Error("expected aiRule to NOT match when only tags changed")
}
// Test word count condition for TTS: word_count >= 5
ttsRule := models.Rule{
ID: "r-tts",
Name: "TTS on long summary",
MatchMode: "all",
Conditions: []models.RuleCondition{
{Field: "variants.readability_full_text.word_count", Operator: ">=", Value: 5},
},
}
if !eval.EvaluateTransition(ttsRule, &oldEntry, &newEntryWithFullText, diff) {
t.Error("expected ttsRule to match when word_count >= 5")
}
}
+218
View File
@@ -0,0 +1,218 @@
package services
import (
"context"
"fmt"
"reflect"
"strings"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/logger"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"go.uber.org/zap"
)
type EntryService struct {
db *db.DB
bus *events.EventBus
log *zap.Logger
}
func NewEntryService(database *db.DB, bus *events.EventBus) *EntryService {
return &EntryService{
db: database,
bus: bus,
log: logger.Named("entry-service"),
}
}
// IngestEntries handles bulk ingestion of parsed feed entries (from crawler or OPML import)
func (s *EntryService) IngestEntries(ctx context.Context, feedID string, entries []models.FeedEntry, feedTitle, feedSiteURL string) ([]models.FeedEntry, error) {
newEntries, err := s.db.UpsertFeedEntries(feedID, entries)
if err != nil {
return nil, err
}
for _, entry := range newEntries {
entry.FeedTitle = feedTitle
entry.FeedSiteURL = feedSiteURL
diff := &models.EntryDiff{
FieldsChanged: []string{"id", "title", "content", "url"},
}
if s.bus != nil {
s.bus.Publish(ctx, events.Event{
Type: events.EventPostCreated,
Entry: entry,
NewEntry: &entry,
Diff: diff,
Depth: 0,
})
s.bus.Publish(ctx, events.Event{
Type: events.EventEntryTransition,
Entry: entry,
NewEntry: &entry,
Diff: diff,
Depth: 0,
})
}
}
return newEntries, nil
}
// PatchEntry applies atomic mutations to an entry, calculates the EntryDiff between Old and New states,
// and fires EventEntryTransition to trigger downstream transition-based automation rules.
func (s *EntryService) PatchEntry(ctx context.Context, entryID string, patch models.PatchEntryRequest) (*models.FeedEntry, *models.EntryDiff, error) {
oldEntry, err := s.db.GetEntry(entryID)
if err != nil {
return nil, nil, fmt.Errorf("entry not found: %w", err)
}
diff := &models.EntryDiff{
FieldsChanged: make([]string, 0),
VariantsAdded: make([]string, 0),
VariantsChanged: make([]string, 0),
VariantsRemoved: make([]string, 0),
}
// 1. Update IsRead
if patch.IsRead != nil && *patch.IsRead != oldEntry.IsRead {
if err := s.db.UpdateEntryRead(entryID, *patch.IsRead); err != nil {
return nil, nil, err
}
diff.FieldsChanged = append(diff.FieldsChanged, "is_read")
}
// 2. Update IsStarred
if patch.IsStarred != nil && *patch.IsStarred != oldEntry.IsStarred {
if err := s.db.UpdateEntryStarred(entryID, *patch.IsStarred); err != nil {
return nil, nil, err
}
diff.FieldsChanged = append(diff.FieldsChanged, "is_starred")
}
// 3. Update Metadata (e.g. tags)
if patch.Metadata != nil {
if oldEntry.Metadata == nil {
oldEntry.Metadata = make(map[string]interface{})
}
metadataChanged := false
for k, v := range patch.Metadata {
if !reflect.DeepEqual(oldEntry.Metadata[k], v) {
oldEntry.Metadata[k] = v
metadataChanged = true
diff.FieldsChanged = append(diff.FieldsChanged, "metadata."+k)
}
}
if metadataChanged {
if err := s.db.UpdateEntryMetadata(entryID, oldEntry.Metadata); err != nil {
return nil, nil, err
}
}
}
// 4. Update Variants
if patch.Variants != nil {
for vKey, vVal := range patch.Variants {
if vVal == nil {
// Variant removal
if oldEntry.Variants != nil {
if _, exists := oldEntry.Variants[vKey]; exists {
_ = s.db.DeleteVariant(entryID, vKey)
diff.VariantsRemoved = append(diff.VariantsRemoved, vKey)
}
}
continue
}
// Variant add or update
vVal.EntryID = entryID
vVal.VariantKey = vKey
oldVariant, exists := oldEntry.Variants[vKey]
if !exists {
diff.VariantsAdded = append(diff.VariantsAdded, vKey)
} else if oldVariant.TextValue != vVal.TextValue || oldVariant.BlobPath != vVal.BlobPath || !reflect.DeepEqual(oldVariant.Metadata, vVal.Metadata) {
diff.VariantsChanged = append(diff.VariantsChanged, vKey)
}
if err := s.db.SetVariant(*vVal); err != nil {
return nil, nil, fmt.Errorf("failed to save variant %s: %w", vKey, err)
}
}
}
// Fetch fresh NewState
newEntry, err := s.db.GetEntry(entryID)
if err != nil {
return nil, nil, fmt.Errorf("failed to retrieve updated entry: %w", err)
}
s.log.Debug("Entry patched with transition diff",
zap.String("entry_id", entryID),
zap.Strings("fields_changed", diff.FieldsChanged),
zap.Strings("variants_added", diff.VariantsAdded),
zap.Strings("variants_changed", diff.VariantsChanged),
)
// Publish transition event to trigger reactive rules
if s.bus != nil {
s.bus.Publish(ctx, events.Event{
Type: events.EventEntryTransition,
Entry: *newEntry,
OldEntry: oldEntry,
NewEntry: newEntry,
Diff: diff,
Depth: 0,
})
s.bus.Publish(ctx, events.Event{
Type: events.EventPostUpdated,
Entry: *newEntry,
OldEntry: oldEntry,
NewEntry: newEntry,
Diff: diff,
Depth: 0,
})
for _, vKey := range diff.VariantsAdded {
s.bus.Publish(ctx, events.Event{
Type: events.EventVariantUpdated,
Entry: *newEntry,
VariantKey: vKey,
Depth: 0,
})
}
for _, vKey := range diff.VariantsChanged {
s.bus.Publish(ctx, events.Event{
Type: events.EventVariantUpdated,
Entry: *newEntry,
VariantKey: vKey,
Depth: 0,
})
}
}
return newEntry, diff, nil
}
func (s *EntryService) GetEntry(ctx context.Context, id string) (*models.FeedEntry, error) {
return s.db.GetEntry(id)
}
func (s *EntryService) ListEntries(ctx context.Context, filter db.EntryFilter) ([]models.FeedEntry, int, error) {
return s.db.ListEntries(filter)
}
func (s *EntryService) MarkAllRead(ctx context.Context, feedID, categoryID *string) error {
return s.db.MarkAllRead(feedID, categoryID)
}
func (s *EntryService) ToggleShare(ctx context.Context, id string) (*string, error) {
return s.db.ToggleShareEntry(id)
}
func (s *EntryService) GetSharedEntry(ctx context.Context, shareUUID string) (*models.FeedEntry, error) {
return s.db.GetSharedEntry(strings.TrimSuffix(shareUUID, ".html"))
}
+85
View File
@@ -0,0 +1,85 @@
package services
import (
"context"
"path/filepath"
"testing"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
)
func TestEntryServicePatchAndTransition(t *testing.T) {
tempDBPath := filepath.Join(t.TempDir(), "test.db")
database, err := db.InitDB(tempDBPath)
if err != nil {
t.Fatalf("failed to init db: %v", err)
}
defer func() { _ = database.Close() }()
bus := events.NewEventBus()
svc := NewEntryService(database, bus)
// Create test category and feed
cat, _ := database.CreateCategory("Tech", nil)
feed, _ := database.CreateFeed("Test Feed", "https://example.com/rss", "https://example.com", "", &cat.ID)
// Ingest an entry
entry := models.FeedEntry{
ID: "e_test_1",
FeedID: feed.ID,
GUID: "guid_1",
URL: "https://example.com/p/1",
Title: "Test Post",
Content: "Hello world",
PublishedAt: time.Now().UTC(),
}
ingested, err := svc.IngestEntries(context.Background(), feed.ID, []models.FeedEntry{entry}, feed.Title, feed.SiteURL)
if err != nil || len(ingested) == 0 {
t.Fatalf("failed to ingest entry: %v", err)
}
// 1. Test patching tags
isStarred := true
updated, diff, err := svc.PatchEntry(context.Background(), ingested[0].ID, models.PatchEntryRequest{
IsStarred: &isStarred,
Metadata: map[string]interface{}{
"tags": []string{"AI", "Kubernetes"},
},
})
if err != nil {
t.Fatalf("failed to patch entry: %v", err)
}
if !updated.IsStarred {
t.Errorf("expected entry to be starred")
}
if !diff.HasFieldChange("is_starred") {
t.Errorf("expected diff to contain is_starred change")
}
// 2. Test patching variant addition (readability_full_text)
fullTextVariant := &models.EntryVariant{
ContentType: "html",
TextValue: "<p>Full article extracted content...</p>",
}
updated2, diff2, err := svc.PatchEntry(context.Background(), ingested[0].ID, models.PatchEntryRequest{
Variants: map[string]*models.EntryVariant{
"readability_full_text": fullTextVariant,
},
})
if err != nil {
t.Fatalf("failed to patch variant: %v", err)
}
if !diff2.HasVariantAddition("readability_full_text") {
t.Errorf("expected diff2 to report variant addition of readability_full_text")
}
if updated2.Variants["readability_full_text"].TextValue == "" {
t.Errorf("expected variant to be loaded in updated entry")
}
}
+186
View File
@@ -0,0 +1,186 @@
package services
import (
"context"
"fmt"
"strings"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/crawler"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/logger"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/opml"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/parser"
"go.uber.org/zap"
)
type FeedService struct {
db *db.DB
entryService *EntryService
parser *parser.Parser
crawler *crawler.Crawler
log *zap.Logger
}
func NewFeedService(database *db.DB, entrySvc *EntryService, cr *crawler.Crawler) *FeedService {
return &FeedService{
db: database,
entryService: entrySvc,
parser: parser.NewParser(),
crawler: cr,
log: logger.Named("feed-service"),
}
}
func (s *FeedService) ListFeeds(ctx context.Context) ([]models.Feed, error) {
return s.db.ListFeeds()
}
func (s *FeedService) GetFeed(ctx context.Context, id string) (*models.Feed, error) {
return s.db.GetFeed(id)
}
func (s *FeedService) CreateFeed(ctx context.Context, req models.CreateFeedRequest) (*models.Feed, error) {
feedURL := strings.TrimSpace(req.FeedURL)
if feedURL == "" {
return nil, fmt.Errorf("feed_url is required")
}
if existing, _ := s.db.GetFeedByURL(feedURL); existing != nil {
return nil, fmt.Errorf("feed is already subscribed")
}
res, err := s.parser.FetchFeed(ctx, feedURL, "", "")
if err != nil {
return nil, fmt.Errorf("failed to fetch feed: %w", err)
}
title := res.Title
if req.Title != nil && strings.TrimSpace(*req.Title) != "" {
title = strings.TrimSpace(*req.Title)
}
iconURL := ""
if req.IconURL != nil && strings.TrimSpace(*req.IconURL) != "" {
iconURL = strings.TrimSpace(*req.IconURL)
}
feed, err := s.db.CreateFeed(title, feedURL, res.SiteURL, iconURL, req.CategoryID)
if err != nil {
return nil, err
}
if len(res.Entries) > 0 {
_, _ = s.entryService.IngestEntries(ctx, feed.ID, res.Entries, feed.Title, feed.SiteURL)
_ = s.db.UpdateFeedFetchStatus(feed.ID, res.ETag, res.LastModified, "")
}
return s.db.GetFeed(feed.ID)
}
func (s *FeedService) UpdateFeed(ctx context.Context, id string, req models.UpdateFeedRequest) (*models.Feed, error) {
if err := s.db.UpdateFeed(id, req); err != nil {
return nil, err
}
return s.db.GetFeed(id)
}
func (s *FeedService) DeleteFeed(ctx context.Context, id string) error {
return s.db.DeleteFeed(id)
}
func (s *FeedService) RefreshFeed(ctx context.Context, id string) (*models.Feed, error) {
if s.crawler != nil {
if err := s.crawler.RefreshSingle(ctx, id); err != nil {
return nil, err
}
}
return s.db.GetFeed(id)
}
func (s *FeedService) RefreshAllFeeds(ctx context.Context) {
if s.crawler != nil {
go s.crawler.RefreshAll(context.Background())
}
}
// Category methods
func (s *FeedService) ListCategories(ctx context.Context) ([]models.Category, error) {
return s.db.ListCategories()
}
func (s *FeedService) CreateCategory(ctx context.Context, title string, parentID *string) (*models.Category, error) {
return s.db.CreateCategory(strings.TrimSpace(title), parentID)
}
func (s *FeedService) DeleteCategory(ctx context.Context, id string) error {
return s.db.DeleteCategory(id)
}
func (s *FeedService) GetStats(ctx context.Context) (*models.Stats, error) {
return s.db.GetStats()
}
// OPML methods
func (s *FeedService) ImportOPML(ctx context.Context, data []byte) (int, int, error) {
importedFeeds, err := opml.ParseOPML(data)
if err != nil {
return 0, 0, fmt.Errorf("invalid OPML XML: %w", err)
}
categoryCache := make(map[string]string)
cats, _ := s.db.ListCategories()
for _, c := range cats {
categoryCache[c.Title] = c.ID
}
createdCount := 0
for _, item := range importedFeeds {
if item.FeedURL == "" {
continue
}
if existing, _ := s.db.GetFeedByURL(item.FeedURL); existing != nil {
continue
}
var catID *string
if item.CategoryTitle != "" {
if cid, ok := categoryCache[item.CategoryTitle]; ok {
catID = &cid
} else {
newCat, err := s.db.CreateCategory(item.CategoryTitle, nil)
if err == nil {
categoryCache[item.CategoryTitle] = newCat.ID
catID = &newCat.ID
}
}
}
title := item.Title
if title == "" {
title = item.FeedURL
}
_, err := s.db.CreateFeed(title, item.FeedURL, item.SiteURL, "", catID)
if err == nil {
createdCount++
}
}
if s.crawler != nil {
go s.crawler.RefreshAll(context.Background())
}
return createdCount, len(importedFeeds), nil
}
func (s *FeedService) ExportOPML(ctx context.Context) ([]byte, error) {
feeds, err := s.db.ListFeeds()
if err != nil {
return nil, err
}
cats, err := s.db.ListCategories()
if err != nil {
return nil, err
}
return opml.GenerateOPML(feeds, cats)
}
+126
View File
@@ -0,0 +1,126 @@
package services
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/logger"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"github.com/google/uuid"
"go.uber.org/zap"
)
type JobService struct {
db *db.DB
bus *events.EventBus
log *zap.Logger
}
func NewJobService(database *db.DB, bus *events.EventBus) *JobService {
return &JobService{
db: database,
bus: bus,
log: logger.Named("job-service"),
}
}
// GenerateIdempotencyKey creates a deterministic key to prevent duplicate jobs
func GenerateIdempotencyKey(jobType, entityID, action string, extra string) string {
raw := fmt.Sprintf("%s:%s:%s:%s", jobType, entityID, action, extra)
h := sha256.Sum256([]byte(raw))
return hex.EncodeToString(h[:16])
}
// Enqueue submits a job to the persistent SQLite queue.
// If an idempotency key is provided and a job already exists with that key, returns the existing job.
func (s *JobService) Enqueue(ctx context.Context, jobType string, payload map[string]interface{}, idempotencyKey string) (*models.Job, error) {
if idempotencyKey != "" {
if existing, err := s.db.GetJobByIdempotencyKey(idempotencyKey); err == nil && existing != nil {
s.log.Debug("Job already exists with idempotency key, skipping duplicate enqueue",
zap.String("job_id", existing.ID),
zap.String("type", jobType),
zap.String("key", idempotencyKey),
)
return existing, nil
}
}
job := models.Job{
ID: uuid.New().String(),
Type: jobType,
Payload: payload,
Status: "pending",
Progress: 0,
Attempts: 0,
MaxAttempts: 3,
IdempotencyKey: idempotencyKey,
RunAt: time.Now().UTC(),
CreatedAt: time.Now().UTC(),
UpdatedAt: time.Now().UTC(),
}
created, err := s.db.CreateJob(job)
if err != nil {
return nil, fmt.Errorf("failed to enqueue job: %w", err)
}
s.log.Info("Job enqueued",
zap.String("job_id", created.ID),
zap.String("type", created.Type),
zap.String("idempotency_key", idempotencyKey),
)
if s.bus != nil {
s.bus.Publish(ctx, events.Event{
Type: events.EventJobProgress,
Job: created,
})
}
return created, nil
}
func (s *JobService) GetJob(ctx context.Context, id string) (*models.Job, error) {
return s.db.GetJob(id)
}
func (s *JobService) ListJobs(ctx context.Context, filter models.JobFilter) ([]models.Job, error) {
return s.db.ListJobs(filter)
}
func (s *JobService) CancelJob(ctx context.Context, id string) error {
err := s.db.CancelJob(id)
if err == nil && s.bus != nil {
if j, err := s.db.GetJob(id); err == nil {
s.bus.Publish(ctx, events.Event{
Type: events.EventJobFailed,
Job: j,
})
}
}
return err
}
func (s *JobService) RetryJob(ctx context.Context, id string) error {
j, err := s.db.GetJob(id)
if err != nil {
return err
}
j.Status = "pending"
j.ErrorMessage = ""
j.Attempts = 0
j.RunAt = time.Now().UTC()
err = s.db.UpdateJob(*j)
if err == nil && s.bus != nil {
s.bus.Publish(ctx, events.Event{
Type: events.EventJobProgress,
Job: j,
})
}
return err
}
+98
View File
@@ -0,0 +1,98 @@
package services
import (
"context"
"encoding/json"
"fmt"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/logger"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"go.uber.org/zap"
)
type SettingsService struct {
db *db.DB
bus *events.EventBus
log *zap.Logger
}
func NewSettingsService(database *db.DB, bus *events.EventBus) *SettingsService {
return &SettingsService{
db: database,
bus: bus,
log: logger.Named("settings-service"),
}
}
func (s *SettingsService) GetPreferences(ctx context.Context, username string) (map[string]interface{}, error) {
user, err := s.db.GetUserByUsername(username)
if err != nil {
return nil, err
}
if user.Preferences == nil {
return make(map[string]interface{}), nil
}
return user.Preferences, nil
}
func (s *SettingsService) UpdatePreferences(ctx context.Context, username string, newPrefs map[string]interface{}) (map[string]interface{}, error) {
user, err := s.db.GetUserByUsername(username)
if err != nil {
return nil, err
}
if user.Preferences == nil {
user.Preferences = make(map[string]interface{})
}
for k, v := range newPrefs {
user.Preferences[k] = v
}
if err := s.db.UpdateUserPreferences(user.ID, user.Preferences); err != nil {
return nil, err
}
if s.bus != nil {
s.bus.Publish(ctx, events.Event{
Type: events.EventPreferencesSync,
DirectAttrs: user.Preferences,
})
}
return user.Preferences, nil
}
func (s *SettingsService) GetAISettings(ctx context.Context) (*models.AISettings, error) {
raw, err := s.db.GetSetting("ai_settings")
if err != nil || raw == "" {
return &models.AISettings{
Enabled: false,
Provider: "openai",
BaseURL: "https://api.openai.com/v1",
APIKey: "",
Model: "gpt-4o-mini",
Temperature: 0.3,
MaxTokens: 1000,
Prompts: []models.PromptTemplate{},
}, nil
}
var cfg models.AISettings
if err := json.Unmarshal([]byte(raw), &cfg); err != nil {
return nil, fmt.Errorf("failed to parse AI settings: %w", err)
}
return &cfg, nil
}
func (s *SettingsService) UpdateAISettings(ctx context.Context, cfg models.AISettings) (*models.AISettings, error) {
bytes, err := json.Marshal(cfg)
if err != nil {
return nil, fmt.Errorf("failed to marshal AI settings: %w", err)
}
if err := s.db.SetSetting("ai_settings", string(bytes)); err != nil {
return nil, err
}
return &cfg, nil
}
+268
View File
@@ -0,0 +1,268 @@
package worker
import (
"context"
"fmt"
"sync"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/logger"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
"go.uber.org/zap"
)
type JobHandlerFunc func(ctx context.Context, job models.Job) (*models.PatchEntryRequest, map[string]interface{}, error)
type EntryPatcher interface {
PatchEntry(ctx context.Context, entryID string, patch models.PatchEntryRequest) (*models.FeedEntry, *models.EntryDiff, error)
GetEntry(ctx context.Context, id string) (*models.FeedEntry, error)
}
type JobEngine struct {
db *db.DB
bus *events.EventBus
patcher EntryPatcher
handlers map[string]JobHandlerFunc
semaphores map[string]chan struct{}
maxWorkers int
pollTicker *time.Ticker
wg sync.WaitGroup
log *zap.Logger
mu sync.RWMutex
}
func NewJobEngine(database *db.DB, bus *events.EventBus, patcher EntryPatcher, maxWorkers int) *JobEngine {
if maxWorkers <= 0 {
maxWorkers = 10
}
engine := &JobEngine{
db: database,
bus: bus,
patcher: patcher,
handlers: make(map[string]JobHandlerFunc),
semaphores: make(map[string]chan struct{}),
maxWorkers: maxWorkers,
log: logger.Named("job-engine"),
}
// Set default concurrency limits per task type
engine.SetConcurrency("ai_summary", 2)
engine.SetConcurrency("ai_enrich", 2)
engine.SetConcurrency("readability_scrape", 5)
engine.SetConcurrency("tts_generate", 2)
engine.SetConcurrency("feed_refresh", 10)
engine.SetConcurrency("add_tags", 10)
return engine
}
func (e *JobEngine) SetConcurrency(jobType string, limit int) {
e.mu.Lock()
defer e.mu.Unlock()
if limit <= 0 {
limit = 2
}
e.semaphores[jobType] = make(chan struct{}, limit)
}
func (e *JobEngine) RegisterHandler(jobType string, handler JobHandlerFunc) {
e.mu.Lock()
defer e.mu.Unlock()
e.handlers[jobType] = handler
if _, ok := e.semaphores[jobType]; !ok {
e.semaphores[jobType] = make(chan struct{}, 5)
}
e.log.Info("Registered job handler", zap.String("type", jobType))
}
func (e *JobEngine) Start(ctx context.Context) {
e.log.Info("Starting persistent SQLite job queue engine")
// 1. Recover crashed/orphaned jobs on boot
if err := e.db.ResetRunningJobsToPending(); err != nil {
e.log.Error("Failed to reset orphaned running jobs", zap.Error(err))
}
// 2. Start dispatcher loop
e.pollTicker = time.NewTicker(500 * time.Millisecond)
go func() {
for {
select {
case <-ctx.Done():
return
case <-e.pollTicker.C:
e.dispatchPendingJobs(ctx)
}
}
}()
}
func (e *JobEngine) Stop() {
if e.pollTicker != nil {
e.pollTicker.Stop()
}
e.wg.Wait()
e.log.Info("Job engine stopped")
}
func (e *JobEngine) dispatchPendingJobs(ctx context.Context) {
e.mu.RLock()
var registeredTypes []string
for t := range e.handlers {
registeredTypes = append(registeredTypes, t)
}
e.mu.RUnlock()
if len(registeredTypes) == 0 {
return
}
// Claim next due job
job, err := e.db.ClaimNextJob(registeredTypes)
if err != nil || job == nil {
return
}
// Acquire concurrency slot for this job type
e.mu.RLock()
sem, hasSem := e.semaphores[job.Type]
handler, hasHandler := e.handlers[job.Type]
e.mu.RUnlock()
if !hasHandler {
e.log.Warn("No handler registered for claimed job", zap.String("job_id", job.ID), zap.String("type", job.Type))
return
}
if hasSem {
select {
case sem <- struct{}{}:
// Slot acquired
default:
// Concurrency limit reached for this type, release job back to pending
job.Status = "pending"
_ = e.db.UpdateJob(*job)
return
}
}
e.wg.Add(1)
go func(j models.Job, h JobHandlerFunc, s chan struct{}) {
defer e.wg.Done()
if s != nil {
defer func() { <-s }()
}
e.executeJob(ctx, j, h)
}(*job, handler, sem)
}
func (e *JobEngine) executeJob(ctx context.Context, job models.Job, handler JobHandlerFunc) {
e.log.Info("Executing job",
zap.String("job_id", job.ID),
zap.String("type", job.Type),
zap.Int("attempt", job.Attempts+1),
)
// Emit Job Progress
if e.bus != nil {
e.bus.Publish(ctx, events.Event{
Type: events.EventJobProgress,
Job: &job,
})
}
start := time.Now()
patchReq, result, err := handler(ctx, job)
duration := time.Since(start)
if err != nil {
job.Attempts++
e.log.Error("Job execution failed",
zap.String("job_id", job.ID),
zap.String("type", job.Type),
zap.Int("attempts", job.Attempts),
zap.Duration("duration", duration),
zap.Error(err),
)
if job.Attempts < job.MaxAttempts {
// Exponential backoff retry: 5s, 10s, 20s...
backoff := time.Duration(1<<job.Attempts) * 5 * time.Second
job.Status = "pending"
job.ErrorMessage = err.Error()
job.RunAt = time.Now().UTC().Add(backoff)
_ = e.db.UpdateJob(job)
return
}
// Permanent failure
job.Status = "failed"
job.ErrorMessage = err.Error()
job.Progress = 100
_ = e.db.UpdateJob(job)
if e.bus != nil {
e.bus.Publish(ctx, events.Event{
Type: events.EventJobFailed,
Job: &job,
})
}
return
}
// Apply Patch to Entry if returned
entryID, _ := job.Payload["entry_id"].(string)
if patchReq != nil && entryID != "" && e.patcher != nil {
_, _, patchErr := e.patcher.PatchEntry(ctx, entryID, *patchReq)
if patchErr != nil {
e.log.Error("Failed to patch entry from job result",
zap.String("job_id", job.ID),
zap.String("entry_id", entryID),
zap.Error(patchErr),
)
}
}
// Mark Job Completed
job.Status = "completed"
job.Progress = 100
job.Result = result
job.ErrorMessage = ""
_ = e.db.UpdateJob(job)
e.log.Info("Job completed successfully",
zap.String("job_id", job.ID),
zap.String("type", job.Type),
zap.Duration("duration", duration),
)
if e.bus != nil {
e.bus.Publish(ctx, events.Event{
Type: events.EventJobCompleted,
Job: &job,
})
}
}
// ExecuteDirect executes a registered action synchronously (e.g. for fallback or test)
func (e *JobEngine) ExecuteDirect(ctx context.Context, jobType string, payload map[string]interface{}) (*models.PatchEntryRequest, map[string]interface{}, error) {
e.mu.RLock()
handler, ok := e.handlers[jobType]
e.mu.RUnlock()
if !ok {
return nil, nil, fmt.Errorf("unknown job type: %s", jobType)
}
job := models.Job{
ID: "direct_exec",
Type: jobType,
Payload: payload,
}
return handler(ctx, job)
}
+84
View File
@@ -0,0 +1,84 @@
package worker
import (
"context"
"path/filepath"
"sync/atomic"
"testing"
"time"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/db"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/events"
"git.pengzhan.dev/haopengzhan/rsspipe/internal/models"
)
type mockPatcher struct {
patchedCount int32
}
func (m *mockPatcher) PatchEntry(ctx context.Context, entryID string, patch models.PatchEntryRequest) (*models.FeedEntry, *models.EntryDiff, error) {
atomic.AddInt32(&m.patchedCount, 1)
return &models.FeedEntry{ID: entryID}, &models.EntryDiff{}, nil
}
func (m *mockPatcher) GetEntry(ctx context.Context, id string) (*models.FeedEntry, error) {
return &models.FeedEntry{ID: id}, nil
}
func TestJobEngineExecution(t *testing.T) {
tempDBPath := filepath.Join(t.TempDir(), "job_test.db")
database, err := db.InitDB(tempDBPath)
if err != nil {
t.Fatalf("failed to init db: %v", err)
}
defer func() { _ = database.Close() }()
bus := events.NewEventBus()
patcher := &mockPatcher{}
engine := NewJobEngine(database, bus, patcher, 5)
var executedCount int32
engine.RegisterHandler("test_task", func(ctx context.Context, job models.Job) (*models.PatchEntryRequest, map[string]interface{}, error) {
atomic.AddInt32(&executedCount, 1)
read := true
return &models.PatchEntryRequest{IsRead: &read}, map[string]interface{}{"done": true}, nil
})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
engine.Start(ctx)
// Enqueue test job
job := models.Job{
Type: "test_task",
Payload: map[string]interface{}{"entry_id": "e_100"},
Status: "pending",
IdempotencyKey: "test_key_1",
RunAt: time.Now().UTC(),
}
created, err := database.CreateJob(job)
if err != nil {
t.Fatalf("failed to create job: %v", err)
}
// Wait for engine to claim and execute
time.Sleep(1 * time.Second)
finalJob, err := database.GetJob(created.ID)
if err != nil {
t.Fatalf("failed to get job: %v", err)
}
if finalJob.Status != "completed" {
t.Errorf("expected job status 'completed', got '%s'", finalJob.Status)
}
if atomic.LoadInt32(&executedCount) != 1 {
t.Errorf("expected executedCount 1, got %d", atomic.LoadInt32(&executedCount))
}
if atomic.LoadInt32(&patcher.patchedCount) != 1 {
t.Errorf("expected patcher to be called once, got %d", atomic.LoadInt32(&patcher.patchedCount))
}
engine.Stop()
_ = database.Close()
}
+1 -1
View File
@@ -3,7 +3,7 @@
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/logo.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, viewport-fit=cover" />
<meta name="theme-color" content="#0f172a" />
<meta name="description" content="RSSPipe - RSS Driven Information Gateway" />
<link rel="manifest" href="/manifest.json" />
+50 -1
View File
@@ -306,9 +306,57 @@ export const App: React.FC = () => {
}
};
// Connect Real-Time SSE Stream for live state sync across tabs and async background jobs
useEffect(() => {
if (!currentUser) return;
const disconnect = api.connectEventStream((eventType, data) => {
if (eventType === 'job_completed' || eventType === 'post_updated' || eventType === 'variant_updated') {
const updatedEntry = data?.entry || data;
if (updatedEntry && updatedEntry.id) {
setEntries((prev) =>
prev.map((e) =>
e.id === updatedEntry.id
? {
...e,
...updatedEntry,
variants: { ...(e.variants || {}), ...(updatedEntry.variants || {}) },
}
: e
)
);
}
} else if (eventType === 'post_created') {
loadSidebarData();
} else if (eventType === 'preferences_updated') {
if (data && typeof data === 'object') {
if (data.auto_read !== undefined) {
localStorage.setItem('rsspipe_auto_read_scroll', data.auto_read ? 'true' : 'false');
}
if (data.view_preference) {
setViewPreference(data.view_preference);
localStorage.setItem('rsspipe_view_preference', data.view_preference);
}
}
}
});
return () => disconnect();
}, [currentUser, loadSidebarData]);
// Update Single Entry Action
const handleUpdateEntry = (updated: FeedEntry) => {
setEntries((prev) => prev.map((e) => (e.id === updated.id ? updated : e)));
setEntries((prev) =>
prev.map((e) =>
e.id === updated.id
? {
...e,
...updated,
variants: { ...(e.variants || {}), ...(updated.variants || {}) },
}
: e
)
);
loadSidebarData();
};
@@ -500,6 +548,7 @@ export const App: React.FC = () => {
onMarkAllRead={handleMarkAllRead}
onUpdateEntry={handleUpdateEntry}
stats={stats}
totalEntries={totalEntries}
onOpenSettings={() => setViewMode('settings')}
onOpenAddFeed={() => setShowAddFeedModal(true)}
onPlayAudio={handlePlayAudio}
@@ -23,6 +23,8 @@ interface DesktopArticleListProps {
onToggleStar: (entry: FeedEntry, e: React.MouseEvent) => void;
feeds?: Feed[];
categories?: Category[];
stats?: import('../../types').Stats | null;
totalEntries?: number;
onFilterChange?: (newFilter: Partial<FilterState>) => void;
}
@@ -52,6 +54,8 @@ export const DesktopArticleList: React.FC<DesktopArticleListProps> = ({
onToggleStar,
feeds = [],
categories = [],
stats,
totalEntries = 0,
onFilterChange,
}) => {
const getFilterTitle = () => {
@@ -70,7 +74,35 @@ export const DesktopArticleList: React.FC<DesktopArticleListProps> = ({
return 'Inbox';
};
const unreadCount = entries.filter((e) => !e.is_read).length;
// Calculate accurate unread and total counts from feeds/categories/stats
const getFilterCounts = () => {
if (filter.feedId) {
const activeFeed = feeds.find((f) => f.id === filter.feedId);
return {
unread: activeFeed?.unread_count || 0,
total: activeFeed?.total_count || totalEntries || 0,
};
}
if (filter.categoryId) {
const catFeeds = feeds.filter((f) => f.category_id === filter.categoryId);
const unread = catFeeds.reduce((acc, f) => acc + (f.unread_count || 0), 0);
const total = catFeeds.reduce((acc, f) => acc + (f.total_count || 0), 0);
return { unread, total: total || totalEntries || 0 };
}
if (filter.view === 'starred') {
return { unread: 0, total: stats?.total_starred || totalEntries || 0 };
}
if (filter.view === 'all') {
return { unread: stats?.total_unread || 0, total: totalEntries || 0 };
}
if (filter.view === 'shared') {
return { unread: 0, total: stats?.total_shared || totalEntries || 0 };
}
// 'inbox'
return { unread: stats?.total_unread || 0, total: stats?.total_unread || totalEntries || 0 };
};
const { unread: trueUnreadCount, total: trueTotalCount } = getFilterCounts();
return (
<div className="desktop-list-pane" style={{ width: `${width}px` }}>
@@ -80,7 +112,21 @@ export const DesktopArticleList: React.FC<DesktopArticleListProps> = ({
<span className="list-title-text" title={getFilterTitle()}>
{getFilterTitle()}
</span>
{unreadCount > 0 && <span className="badge badge-unread">{unreadCount}</span>}
{filter.view === 'inbox' ? (
trueUnreadCount > 0 && <span className="badge badge-unread">{trueUnreadCount}</span>
) : filter.view === 'all' ? (
trueTotalCount > 0 && (
<span className="badge badge-unread" style={{ background: 'rgba(148, 163, 184, 0.15)', color: 'var(--text-secondary)' }}>
{trueTotalCount}
</span>
)
) : filter.view === 'starred' ? (
trueTotalCount > 0 && (
<span className="badge badge-unread" style={{ background: 'rgba(234, 179, 8, 0.15)', color: 'var(--accent-star)' }}>
{trueTotalCount}
</span>
)
) : null}
</div>
<div style={{ display: 'flex', alignItems: 'center', gap: '0.4rem' }}>
@@ -104,7 +150,7 @@ export const DesktopArticleList: React.FC<DesktopArticleListProps> = ({
</div>
)}
{unreadCount > 0 && (
{trueUnreadCount > 0 && (
<button
className="btn-icon"
onClick={onMarkAllRead}
+5 -1
View File
@@ -226,7 +226,11 @@ export const DesktopReader: React.FC<DesktopReaderProps> = ({
setIsSharing(true);
try {
const res = await api.toggleShare(entry.id);
onUpdateEntry?.(res.entry);
const updatedEntry = {
...res.entry,
share_uuid: res.share_uuid ?? null,
};
onUpdateEntry?.(updatedEntry);
if (res.shared && res.share_uuid) {
const shareUrl = `${window.location.origin}/share/${res.share_uuid}`;
navigator.clipboard.writeText(shareUrl);
@@ -18,6 +18,7 @@ interface ThreePaneViewProps {
onMarkAllRead: () => void;
onUpdateEntry?: (entry: FeedEntry) => void;
stats: Stats | null;
totalEntries?: number;
onOpenSettings: () => void;
onOpenAddFeed: () => void;
onPlayAudio?: (entry: FeedEntry, streamUrl: string) => void;
@@ -38,6 +39,7 @@ export const ThreePaneView: React.FC<ThreePaneViewProps> = ({
onMarkAllRead,
onUpdateEntry,
stats,
totalEntries = 0,
onOpenSettings,
onOpenAddFeed,
onPlayAudio,
@@ -115,6 +117,8 @@ export const ThreePaneView: React.FC<ThreePaneViewProps> = ({
onToggleStar={onToggleStar}
feeds={feeds}
categories={categories}
stats={stats}
totalEntries={totalEntries}
onFilterChange={onFilterChange}
/>
@@ -201,7 +201,11 @@ export const MobileArticleSheet: React.FC<MobileArticleSheetProps> = ({
try {
const res = await api.toggleShare(entry.id);
if (res && res.entry) {
onUpdateEntry?.(res.entry);
const updatedEntry = {
...res.entry,
share_uuid: res.share_uuid ?? null,
};
onUpdateEntry?.(updatedEntry);
}
if (res && res.shared && res.share_uuid) {
const shareUrl = `${window.location.origin}/share/${res.share_uuid}`;
+74
View File
@@ -306,8 +306,82 @@ export const api = {
return request<void>(`/search/history${qParam}`, {
method: 'DELETE',
});
},
// Patch Entry (Atomic transition mutation)
async patchEntry(
id: string,
patch: {
is_read?: boolean;
is_starred?: boolean;
metadata?: Record<string, any>;
variants?: Record<string, any>;
}
): Promise<{ entry: FeedEntry; diff: any }> {
return request<{ entry: FeedEntry; diff: any }>(`/entries/${id}`, {
method: 'PATCH',
body: JSON.stringify(patch),
});
},
// Background Task Queue & Jobs
async listJobs(params?: { type?: string; status?: string; limit?: number; offset?: number }): Promise<{ jobs: any[]; limit: number; offset: number }> {
const query = new URLSearchParams();
if (params?.type) query.set('type', params.type);
if (params?.status) query.set('status', params.status);
if (params?.limit) query.set('limit', params.limit.toString());
if (params?.offset) query.set('offset', params.offset.toString());
return request(`/jobs?${query.toString()}`);
},
async getJob(id: string): Promise<any> {
return request<any>(`/jobs/${id}`);
},
async cancelJob(id: string): Promise<{ message: string }> {
return request<{ message: string }>(`/jobs/${id}/cancel`, { method: 'POST' });
},
async retryJob(id: string): Promise<{ message: string }> {
return request<{ message: string }>(`/jobs/${id}/retry`, { method: 'POST' });
},
// Real-Time Server-Sent Events Stream
connectEventStream(onEvent: (eventType: string, data: any) => void): () => void {
const eventSource = new EventSource(`${API_BASE}/events/stream`);
const eventNames = [
'connected',
'ping',
'preferences_updated',
'job_progress',
'job_completed',
'job_failed',
'post_created',
'post_updated',
];
eventNames.forEach((name) => {
eventSource.addEventListener(name, (e: MessageEvent) => {
try {
const data = JSON.parse(e.data);
onEvent(name, data);
} catch {
onEvent(name, e.data);
}
});
});
eventSource.onerror = (err) => {
console.warn('SSE connection error:', err);
};
return () => {
eventSource.close();
};
}
};
+3 -2
View File
@@ -64,8 +64,9 @@
/* Layout Dimensions */
--header-height: 50px;
--bottom-nav-height: 72px;
--docked-audio-height: 56px;
--safe-area-bottom: env(safe-area-inset-bottom, 0px);
--bottom-nav-height: calc(50px + env(safe-area-inset-bottom, 0px));
--docked-audio-height: 54px;
--sidebar-min-width: 200px;
--sidebar-default-width: 250px;
--list-min-width: 300px;
+7 -5
View File
@@ -375,11 +375,12 @@
-webkit-backdrop-filter: blur(20px);
border-top: 1px solid var(--border-subtle);
display: flex;
align-items: flex-start;
align-items: center;
justify-content: space-around;
padding: 6px 0.5rem calc(16px + env(safe-area-inset-bottom, 0px));
padding: 4px 0.5rem calc(var(--safe-area-bottom) + 2px);
z-index: 50;
user-select: none;
box-sizing: border-box;
}
.bottom-nav-item {
@@ -467,7 +468,7 @@
overflow-y: auto;
overflow-x: hidden;
-webkit-overflow-scrolling: touch;
padding: 1.15rem 1rem calc(65px + env(safe-area-inset-bottom, 0px));
padding: 1.15rem 1rem calc(56px + var(--safe-area-bottom));
width: 100%;
max-width: 100vw;
box-sizing: border-box;
@@ -479,8 +480,8 @@
bottom: 0;
left: 0;
right: 0;
height: calc(52px + env(safe-area-inset-bottom, 0px));
padding: 0 0.85rem env(safe-area-inset-bottom, 0px);
height: calc(48px + var(--safe-area-bottom));
padding: 4px 0.85rem calc(var(--safe-area-bottom) + 2px);
background: var(--bg-docked);
backdrop-filter: blur(20px);
-webkit-backdrop-filter: blur(20px);
@@ -490,6 +491,7 @@
justify-content: space-between;
z-index: 110;
user-select: none;
box-sizing: border-box;
}
.mobile-sheet-bottom-left {
+1 -1
View File
@@ -67,7 +67,7 @@ export interface FeedEntry {
published_at: string;
is_read: boolean;
is_starred: boolean;
share_uuid?: string;
share_uuid?: string | null;
variants?: Record<string, EntryVariant>;
metadata?: Record<string, any>;
created_at: string;