The existing daily cleanup never effectively ran: it was an interval
job counted from container start (reset on every deploy), it called
asyncio.get_event_loop() from a worker thread, and even when deleting,
SQLite never returns freed pages to the filesystem. Live DB: 4.1 GB,
of which 2.6 GB log and 0.6 GB unused pages.
New maintenance service, scheduled daily at MAINTENANCE_HOUR_UTC:30
and once 5 minutes after every start:
- delete log entries older than LOG_RETENTION_DAYS (30) and keep at
most LOG_MAX_ENTRIES (200k) newest ones
- delete processed_mails markers of deleted rules/accounts (FKs are
not enforced by SQLite)
- VACUUM when >25% of the file is unused, only if free disk space
covers the copy; never runs twice in parallel
GET/POST /api/logs/maintenance expose status and a manual run; the log
menu shows DB size, entry count and last run, with "Datenbank
aufräumen & verkleinern". Removed the dead cleanup_old_logs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The log had 6.45M rows, of which 434 were actual actions: every mail x
every non-matching rule wrote "Keine Übereinstimmung" (5.3M rows, ~1 GB),
plus "keine neuen Mails" per rule every 30s and a "Poll gestartet" row.
Free-text searches took 50-70s.
- Routine entries are only written with LOG_VERBOSE=true; the poll
summary is logged only when there were new mails, actions or errors.
"Why didn't my rule match" is now answered by the rule tester.
- The scheduler commits every 25 evaluated mails and after each rule,
since the dropped log calls used to be the implicit commit points.
- Migration rebuilds filter_logs without the routine rows (live:
6.45M -> ~38k) via copy instead of DELETE.
- /api/logs: keyset pagination (before_id) instead of OFFSET, count
capped at 1000, since/until range, multi-level filter, per-field
filters; ordner: also matches the move target stored in details.
- New log page: one search field with shortcuts (von:, betreff:,
regel:, ordner:, text:), live search with debounce and abort, active
filters as removable chips, click on rule/sender/folder in a row to
filter, time range (default 7 days), level quick filter, expandable
row details, "Ältere laden", filters kept in the URL.
- Disable Pico's colour transition on segmented buttons (they briefly
flashed in the primary colour on load).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rule tester (POST /api/filters/test): evaluates an unsaved rule
read-only against the newest matching mails in the source folder,
using an IMAP SEARCH on the first text condition to narrow candidates.
Shows matches, near misses with the failing condition, and whether
any candidate had an attachment at all (would have caught the
Klarmobil and it-moench misconfigurations immediately).
- Per-rule match_count / last_match_at (new migration), incremented by
the scheduler on every match and shown in the list.
- PATCH /api/filters/{id}/enabled toggles a rule without resetting its
processed state (a full PUT re-evaluates every mail for that rule).
- New filter list: cards with enable switch, priority, condition and
action chips, match stats, and warnings (display name in from/to,
"am (exakt)" date, duplicates, empty values, invalid regex, move
without folder). Quick filters for active/inactive/warnings/never
matched; copy-as-template; remembered account selection.
- Edit dialog: wider layout, live warnings, integrated test panel,
sensible defaults for new rules (from + date after Jan 1 + move).
- Light global refresh: nav with active item, page background, cards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/filters/reset-all-processed drops and recreates the
processed_mails table — sekundenschnell auch bei Millionen Zeilen
via DROP+CREATE (same pattern as the fast clear-logs). Safe because
already-moved mails are no longer in the source folder, so a fresh
re-evaluation cannot re-move them, only pick up ones that were
skipped due to a rule bug (e.g. the has_attachment fix).
Button "Alle Regeln zurücksetzen" next to "Neue Regel" on the filter
list, with a confirmation dialog and result feedback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fix: "Log leeren" without account filter used a plain DELETE over
millions of rows. That held the sqlite write lock long enough that
the scheduler's own log writes started failing with
"database is locked", and the request itself eventually 500'd back
as HTML "Internal Server Error" — which the frontend then tried to
JSON.parse, producing the popup the user reported. Now uses
DROP+CREATE for full clears (near-instant) and wal_checkpoint to
reclaim space; per-account clear still uses a normal DELETE.
- Raise sqlite busy_timeout from 10s to 60s so a legitimately slow
writer no longer starves smaller ones.
- retry-failed: single bulk DELETE with mail_uid IN (...) per account
instead of one query per uid.
- Both endpoints now return proper HTTPException(500, ...) with detail
instead of letting the exception bubble as plain text.
- Frontend: parseResponse() reads text first, then tries JSON. Both
clearLogs() and retryFailed() surface the actual server message
instead of crashing on non-JSON responses.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fix: fetch_mail crashed on MIME pseudo-encodings like 'unknown-8bit',
'x-unknown', '8bit'. New _safe_decode helper maps those (and any
LookupError from unknown codecs) to latin-1, which never fails on
8-bit input. Used in _extract_body and _decode_header_value.
- Consequence of the crash: scheduler marked the affected UIDs as
processed to avoid retry loops, so those mails never got sorted
even after the underlying issue was fixable.
- feat: POST /api/filters/retry-failed drops the ProcessedMail markers
for UIDs that appear in "Fehler beim Abrufen" error logs, so the
next poll re-evaluates them. Reachable via a button in the log UI
("Fehlgeschlagene neu einlesen"), optionally scoped per account.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New filter rules no longer default to priority 100. Server picks
max(priority)+10 per account when client sends priority=None (or 10
if the account has no rules yet), and the UI pre-fills the input
with the next value instead of a fixed 100. Avoids ambiguous ordering
when multiple rules end up with the same priority.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fix: Starlette 1.0 changed TemplateResponse signature — request is now
the first positional argument, not nested inside the context dict.
All HTML routes returned 500 (unhashable dict in jinja2 template cache)
after the image rebuild picked up the new starlette version.
- fix: processed_mails (1.7M rows in production: 1 account × 12 rules ×
141k inbox mails) made backup export hit 558 MB / 90s. Moved to opt-in
checkbox in the UI alongside the logs option, default off.
- yield_per for streaming the processed_mails query when included.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fix: imap folder names with spaces (RFC3501 quoting in move/create/select)
- fix: move only deletes source after COPY + Message-ID verification in target
- fix: backup endpoint hung on sqlite write locks — enable WAL + busy_timeout
- perf: indexes on filter_logs(created_at, level, account_id+created_at) for
fast log queries on millions of rows
- feat: optional "logs mit sichern" checkbox in backup export, restore on import
- UI: backup download uses fetch+blob with error display instead of location.href
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
duffyduckandClaude Opus 5.5
Stefan Hacker