The log had 6.45M rows, of which 434 were actual actions: every mail x
every non-matching rule wrote "Keine Übereinstimmung" (5.3M rows, ~1 GB),
plus "keine neuen Mails" per rule every 30s and a "Poll gestartet" row.
Free-text searches took 50-70s.
- Routine entries are only written with LOG_VERBOSE=true; the poll
summary is logged only when there were new mails, actions or errors.
"Why didn't my rule match" is now answered by the rule tester.
- The scheduler commits every 25 evaluated mails and after each rule,
since the dropped log calls used to be the implicit commit points.
- Migration rebuilds filter_logs without the routine rows (live:
6.45M -> ~38k) via copy instead of DELETE.
- /api/logs: keyset pagination (before_id) instead of OFFSET, count
capped at 1000, since/until range, multi-level filter, per-field
filters; ordner: also matches the move target stored in details.
- New log page: one search field with shortcuts (von:, betreff:,
regel:, ordner:, text:), live search with debounce and abort, active
filters as removable chips, click on rule/sender/folder in a row to
filter, time range (default 7 days), level quick filter, expandable
row details, "Ältere laden", filters kept in the URL.
- Disable Pico's colour transition on segmented buttons (they briefly
flashed in the primary colour on load).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rule tester (POST /api/filters/test): evaluates an unsaved rule
read-only against the newest matching mails in the source folder,
using an IMAP SEARCH on the first text condition to narrow candidates.
Shows matches, near misses with the failing condition, and whether
any candidate had an attachment at all (would have caught the
Klarmobil and it-moench misconfigurations immediately).
- Per-rule match_count / last_match_at (new migration), incremented by
the scheduler on every match and shown in the list.
- PATCH /api/filters/{id}/enabled toggles a rule without resetting its
processed state (a full PUT re-evaluates every mail for that rule).
- New filter list: cards with enable switch, priority, condition and
action chips, match stats, and warnings (display name in from/to,
"am (exakt)" date, duplicates, empty values, invalid regex, move
without folder). Quick filters for active/inactive/warnings/never
matched; copy-as-template; remembered account selection.
- Edit dialog: wider layout, live warnings, integrated test panel,
sensible defaults for new rules (from + date after Jan 1 + move).
- Light global refresh: nav with active item, page background, cards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- IMAP connections now use a 120s socket timeout. Without it the poll
hung for 5 days in logout() on a silently dropped connection; every
later poll was skipped ("max instances reached"), so nothing got
sorted and newly created rules were never evaluated.
- Header decoding joined decode_header() parts with " " although the
whitespace is already part of the unencoded chunks, producing
"IT-Mönch <noreply@…>" (two spaces) — the it-moench rule could never
match. Join without separator and collapse whitespace/folded lines.
- Rule text matching (contains/exact) normalizes whitespace on both
sides, so stray or doubled spaces in rule values no longer break them.
- Pseudo charsets (unknown-8bit, …) try UTF-8 before latin-1, fixing
mojibake like "nächster" in raw 8-bit headers.
- Fetch via BODY.PEEK[] instead of RFC822, which set \Seen on every
evaluated mail. Payload is located by shape in the FETCH response,
fixing "'int' object has no attribute 'decode'".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live poll had been silently dying every 30s with
"Fehler beim Abrufen von Ordner 'INBOX' — got more than 1000000 bytes"
Root cause: Python's imaplib hard-caps a single response at 1 MB. With
~150k UIDs in INBOX, "UID SEARCH ALL" returns a longer line than that,
imaplib throws, get_all_uids() returns [], the poll reports "0 Mail(s)
im Ordner, 0 Treffer" and nothing gets sorted. Bump _MAXLINE to 100 MB
(covers mailboxes with millions of mails).
Filter search haystack now also includes the German condition/action
labels ("am (exakt)", "ab / nach", "verschieben in Ordner", …) and a
"stop" token, so the user can spot mis-typed date rules by typing
"am" and see any on_date rules rather than having to know the
internal enum names.
Also audited every rule via SSH: no on_date/date-exact rules exist —
all 32 rules use `date after '2026-01-01'` correctly. cps datensysteme
still has its known duplicate conditions (harmless).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cps datensysteme rule (from cps-datensysteme.de + subject "Rechnung"
+ has_attachment true + date after) was failing on real invoices from
CPS ORMS. Their PDFs arrive without a Content-Disposition header — the
filename lives only on the Content-Type parameter — so _has_attachment
returned False and the rule never matched.
Extended the check to also count a part as an attachment when it has a
filename() and its content-type is neither text/* nor multipart/*.
That catches PDFs, Office and image attachments regardless of whether
the sender set Content-Disposition, without treating body parts as
attachments. Verified against four synthetic cases including the exact
CPS ORMS shape.
Same helper is now applied to non-multipart mails too, in case an entire
mail is a single attached file (rare but possible).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-rule batch was sliced from all_uids in server order (oldest
UID first), so with a big backlog the scheduler chewed through years
of unmatched newsletters before reaching the current invoices the
user was actually waiting on. Sort new_uids descending by int (with a
plain reverse() fallback for non-numeric UIDs) so each poll picks the
newest 500 first. Old backlog still gets worked off in the background.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fix: fetch_mail crashed on MIME pseudo-encodings like 'unknown-8bit',
'x-unknown', '8bit'. New _safe_decode helper maps those (and any
LookupError from unknown codecs) to latin-1, which never fails on
8-bit input. Used in _extract_body and _decode_header_value.
- Consequence of the crash: scheduler marked the affected UIDs as
processed to avoid retry loops, so those mails never got sorted
even after the underlying issue was fixable.
- feat: POST /api/filters/retry-failed drops the ProcessedMail markers
for UIDs that appear in "Fehler beim Abrufen" error logs, so the
next poll re-evaluates them. Reachable via a button in the log UI
("Fehlgeschlagene neu einlesen"), optionally scoped per account.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fix: Starlette 1.0 changed TemplateResponse signature — request is now
the first positional argument, not nested inside the context dict.
All HTML routes returned 500 (unhashable dict in jinja2 template cache)
after the image rebuild picked up the new starlette version.
- fix: processed_mails (1.7M rows in production: 1 account × 12 rules ×
141k inbox mails) made backup export hit 558 MB / 90s. Moved to opt-in
checkbox in the UI alongside the logs option, default off.
- yield_per for streaming the processed_mails query when included.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fix: imap folder names with spaces (RFC3501 quoting in move/create/select)
- fix: move only deletes source after COPY + Message-ID verification in target
- fix: backup endpoint hung on sqlite write locks — enable WAL + busy_timeout
- perf: indexes on filter_logs(created_at, level, account_id+created_at) for
fast log queries on millions of rows
- feat: optional "logs mit sichern" checkbox in backup export, restore on import
- UI: backup download uses fetch+blob with error display instead of location.href
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
duffyduckandClaude Opus 5.5
Stefan Hacker