I wanted to share a small workflow tool I built for my own upgrade planning and get your thoughts on whether this could be useful for others in the community.
Background & Challenge
When preparing to upgrade an IPFire instance between Core Updates (e.g., sequentially planning from Core 200 through 203), I found myself trying to get a clear picture of issues reported or updated after a specific Core Update was released.
Looking at Bugzilla:
The Version field in Bugzilla is structured as major releases (e.g., 2), while specific Core Updates (like Core Update 203) are tracked dynamically within bug summaries, comments, or git commits rather than as discrete search dropdown metadata.
When evaluating an upgrade, knowing which bugs were created or changed since a specific release blog post date helps gauge potential regression risks or known workarounds before deploying.
What the Script Does
I wrote a lightweight Bash script (fetch-cu-bugs.sh) to automate this correlation locally on a workstation:
Date Resolution: It fetches the official IPFire blog RSS feed (/blog/feed.xml) and extracts the exact publication date for "Core Update N released".
Bugzilla Query: Using the Bugzilla CSV API, it queries:
Bugs created (chfield=[Bug creation]) since that release date.
Bugs modified/changed (chfieldfrom=...) since that release date.
Structured Output: It exports a clean TSV file containing the bug ID, product, component, status, summary, and last changed date.
Optional workflow: I feed this structured output into a local tool or LLM to quickly parse summaries and highlight whether an open bug applies to a component I currently use (e.g., Knot Resolver, WIO, Samba).
Example Output (Core Update 203)
Running ./fetch-cu-bugs.sh -c 203 generates a report like this:
# Core Update : 203
# Release date : 2026-07-20
# Query timestamp: 2026-08-08 12:00:00 UTC
TYPE BUG_ID PRODUCT COMPONENT STATUS SUMMARY
CREATED 14035 IPFire wio NEW General::ipcidr2msk removed...
CREATED 14034 IPFire knot-resolver NEW NXDOMAIN queried 3-4x...
CHANGED 13979 IPFire knot-resolver CLOSED Rethink Safe Search
Question for the Community
Is this correlation (linking blog release timestamps directly to subsequent Bugzilla activity) something others in the community find helpful when planning maintenance windows?
If there is interest, I would be happy to publish the script repository for anyone who wants to use or refine it.
Script does exactly that by using as input date the date of each Core Update versions extracted from blog posts. Script then uses bugzilla official API schema (posted here Bugzilla Schema for Versions 5.0 to 5.2) to make a snapshot of the bugs opened after a certain core version was launcged - the csv is then fed to an AI agent that checks each bug against current ipfire usage and determine the risk/impact of executing the upgrade.
The result of AI analysis against the CSV produced by the script + knowledge information of each ipfire machine (how I use it - what services are there and how critical are those services for me) looks like this:
data/known_breakage.tsv
Source: Bugzilla entries confirmed via the IPFire community forum and direct Bugzilla reads.
Last refreshed: 2026-08-08 using utilities/fetch-cu-bugs.sh -c 203.
DNS resolution fails for OpenVPN and WireGuard road-warrior clients; kresd wildcard bind (0.0.0.0) causes wrong source IP on UDP replies; workaround: add specific listen IP to /etc/knot-resolver/config.yaml; PoC patch submitted upstream
NXDOMAIN queried 3-4 times triggers infinite randomised-case NS query flood; drives CPU to 100%; upstream hub blocks ALL traffic until kresd is restarted; workaround: watchdog script monitoring DNS packet rate
General::ipcidr2msk and GetDyndnsRedIP removed/relocated in CU203 Perl functions; WIO 1.3.2-19 breaks on start; patch under review by Adolf Belka (attachment 1744)
That CU203 will never be deployed on any of my machines - wait for future versions and hope (prey) that problems will be solved w/o new ones arise and produce severe impact on services delivered by each IPFire machine
And here is the AI agent monitoring the forum (and saw my reply!) has to add:
The Human layer is where the risk becomes **personal**.
The AI sees the bug list and the machine profile — installed addons, services, how critical each one is (as per Human inputs). It produces an impact table. Human's job is to read that table and answer one question: "can I accept this failure mode on this box"?
For CU203 on Human machines the answer was immediate:
Both boxes run OpenVPN road-warrior and WireGuard for an undisclosed number of Road Warriors clients. Bug #14030 means DNS breaks for every VPN client the moment kresd starts misrouting UDP replies. That is not a degraded service — it is a dead service. No workaround.
Bug #14034 means one NXDOMAIN storm drives the upstream hub to block all traffic.
Bug #14024 has no workaround at all and needs an LMDB upstream fix. Timeline: unknown.
Three bugs, all knot-resolver, all with no confirmed fix date. The decision should writes itself.
What the AI cannot do is assign tolerance to an outage window or weigh the value of a specific service to the people depending on it. It flags severity and workarounds — Human decide whether the workaround is acceptable given who is on the other side of the connection.
The tool (`fetch-cu-bugs`) is used at every new Core Update cycle and results fed to AI to evaluate the remaining unresolved bugs and their impact to imfrastructure.
If CU204 ships with those bugs closed, the report generated by AI changes and Human makes the the decision. Until then both AI & Human recommendation is **hold** - this is stored in a custom pakfire-manual utility that is used by both Human and AI agent to execute upgrades only if the risk assessment indicate acceptable losses in existing services.
And here is how AI is using the tool to assess each IPFire instance I have: all below are from the test machine
/usr/local/bin/pakfire-planner.sh 1 2>&1
--- Phase 1: local state ---
Installed core: 202
State written to /var/run/pakfire-manual/state.tsv
Available core: 203
Upgrade steps required: 203
Pending add-on changes:
guardian 27 -> 28
htop 23 -> 25
iperf3 8 -> 9
/usr/local/bin/pakfire-planner.sh 1 2>&1
--- Phase 2: per-step upgrade plan ---
=== Core Update 203 ===
ADD-ONS (source: blog matrix):
dnsdist 2.0.6 [BLOG]
ntfs-3g 2026.2.25 [BLOG]
postfix 3.11.3 [BLOG]
rsync 3.4.3 [BLOG]
samba 4.24.2 [BLOG]
spice 0.16.0 [BLOG]
spice-protocol 0.14.5 [BLOG]
tmux 3.6b [BLOG]
tshark 4.6.6 [BLOG]
strongswan 6.0.7 [BLOG] # CVE-2026-47895; core security component not an add-on; included for completeness
intel-microcode 20260512 [BLOG] # INTEL-SA-01420; core component included for completeness
KNOWN BREAKAGE (source: Bugzilla):
[crash] Bug #14024 knot-resolver NEW assertion crash when >=16 host definitions share one IP; threshold confirmed at 16; no workaround; fix requires LMDB upstream
[crash] Bug #14037 knot-resolver NEW kresd fails to start when DNS upstream protocol set to TLS but no TLS hostnames configured; Lua stack traceback on tls_client init; workaround: switch DNS protocol back to UDP in WUI
[major] Bug #14030 knot-resolver ASSI DNS resolution fails for OpenVPN and WireGuard road-warrior clients; kresd wildcard bind (0.0.0.0) causes wrong source IP on UDP replies; clients reject responses; workaround: add specific listen IP to /etc/knot-resolver/config.yaml; PoC patch submitted to upstream Knot project
[major] Bug #14034 knot-resolver NEW infinite randomised-case NS query flood on NXDOMAIN queried 3-4 times; saturates upstream resolver; upstream hub blocks ALL traffic until kresd is restarted; workaround: watchdog script that monitors DNS packet rate and restarts kresd; no upstream fix yet
[major] Bug #14031 knot-resolver ASSI WUI lease table and /var/lib/knot-resolver/leases.db show different entries; kresd writes leases independently from WUI; workaround: none; assigned to Michael Tremer
[major] Bug #13973 dhcpd.conf - lease hook not migrated; update.sh replaces unbound-dhcp-leases-client with dhcp-lease but does not rewrite argument format; NAME= expected; HOSTNAME= required; workaround: WUI DHCP page untick and re-tick interfaces
[major] Bug #14035 wio ASSI General::ipcidr2msk and GetDyndnsRedIP removed or relocated in CU203 Perl functions; WIO 1.3.2-19 breaks on start; patch by reporter (attachment 1744) under review by Adolf Belka; no fixed release yet
*** DEPLOY HOLD: severity=crash bug open for core 203 ***
Phase 5 will refuse this step unless --override-hold is passed.
Full capacity of the of the tool:
/usr/local/bin/pakfire-planner.sh -h
pakfire-planner.sh v1.1.0: IPFire 2.29 x86_64 controlled-upgrade planner
USAGE
pakfire-planner.sh [OPTIONS]
pakfire-planner.sh [OPTIONS] <phase>
PHASES
1 local-state Read pakfire and system state; write run-state file
2 per-step-plan Show per-core-step upgrade plan with known breakage
3 mirror-survey Test configured mirrors for geo-policy and reachability
4 fetch-verify Download and GPG-verify intermediate core upgrade files
5 deploy Apply ONE core step with snapshot, guards, and logging
6 handoff Write HANDOFF.md for follow-up analysis
OPTIONS
-D, --debug Enable verbose output (or export DEBUG=1)
--dry-run Parse and plan only; no disk writes, no pakfire calls
-k, --keep Preserve temporary artefacts on exit
--override-hold Required for phase 5 when a severity=crash bug is open
-v, --version Print version and exit
-h, --help Print this text and exit
NOTES
Phases 1-3 are non-destructive and safe to re-run.
Phase 5 requires typed confirmation of the core number.
Phase 5 refuses crash-severity bugs unless --override-hold is passed.
Never reboots. Manual reboot is required after phase 5.
Upgrade steps must be applied one at a time: 200->201->202->203.
WUI is work in progress: in the control panel all data accumulated by several helpers are presented - what mirror contains the necessary files as per run-file plan, and if those are accessible (strict exit control - several geoIP bloks + large number of ASNs/CIDR blocked via Release firewall-local v1.8.0-ui · horace-michael/firewall-local · GitHub add-on)
To be added the run-plan and remaining list of problems if upgrade is carried on against recommended “hold”.
Agree. Helper scripts only collect bugzilla data at a certain point in time and store them on local files (I believedata is at least 1 day old?! Or maybe 1,5 days old…)
Once the WUI will be complete the planner will call each helper script (to update each bug status - goal is to see them all CLOSED)
But, because a bug will eventually get closed in future versions, the planner script will have a memory file where cu203 can never be last version (i.e. put in production) and will continue to indicate the “hold” output as long as CU203 will be last version available on upstream IPFire servers.
So the pakfire-planner will look for CU204 and (!) that known CU203 bugs listed in its history file are declared CLOSED.
Then and only then the “hold” status might change, unless the AI agent does not inject new bugs in the planner list for CU204 (or CU203 - the helper scripts are run by AI agent only when I request it to do it and then AI agent execute search on forum to incorporate forum feedback along with bugzilla entries inside known_breakage.tsv consumed by pakfire-planner.
As I said: still work in progress - the architecture is there, some helpers are there, WUI is definitelly not ready and SoP is not ever written: how I should use the tool as risk evaluator for any future version to be deployed.
