Add-on backup issue relate to WebGUI backup/restore

Hello,
I tested the restore configuration and it failed.
My steps:

  1. New HW , installed core 189
  2. Downloaded rpz-beta-0.1.16-16.ipfire.tar
  3. Uncompressed it inside /opt/pakfire/tmp/
  4. Installed it NAME=rpz ./install.sh
  5. Restored backup from an older box using the WUI
  6. The folder /var/ipfire/backup/addons/backup contains the rpz.ipf created on older machine
 ls -lth rpz.ipf
-rw-r--r-- 1 root root 337 Dec  3 13:00 rpz.ipf

But the /etc/unbound/zonefiles/ does not contain all zonefiles like source machine

ls -lth /etc/unbound/zonefiles/
total 0
-rw-r--r-- 1 nobody nobody 0 Nov 19 08:46 allow.rpz

What am I doing wrong?
Thank you!

@hjkl,

Find the backup file from the restored box. And enter this command to view all of the files in the old rpz.ipf file:

tar -tvf rpz.ipf

Please post the results. FYI - I am looking for the allow.rpz file.


EDIT:
If the allow.rpz file is included in the tar -tvf rpz.ipf list, then you can:

Restore the rpz.ipf file

go to menu System > Backup and scroll down to Restore

and then click Choose File and pick the rpz.ipf from your desktop computer

This will RESTORE ALL OF THE RPZ FILES and that may not be what you want.

View content of the allow.rpz file

This will view the contents of the allow.rpz file and then you can cut & paste to the WebGUI.

tar -axf rpz.ipf etc/unbound/zonefiles/allow.rpz -O

Archive content:

\etc\unbound\local.d\

\etc\unbound\zonefiles\

This is what I meant by “I used WUI to restore”

None of above listed files were restored…

I checked the content of allow.rpz file from archive I got from old machine: it is containing the correct whitelist.
The problem is NOT the archive. The archive contains everything.
The problem is the Restore process via WUI: none of the files from the archive were restored…

Late edit: does the add-on have a build in “restore”?
I mean: I can copy the rpz.ipf inside /var/ipfire/backup/addons/ and then use the RPZ add-on command to restore that?
The WUI is not doing its job!

I didn’t do the restore process, yet. Shortage of time :slight_smile:
But looking inside the sources the WUI restore addon process should succeed.
I’l try it later this day.

There is no restore command. @jon uses the processes for IPFire addons.

1 Like

There is no RPZ add-on restore command since we use the IPFire backup/restore system.

But we can do a command line restore. Can you do the following steps to help troubleshoot?

  1. Please enter this and post results:

    ls -al /var/ipfire/backup/addons/backup
    
  2. Find the backup file from the restored box and copy it to the /var/ipfire/backup/addons/backup directory (just like you mentioned above). Did it copy OK?

    • Please enter this and post results:
    ls -al /var/ipfire/backup/addons/backup
    
  3. Run the IPFire restore and post results:

    backupctrl restoreaddon rpz.ipf
    

Your RPZ files should be listed and installed.

PS - Sorry for asking the detail questions. Something odd is happening and it doesn’t make sense!

1 Like

To reproduce the issue, I did

  • save rpz.ipf to my desktop,
  • backupctrl restore rpz.ipf,
    make a new backup,
    save the new backup to rpz1.ipf on my desktop
    compare rpz.ipf and rpz.ipf1
    result: equal
  • addon restore in WUI ( rpz.ipf from my desktop )
    make a new backup
    save the new backup to rpz2.ipf on my desktop
    compare rpz.ipf, rpz1.ipf, rpz2.ipf
    result: equal

This means backupctrl and the WUI rpz.cgi do what they should. As @jon supposed there must be some oddity on your system.

just show the output of
tar tvzf /var/ipfire/backup/addons/backup/rpz.ipf

A size of 337 seems a bit small for the contents you listed ( I suppose the output of 7zip on your dektop ).

337 bytes is too small.

This looks like a backup of a brand new rpz add-on install with no configuration and no custom lists.

I did a new install of CU 189 and installed the rpz add-on (rpz-beta-0.1.16-16.ipfire). And then I did a backup and I see this:

[root@ipfireAPU ~] # ls -lth /var/ipfire/backup/addons/backup/rpz.ipf
-rw-r--r-- 1 root root 381 Dec  5 12:54 /var/ipfire/backup/addons/backup/rpz.ipf

. . . very similar file size to your rpz.ipf file above.

These are the items in my test rpz.ipf file:

[root@ipfireAPU ~] # tar -tvf /var/ipfire/backup/addons/backup/rpz.ipf
-rw-r--r-- nobody/nobody   301 2024-11-19 00:46 etc/unbound/local.d/00-rpz.conf
-rw-r--r-- nobody/nobody     0 2024-11-19 00:46 etc/unbound/zonefiles/allow.rpz
-rw-r--r-- nobody/nobody     0 2024-11-19 00:46 var/ipfire/dns/rpz/allowlist
-rw-r--r-- nobody/nobody     0 2024-11-19 00:46 var/ipfire/dns/rpz/blocklist
-rw-r--r-- nobody/nobody     0 2024-12-05 12:54 var/ipfire/dns/rpz/customlists.conf
-rw-r--r-- nobody/nobody     0 2024-12-05 12:54 var/ipfire/dns/rpz/zonefiles.conf

rpz.ipf.tar (381 Bytes)

1 Like

Hi Jon,
I started the process from scratch on another box with core 189 where there is no rpz addon.

Status of the box: no trace of the rpz addon, or the rpz.ipf backup

ls -lth /var/ipfire/dns/r* ; ls -lth /etc/unbound/z*; ls -lth /etc/unbound/local.d/ ;  ls -lth /var/ipfire/backup/addons/backup/
ls: cannot access '/var/ipfire/dns/r*': No such file or directory
ls: cannot access '/etc/unbound/z*': No such file or directory
total 0
total 0

Then I installed the exact same rpz version as the one from the “production” machine: rpz-beta-0.1.10-10.ipfire.zip

I know this is not the latest version but I wanted to replicate my exact steps:

curl -k https://community.ipfire.org/uploads/short-url/q1sbCBTynBmAKDhoMThogHfV5Gk.zip -o /opt/pakfire/tmp/rpz-beta-0.1.10-10.ipfire.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15553    0 15553    0     0  62973      0 --:--:-- --:--:-- --:--:-- 63741

ls -lth /opt/pakfire/tmp/rpz-beta-0.1.10-10.ipfire.zip
-rw-r--r-- 1 root root 16K Dec  9 13:56 /opt/pakfire/tmp/rpz-beta-0.1.10-10.ipfire.zip

What is inside the file /opt/pakfire/tmp/rpz-beta-0.1.10-10.ipfire.zip

tar tvzf /opt/pakfire/tmp/rpz-beta-0.1.10-10.ipfire.zip
gzip: stdin has more than one entry--rest ignored
drwxr-xr-x root/root         0 2024-10-15 09:08 ./
-rw-r--r-- root/root     12728 2024-10-15 09:08 files.tar.xz
-rw-r--r-- root/root       427 2024-10-15 09:08 ROOTFILES
-rwxr--r-- root/root      2340 2024-10-15 09:08 update.sh
-rwxr--r-- root/root      2024 2024-10-15 09:08 uninstall.sh
-rwxr--r-- root/root      1945 2024-10-15 09:08 install.sh
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Unpack /opt/pakfire/tmp/rpz-beta-0.1.10-10.ipfire.zip:

tar -zxvf rpz-beta-0.1.10-10.ipfire.zip
gzip: stdin has more than one entry--rest ignored
./
files.tar.xz
ROOTFILES
update.sh
uninstall.sh
install.sh
tar: Child returned status 2
tar: Error is not recoverable: exiting now


ls -lth
total 48K
-rw-r--r-- 1 root root  16K Dec  9 13:56 rpz-beta-0.1.10-10.ipfire.zip
-rw-r--r-- 1 root root  13K Oct 15 09:08 files.tar.xz
-rw-r--r-- 1 root root  427 Oct 15 09:08 ROOTFILES
-rwxr--r-- 1 root root 1.9K Oct 15 09:08 install.sh
-rwxr--r-- 1 root root 2.0K Oct 15 09:08 uninstall.sh
-rwxr--r-- 1 root root 2.3K Oct 15 09:08 update.sh

Install RPZ 1.10:

 NAME=rpz ./install.sh
Extracting files...
./
var/
var/ipfire/
var/ipfire/menu.d/
var/ipfire/menu.d/EX-rpz.menu
var/ipfire/dns/
var/ipfire/dns/rpz/
var/ipfire/dns/rpz/blocklist
var/ipfire/dns/rpz/allowlist
var/ipfire/backup/
var/ipfire/backup/addons/
var/ipfire/backup/addons/includes/
var/ipfire/backup/addons/includes/rpz
var/ipfire/addon-lang/
var/ipfire/addon-lang/rpz.en.pl
var/ipfire/addon-lang/rpz.de.pl
usr/
usr/sbin/
usr/sbin/rpz-sleep
usr/sbin/rpz-metrics
usr/sbin/rpz-make
usr/sbin/rpz-functions
usr/sbin/rpz-config
srv/
srv/web/
srv/web/ipfire/
srv/web/ipfire/cgi-bin/
srv/web/ipfire/cgi-bin/rpz.cgi
etc/
etc/unbound/
etc/unbound/zonefiles/
etc/unbound/zonefiles/allow.rpz
etc/unbound/local.d/
etc/unbound/local.d/00-rpz.conf
...Finished.
ownership of '/var/ipfire/dns/rpz/allowlist' retained as nobody:nobody
ownership of '/var/ipfire/dns/rpz/blocklist' retained as nobody:nobody
ownership of '/var/ipfire/dns/rpz' retained as nobody:nobody
ownership of '/etc/unbound/zonefiles/allow.rpz' retained as nobody:nobody
ownership of '/etc/unbound/zonefiles' retained as nobody:nobody
ownership of '/etc/unbound/local.d/00-rpz.conf' retained as nobody:nobody
changed ownership of '/etc/unbound/local.d' from root:root to nobody:nobody
Stopping Unbound DNS Proxy...                                                                                                                                                       [  OK  ]
Starting Unbound DNS Proxy...                                                                                                                                                       [  OK  ]

Status post install:

ls -lth /var/ipfire/dns/r* ; ls -lth /etc/unbound/z*; ls -lth /etc/unbound/local.d/ ;  ls -lth /var/ipfire/backup/addons/backup/
total 0
-rw-r--r-- 1 nobody nobody 0 Oct 15 08:54 allowlist
-rw-r--r-- 1 nobody nobody 0 Oct 15 08:54 blocklist
total 0
-rw-r--r-- 1 nobody nobody 0 Oct 15 08:54 allow.rpz
total 4.0K
-rw-r--r-- 1 nobody nobody 301 Oct 15 08:54 00-rpz.conf
total 0

Using cgi-bin/backup.cgi to restore the below rpz.ipf file copied from “production” system that is also running 1.10 RPZ addon

ls -lth ~/rpz.ipf
-rw-r--r-- 1 root root 2.6K Dec  3 16:31 /root/rpz.ipf

tar tvzf ~/rpz.ipf
-rw-r--r-- nobody/nobody   295 2024-11-22 21:53 etc/unbound/local.d/00-rpz.conf
-rw-r--r-- nobody/nobody   280 2024-08-04 19:23 etc/unbound/local.d/BlockDOH_jpgpi250.rpz.conf
-rw-r--r-- nobody/nobody   261 2024-08-04 21:37 etc/unbound/local.d/HagheziDOH.rpz.conf
-rw-r--r-- nobody/nobody   298 2024-08-04 21:20 etc/unbound/local.d/HagheziHuaweiNative.rpz.conf
-rw-r--r-- nobody/nobody   290 2024-08-04 21:28 etc/unbound/local.d/HagheziLgTVWebOS.rpz.conf
-rw-r--r-- nobody/nobody   269 2024-08-04 21:31 etc/unbound/local.d/HagheziMulti.rpz.conf
-rw-r--r-- nobody/nobody   281 2024-08-04 21:30 etc/unbound/local.d/HagheziPopupAds.rpz.conf
-rw-r--r-- nobody/nobody   298 2024-08-04 21:29 etc/unbound/local.d/HagheziXiaomiNative.rpz.conf
-rw-r--r-- nobody/nobody   237 2024-08-04 19:17 etc/unbound/local.d/SSLblAbuseCh.rpz.conf
-rw-r--r-- nobody/nobody   240 2024-08-04 19:20 etc/unbound/local.d/URLHausAbuseCh.rpz.conf
-rw-r--r-- nobody/nobody   239 2024-11-22 21:28 etc/unbound/local.d/block.rpz.conf
-rw-r--r-- nobody/nobody  1662 2024-11-22 21:53 etc/unbound/zonefiles/allow.rpz
-rw-r--r-- nobody/nobody  1665 2024-11-22 21:28 etc/unbound/zonefiles/block.rpz
-rw-r--r-- nobody/nobody   653 2024-11-22 21:53 var/ipfire/dns/rpz/allowlist
-rw-r--r-- nobody/nobody   905 2024-11-22 21:27 var/ipfire/dns/rpz/blocklist
-rw-r--r-- nobody/nobody  1481 2024-10-25 13:01 var/ipfire/dns/rpz/customlists.conf
-rw-r--r-- nobody/nobody     0 2024-10-18 09:20 var/ipfire/dns/rpz/zonefiles.conf

Nothing changed on the system as result of using the above page to restore rpz.ipf

ls -lth /var/ipfire/dns/r* ; ls -lth /etc/unbound/z*; ls -lth /etc/unbound/local.d/ ;  ls -lth /var/ipfire/backup/addons/backup/
total 0
-rw-r--r-- 1 nobody nobody 0 Oct 15 08:54 allowlist
-rw-r--r-- 1 nobody nobody 0 Oct 15 08:54 blocklist
total 0
-rw-r--r-- 1 nobody nobody 0 Oct 15 08:54 allow.rpz
total 4.0K
-rw-r--r-- 1 nobody nobody 301 Oct 15 08:54 00-rpz.conf
total 0

Then I used the process posted by @bbitsch: backupctrl restore rpz.ipf

 backupctrl restore ~/rpz.ipf
etc/unbound/local.d/00-rpz.conf
etc/unbound/local.d/BlockDOH_jpgpi250.rpz.conf
etc/unbound/local.d/HagheziDOH.rpz.conf
etc/unbound/local.d/HagheziHuaweiNative.rpz.conf
etc/unbound/local.d/HagheziLgTVWebOS.rpz.conf
etc/unbound/local.d/HagheziMulti.rpz.conf
etc/unbound/local.d/HagheziPopupAds.rpz.conf
etc/unbound/local.d/HagheziXiaomiNative.rpz.conf
etc/unbound/local.d/SSLblAbuseCh.rpz.conf
etc/unbound/local.d/URLHausAbuseCh.rpz.conf
etc/unbound/local.d/block.rpz.conf
etc/unbound/zonefiles/allow.rpz
etc/unbound/zonefiles/block.rpz
var/ipfire/dns/rpz/allowlist
var/ipfire/dns/rpz/blocklist
var/ipfire/dns/rpz/customlists.conf
var/ipfire/dns/rpz/zonefiles.conf
groupadd: group 'dhcpcd' already exists
useradd: user 'dhcpcd' already exists

That process worked:

ls -lth /var/ipfire/dns/r* ; ls -lth /etc/unbound/z*; ls -lth /etc/unbound/local.d/ ;  ls -lth /var/ipfire/backup/addons/backup/
total 12K
-rw-r--r-- 1 nobody nobody  653 Nov 22 21:53 allowlist
-rw-r--r-- 1 nobody nobody  905 Nov 22 21:27 blocklist
-rw-r--r-- 1 nobody nobody 1.5K Oct 25 13:01 customlists.conf
-rw-r--r-- 1 nobody nobody    0 Oct 18 09:20 zonefiles.conf
total 8.0K
-rw-r--r-- 1 nobody nobody 1.7K Nov 22 21:53 allow.rpz
-rw-r--r-- 1 nobody nobody 1.7K Nov 22 21:28 block.rpz
total 44K
-rw-r--r-- 1 nobody nobody 295 Nov 22 21:53 00-rpz.conf
-rw-r--r-- 1 nobody nobody 239 Nov 22 21:28 block.rpz.conf
-rw-r--r-- 1 nobody nobody 261 Aug  4 21:37 HagheziDOH.rpz.conf
-rw-r--r-- 1 nobody nobody 269 Aug  4 21:31 HagheziMulti.rpz.conf
-rw-r--r-- 1 nobody nobody 281 Aug  4 21:30 HagheziPopupAds.rpz.conf
-rw-r--r-- 1 nobody nobody 298 Aug  4 21:29 HagheziXiaomiNative.rpz.conf
-rw-r--r-- 1 nobody nobody 290 Aug  4 21:28 HagheziLgTVWebOS.rpz.conf
-rw-r--r-- 1 nobody nobody 298 Aug  4 21:20 HagheziHuaweiNative.rpz.conf
-rw-r--r-- 1 nobody nobody 280 Aug  4 19:23 BlockDOH_jpgpi250.rpz.conf
-rw-r--r-- 1 nobody nobody 240 Aug  4 19:20 URLHausAbuseCh.rpz.conf
-rw-r--r-- 1 nobody nobody 237 Aug  4 19:17 SSLblAbuseCh.rpz.conf

My conclusion: the cgi-bin/backup.cgi has a problem in restoring addons → using command in the shell does the restore ok.

Thank you!

PS: also from Graphical user interface I can see that restore was OK: cgi-bin/rpz.cgi:

To reproduce exactly my steps I would suggest to remove all rpz.ipf from the addon backup folder - my system was always “brand new” (never had RPZ) but your had the rpz.ipf on the addon backup folder.

This is the only difference I could spot between my process and your process…

1 Like

Bingo!

I found the bug.

  • backup.cgi transfers the file to /tmp
  • and calls backupctrl <addon>
  • backupctrl / /var/ipfire/backup/bin/backup.pl should move the file
    from /tmp to /var/ipfire/backup/addons/backup/ ( if existent )
  • the file in /var/ipfire/backup/addons/backup/ is extracted

The test of existence in backup.pl is done by
if [ -d "/tmp/${name}.ipf" ]; then
which should be
if [ -f "/tmp/${name}.ipf" ]; then

I’ll post a bug.

Thx for finding this problem.

1 Like

That has been there for a very long time then. That existence test first went into backup.pl in Core Update 125, a bit over 6 years ago and no one has noticed that the restores for the addons don’t work, me included.

EDIT:
I took a vm that had no addons and then I installed bacula, which I use and I have backups for.

The first couple of times I tried to do the restore, nothing got changed so duplicating what has been mentioned here.

I then shut the vm down, then after a bit I thought what happens if I do a full restore so I restarted the vm, restored my IPFire which also provided all the addon backups listed in the backup WUI page and then I did a restore of the same bacula addon backup file via the WUI and this time the bacula config file was updated correctly.

So it is not as simple as it looked. Under some conditions the addon backups are restore correctly.

2 Likes

Digging a bit in the git repo, shows that is ‘only’ about 3 years old.
But nevertheless shame on us all devs and users. :hot_face:

Please see the edit I made to my post.

Can you tell me the date for the change you mention because the specific test you showed

I found in backup.pl in CU125 for the first time using a search by the midpoint after I tested back in CU120 and found that the line did not exist there.

Seaching a bit more, I think you are right. CU 130 contains the false line.

Your observations about restores are reproducible.
If the backup repo contains a file yet, this restored. Nobody has checked whether changes in the new file are transferred.

Let’s say it is a bit odd.

Okay, I have now shown that if I start with a system with no bacula and then I install it, it has a default bacula-fd.conf file.

Choosing a bacula.ipf file to restore from on the WUI page results in no update. Tried it several times and nothing was updated.

I then cleared the bacula.ipf file from the /tmp/ directory and then did a main backup restore first. That put a restore.ipf file into /tmp/ but the bacula-fd.conf file was still not correct.

I then selected the same bacula.ipf file for restore and now the bacula-fd.conf file was correctly restored.

So the sequence of things affects whether an addon restore works or not.

I did not used system restore prior to restore RPZ - I only used addon restore targeting RPZ config.

Hope it helps.

Now I can start again to test latest RPZ - now I know how to restore an older config if will ever be needed.

Thank you guys - you are the best!

Could the difference be using:

backupctrl restore <filename>

vs

backupctrl restoreaddon <addon>

I only tested with the backupctrl restoreaddon <addon>

I didn’t check the restore action.
The error is in the restoreaddon action. See restore_addon_backup function in
/var/ipfire/backup/bin/backup.pl ( a bash file :wink: ).

@bonnietwin @bbitsch - just to summarize…

Is the above issue in Post #235 is a backup bug (or a tentative bug)?

In bugzilla:
https://bugzilla.ipfire.org/show_bug.cgi?id=13799

1 Like