Upgrade to 181 hangs - no WebGUI

grisu127 · 25 November 2023 11:36

same on my APU4d4. pakfire seems to have finished the update but wui/apache not running.
logon via console was possible
error_log in /var/log/httpd from yesterday, when i ran the update, shows the following

[root@myipfire01 httpd]# grep -i "Nov 24" error_log`
[Fri Nov 24 16:41:14.178891 2023] [cgid:error] [pid 8275:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Fri Nov 24 16:41:14.181527 2023] [cgid:error] [pid 13136:tid 139353600812736] [client 10.0.3.2:65372] End of script output before headers: speed.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:15.326646 2023] [cgid:error] [pid 8278:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Fri Nov 24 16:41:15.328501 2023] [cgid:error] [pid 13136:tid 139353368032960] [client 10.0.3.2:65370] End of script output before headers: pakfire.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:16.352154 2023] [cgid:error] [pid 8280:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Fri Nov 24 16:41:16.353809 2023] [cgid:error] [pid 13136:tid 139353326069440] [client 10.0.3.2:65435] End of script output before headers: speed.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:17.349347 2023] [cgid:error] [pid 8282:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Fri Nov 24 16:41:17.351136 2023] [cgid:error] [pid 13136:tid 139353477072576] [client 10.0.3.2:65436] End of script output before headers: pakfire.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:18.349898 2023] [cgid:error] [pid 8283:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Fri Nov 24 16:41:18.351747 2023] [cgid:error] [pid 13136:tid 139353351247552] [client 10.0.3.2:65437] End of script output before headers: speed.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:19.352410 2023] [cgid:error] [pid 8284:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Fri Nov 24 16:41:19.354259 2023] [cgid:error] [pid 13136:tid 139353493857984] [client 10.0.3.2:65438] End of script output before headers: pakfire.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:20.356040 2023] [cgid:error] [pid 8285:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Fri Nov 24 16:41:20.357793 2023] [cgid:error] [pid 13136:tid 139353485465280] [client 10.0.3.2:65439] End of script output before headers: speed.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:21.355948 2023] [cgid:error] [pid 8286:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Fri Nov 24 16:41:21.357649 2023] [cgid:error] [pid 13136:tid 139353342854848] [client 10.0.3.2:65440] End of script output before headers: pakfire.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:22.350373 2023] [cgid:error] [pid 8287:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Fri Nov 24 16:41:22.352185 2023] [cgid:error] [pid 13136:tid 139353460287168] [client 10.0.3.2:65441] End of script output before headers: speed.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:23.364872 2023] [cgid:error] [pid 8288:tid 139353617625280] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Fri Nov 24 16:41:23.367028 2023] [cgid:error] [pid 13136:tid 139353368032960] [client 10.0.3.2:65442] End of script output before headers: pakfire.cgi, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:25.328701 2023] [core:error] [pid 13136:tid 139353451894464] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:27.335766 2023] [core:error] [pid 13136:tid 139353477072576] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:27.339631 2023] [core:error] [pid 13136:tid 139353334462144] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:28.360168 2023] [core:error] [pid 13136:tid 139353359640256] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:29.360042 2023] [core:error] [pid 13136:tid 139353493857984] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:30.360187 2023] [core:error] [pid 13136:tid 139353502250688] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:31.361788 2023] [core:error] [pid 13136:tid 139353485465280] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:32.363708 2023] [core:error] [pid 13136:tid 139353584027328] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:33.364559 2023] [core:error] [pid 13136:tid 139353342854848] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:34.364631 2023] [core:error] [pid 13136:tid 139353600812736] (13)Permission denied: [client 10.0.3.2:65443] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://myipfire01.mydomain.lan:444/
[Fri Nov 24 16:41:44.230454 2023] [mpm_event:notice] [pid 5039:tid 139353617625280] AH00494: SIGHUP received.  Attempting to restart

then restarted apache

[root@myipfire01 httpd]# /etc/init.d/apache restart
Restarting Apache daemon...
httpd not running, trying to start                                     [  OK  ]

an the error_log from today

[root@myipfire01 httpd]# grep -i "Nov 25" error_log
[Sat Nov 25 12:08:32.799300 2023] [ssl:warn] [pid 1431:tid 125097346740416] AH01909: myipfire01.mydomain.lan:444:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 25 12:08:32.833089 2023] [ssl:warn] [pid 1432:tid 125097346740416] AH01909: myipfire01.mydomain.lan:444:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 25 12:08:32.835612 2023] [core:warn] [pid 1432:tid 125097346740416] AH00098: pid file /var/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Sat Nov 25 12:08:32.840688 2023] [mpm_event:notice] [pid 1432:tid 125097346740416] AH00489: Apache/2.4.58 (Unix) OpenSSL/3.1.4 configured -- resuming normal operations
[Sat Nov 25 12:08:32.840942 2023] [core:notice] [pid 1432:tid 125097346740416] AH00094: Command line: '/usr/sbin/httpd'

maybe this can help to clarify the situation

bonnietwin · 25 November 2023 14:29

The IPFire devs have also had one of their firewalls for the IPFire Infrastructure have the same problem. This allowed a closer look at what was happening.

https://lists.ipfire.org/hyperkitty/list/development@lists.ipfire.org/thread/6TWUUXSCC2ZIDDPOFXZMYDSC2R4BQAJF/

What was reported was:-

Yes, I can confirm this. This happened on one of the firewalls in our infrastructure.

It seems that the runtime linker cache isn’t up to date at the time Apache is being restarted, and therefore it simply fails to restart.

However, ldconfig is executed before restarting apache, so I am not sure why this is happening.

It is however safe to reboot the affected systems after this point and Apache will come back up just fine.

So what is happening is now understood but not why it is happening.

cfusco · 26 November 2023 07:37

I just updated and this problem did not happened to me. My machine is an APU2.

bonnietwin · 26 November 2023 08:19

It hasn’t happened to the majority of people so there is probably some sort of delay in the linker process for some machines so that apache fails to find the required linked libraries when it tries to start.

jon · 26 November 2023 16:50

During my update from CU 180 to CU 181 (on a PC Engine APU4D4), the WebGUI stalled for a long time after:

10:20:18	pakfire:	PAKFIRE UPGR: core-upgrade-181: Upgrading files and running post-upgrading scri pts...

.
And I could no longer refresh the browser to see the System Logs > Pakfire log:

Note: I was watching the message log and the Pakfire WebGUI and the System Logs > Pakfire at the same time (a belts and suspenders approach!!)

For me it seems like the Apache restarted and after a minute or so it finally came back. Maybe something is timing out during the restart??

Apache finally started working, but then it stopped, waited, and then started again. Keep in mind I am seeing this from my browser and it displaying the WebGUI and then not displaying.

There are no references to Apache in the message logs. There are messages in the httpd > error log

[Sun Nov 26 09:55:17.255791 2023] [core:error] [pid 31949:tid 124702664144576] (70007)The timeout specified has expired: [client 192.168.60.212:54734] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:05:02.933632 2023] [core:error] [pid 31949:tid 124702680929984] (70007)The timeout specified has expired: [client 192.168.60.212:57651] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:30.279349 2023] [cgid:error] [pid 30884:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:30.284639 2023] [cgid:error] [pid 31949:tid 124702697715392] [client 192.168.60.212:49399] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:30.810333 2023] [cgid:error] [pid 30885:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:30.812930 2023] [cgid:error] [pid 31949:tid 124702798362304] [client 192.168.60.212:49946] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:31.077069 2023] [cgid:error] [pid 30886:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:31.081893 2023] [cgid:error] [pid 31949:tid 124702664144576] [client 192.168.60.212:50016] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:32.337705 2023] [cgid:error] [pid 30887:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:32.341779 2023] [cgid:error] [pid 31949:tid 124702781576896] [client 192.168.60.212:50024] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:32.849874 2023] [cgid:error] [pid 30892:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:32.852377 2023] [cgid:error] [pid 31949:tid 124702806755008] [client 192.168.60.212:50031] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:33.787879 2023] [cgid:error] [pid 30893:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:33.790362 2023] [cgid:error] [pid 31949:tid 124702831933120] [client 192.168.60.212:49320] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:34.028659 2023] [cgid:error] [pid 30894:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:34.031572 2023] [cgid:error] [pid 31949:tid 124702815147712] [client 192.168.60.212:50040] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:34.279465 2023] [cgid:error] [pid 30895:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:34.282126 2023] [cgid:error] [pid 31949:tid 124702672537280] [client 192.168.60.212:50041] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:34.882606 2023] [cgid:error] [pid 30896:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:34.884984 2023] [cgid:error] [pid 31949:tid 124702680929984] [client 192.168.60.212:50044] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:34.973131 2023] [cgid:error] [pid 30897:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:34.975695 2023] [cgid:error] [pid 31949:tid 124702913717952] [client 192.168.60.212:50045] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:35.559319 2023] [cgid:error] [pid 30899:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/logs.cgi/log.dat' failed
[Sun Nov 26 10:24:35.561932 2023] [cgid:error] [pid 31949:tid 124702697715392] [client 192.168.60.212:50052] End of script output before headers: log.dat, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:36.074747 2023] [cgid:error] [pid 30900:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:36.077146 2023] [cgid:error] [pid 31949:tid 124702664144576] [client 192.168.60.212:50056] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:36.100123 2023] [cgid:error] [pid 30901:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:36.108909 2023] [cgid:error] [pid 31949:tid 124702689322688] [client 192.168.60.212:50054] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:36.256246 2023] [cgid:error] [pid 30902:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:36.261312 2023] [cgid:error] [pid 31949:tid 124702781576896] [client 192.168.60.212:50060] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:37.286849 2023] [cgid:error] [pid 30903:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:37.289302 2023] [cgid:error] [pid 31949:tid 124702789969600] [client 192.168.60.212:50065] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:38.275040 2023] [cgid:error] [pid 30904:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:38.277696 2023] [cgid:error] [pid 31949:tid 124702655751872] [client 192.168.60.212:50073] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:38.386826 2023] [cgid:error] [pid 30905:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:38.389262 2023] [cgid:error] [pid 31949:tid 124702806755008] [client 192.168.60.212:50075] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:38.593321 2023] [cgid:error] [pid 30906:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:38.595911 2023] [cgid:error] [pid 31949:tid 124702672537280] [client 192.168.60.212:50076] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:39.509988 2023] [cgid:error] [pid 30907:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:39.514402 2023] [cgid:error] [pid 31949:tid 124702555104960] [client 192.168.60.212:50084] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:40.317722 2023] [cgid:error] [pid 30908:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:40.321114 2023] [cgid:error] [pid 31949:tid 124702680929984] [client 192.168.60.212:50090] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:40.598625 2023] [cgid:error] [pid 30909:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:40.606648 2023] [cgid:error] [pid 31949:tid 124702697715392] [client 192.168.60.212:50095] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:40.648084 2023] [cgid:error] [pid 30910:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:40.651488 2023] [cgid:error] [pid 31949:tid 124702913717952] [client 192.168.60.212:50094] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:41.722693 2023] [cgid:error] [pid 30911:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:41.726014 2023] [cgid:error] [pid 31949:tid 124702664144576] [client 192.168.60.212:50106] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:42.384684 2023] [cgid:error] [pid 30912:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:42.386831 2023] [cgid:error] [pid 31949:tid 124702689322688] [client 192.168.60.212:50113] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:42.601752 2023] [cgid:error] [pid 30913:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:42.604590 2023] [cgid:error] [pid 31949:tid 124702789969600] [client 192.168.60.212:50115] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:42.853641 2023] [cgid:error] [pid 30914:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:42.855652 2023] [cgid:error] [pid 31949:tid 124702655751872] [client 192.168.60.212:50117] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:44.112308 2023] [cgid:error] [pid 30915:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:44.114450 2023] [cgid:error] [pid 31949:tid 124702815147712] [client 192.168.60.212:50126] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:44.278682 2023] [cgid:error] [pid 30916:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/speed.cgi' failed
[Sun Nov 26 10:24:44.281159 2023] [cgid:error] [pid 31949:tid 124702672537280] [client 192.168.60.212:50130] End of script output before headers: speed.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:45.339707 2023] [cgid:error] [pid 30917:tid 124702930366656] (13)Permission denied: AH01241: exec of '/srv/web/ipfire/cgi-bin/pakfire.cgi' failed
[Sun Nov 26 10:24:45.342195 2023] [cgid:error] [pid 31949:tid 124702555104960] [client 192.168.60.212:50135] End of script output before headers: pakfire.cgi, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:46.494860 2023] [core:error] [pid 31949:tid 124702697715392] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:46.583137 2023] [core:error] [pid 31949:tid 124702913717952] (13)Permission denied: [client 192.168.60.212:50145] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:47.627709 2023] [core:error] [pid 31949:tid 124702823540416] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:48.166043 2023] [core:error] [pid 31949:tid 124702798362304] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:48.635594 2023] [core:error] [pid 31949:tid 124702664144576] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:49.641482 2023] [core:error] [pid 31949:tid 124702689322688] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:50.167054 2023] [core:error] [pid 31949:tid 124702563497664] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:50.653669 2023] [core:error] [pid 31949:tid 124702781576896] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:51.757655 2023] [core:error] [pid 31949:tid 124702789969600] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:52.167593 2023] [core:error] [pid 31949:tid 124702655751872] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:52.853145 2023] [core:error] [pid 31949:tid 124702806755008] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:53.858357 2023] [core:error] [pid 31949:tid 124702831933120] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:54.167573 2023] [core:error] [pid 31949:tid 124702815147712] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:54.807291 2023] [core:error] [pid 31949:tid 124702672537280] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:54.893635 2023] [core:error] [pid 31949:tid 124702555104960] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:55.899087 2023] [core:error] [pid 31949:tid 124702680929984] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:56.168198 2023] [core:error] [pid 31949:tid 124702697715392] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:56.904659 2023] [core:error] [pid 31949:tid 124702913717952] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:57.923912 2023] [core:error] [pid 31949:tid 124702823540416] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:58.168720 2023] [core:error] [pid 31949:tid 124702798362304] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:58.929592 2023] [core:error] [pid 31949:tid 124702664144576] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:24:59.935664 2023] [core:error] [pid 31949:tid 124702689322688] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:00.176601 2023] [core:error] [pid 31949:tid 124702563497664] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:00.941561 2023] [core:error] [pid 31949:tid 124702789969600] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:01.994318 2023] [core:error] [pid 31949:tid 124702655751872] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:02.170727 2023] [core:error] [pid 31949:tid 124702806755008] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:03.000844 2023] [core:error] [pid 31949:tid 124702831933120] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:04.007389 2023] [core:error] [pid 31949:tid 124702815147712] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:04.171105 2023] [core:error] [pid 31949:tid 124702672537280] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:05.040621 2023] [core:error] [pid 31949:tid 124702555104960] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:06.046836 2023] [core:error] [pid 31949:tid 124702680929984] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/pakfire.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:06.171916 2023] [core:error] [pid 31949:tid 124702697715392] (13)Permission denied: [client 192.168.60.212:50144] AH00035: access to /cgi-bin/speed.cgi denied (filesystem path '/srv/web') because search permissions are missing on a component of the path, referer: https://ipfire.localdomain:444/
[Sun Nov 26 10:25:19.398723 2023] [mpm_event:notice] [pid 3838:tid 124702930366656] AH00494: SIGHUP received.  Attempting to restart
[Sun Nov 26 10:25:20.785820 2023] [ssl:warn] [pid 3838:tid 124702930366656] AH01882: Init: this version of mod_ssl was compiled against a newer library (OpenSSL 3.1.4 24 Oct 2023, version currently loaded is OpenSSL 3.1.2 1 Aug 2023) - may result in undefined or erroneous behavior
[Sun Nov 26 10:25:20.807057 2023] [mpm_event:notice] [pid 3838:tid 124702930366656] AH00489: Apache/2.4.57 (Unix) OpenSSL/3.1.2 configured -- resuming normal operations
[Sun Nov 26 10:25:20.807202 2023] [core:notice] [pid 3838:tid 124702930366656] AH00094: Command line: '/usr/sbin/httpd'

After the Apache stops and starts the pakfire came back and gave me the An update requires a restart! message. And it looks like the 181 updated completed.

EDIT: there seems to me lots of permission denied errors in the above logs…

EDIT2: and an odd OpenSSL message at the bottom of log…

Ha! the OpenSSL message includes the statement “may result in undefined or erroneous behavior” and this definitely applies!

EDIT3: I saw there were updates to OpenSSL in CU 181 but I have no ideal if this is related to the original issue:

praful · 26 November 2023 23:39

I too found, after upgrading, that the Web GUI didn’t return. I didn’t diagnose it and left it for a day since internet access, samba, and VPN were working. The following day, when the Web GUI was still not working, I had to go to my IPFire device to logon and reboot it. I didn’t know if a reboot was required after the upgrade since the Web GUI didn’t return. i assumed it was and I didn’t know about starting apache.

I could provide logs but it looks like there are enough provided already. I appreciate these things slip through testing and this is only the second time in several years this has happened to me.

My concern is that if I’m away from home, say in another country, and I need to upgrade (which I can do via VPN) then can’t logon to the device, what do I do? My only option would be to ask someone to go to my home and do a hard reboot. I disable ssh by default.

For now, I’m going to add apache to my monit list. However, it would be nice if an IPFire device is never left in a state that prevents remote access. Perhaps when an upgrade is required, there is an optional checkbox to say that the device should be rebooted after the upgrade? You could even have a scheduled reboot after the upgrade. I’d prefer to take my chances with a reboot than to hope an update leaves the device remotely accessible.

cfusco · 27 November 2023 11:19

When I do an update and I am not onsite, before the update I connect by SSH by activating the option on WEB UI. Just in case something like this happens. I also never reboot unless I am on site or I am willing to risk a disconnection.

ms · 27 November 2023 11:28

This is indeed not optimal, but we currently only have very few people where the problem happens. Obviously it should not happen at all.

I just pushed a fix which will need a couple of hours to build:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=ece94c7edfa45b724ef05b57ff7a8a18c43fb99b

After the build is done, this problem should no longer arise at all.

ralph · 27 November 2023 13:14

I have got a weekly “scheduled action” -reboot- in the “Connection Scheduler”

praful · 27 November 2023 17:03

@ms I appreciate all that the ipfire team do in providing a world class firewall. These things will happen occasionally. I want to have a contingency plan to minimise the impact when these issues slip through the testing net.

@cfusco thanks - that’s a good idea. The only other time I couldn’t access the server remotely was when VPN stopped working after an upgrade a few years ago! I think it was a result of that that I installed monit to restart certain essential services. In this case I didn’t allow for apache not starting – although as an ex-developer, I should have known better

@ralph thanks - something like this occurred to me after I posted the message. I know I once considered scheduling a reboot but IPfire is so reliable, I’ve never required. However, I’ll reconsider. At the very least, when I’m away, I’ll schedule a restart for an hour (say) after the upgrade in case the system doesn’t come up and I can’t VPN/ssh (as per @cfusco idea) into the system.

I’ve also added apache to monit, which will now attempt to restart it. In case it helps others or someone has suggestions/comments, my monit restarts these services:

apache
apcupsd
dhcpd
nmbd
ntpd
openvpn
rsyncd
smbd
unbound
winbindd

Thanks again all for replies.

ms · 27 November 2023 17:27

A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more.

It is safe to just reboot the firewall a couple of minutes after the update, or you can run “/etc/init.d/apache start” on the console and we the web UI will be back immediately.

ip-mfg · 27 November 2023 18:53

same procedure for me.
The firewall works, but I can not connect to it.

ip-mfg · 28 November 2023 09:05

I am not an experienced user.
Is there another workaround as to use the terminal-cable?
If not what cable should I buy and where?

ms · 28 November 2023 09:58

just reboot…

ip-mfg · 28 November 2023 16:07

works.
relieved.
Thanks

adrb04 · 29 November 2023 00:16

On two APU boxes I had to start the apache daemon manually from the ssh console.
But on my newest mini PC apache worked fine after the pakfire upgrade finished.

coralex · 1 December 2023 05:35

Similar problem too , web interface not responding after updating , but main functions seems to be working now.

bonnietwin · 1 December 2023 09:51

This problem where the WUI hangs is different to the issue in this thread.

This thread is related to after the upgrade is successfully completed and the machine is then rebooted it is failing to reboot because it can’t find one of the partitions.

The problem with the WUI hanging is related to apache and is covered in this thread.
https://community.ipfire.org/t/upgrade-to-181-hangs-no-webgui/10634

A fix for this was pushed into the upgrade mirrors 4 days ago. Did you run your upgrade that had the WUI hang in the last 4 days. If yes then that indicates that something went wrong with the push of the fix.
With the fix pushed this problem of apache not restarting correctly should no longer exist.

EDIT:
I just set up a vm machine with Core Update 180 and then did the upgrade to 181.

The screen stopped changing and hung for a while. I then checked via the console and found that apache was not running.

I checked the apache initscript and it has the changes from the patch mentioned earlier so it looks like that patch fix has not solved whatever the problem was.

I the ran the initscript to start apache and immediately the WUI screen was updated again.

Checked the log for the Core Update 181 and it had

Stopping Apache daemon...
OK
Starting Apache daemon...
httpd (pid 6165) already running

So it thought for some reason that the apache server was running.

I will feed this into the IP{Fire dev mailing list.

tphz · 1 December 2023 12:04

For a test, could you check if the problem also occurs in a command-line upgrade?

Regards

bonnietwin · 1 December 2023 12:33

Yes it did, but with a different error message for when apache tried to restart.

Stopping Apache daemon...     [  OK  ]
Starting Apache daemon...
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:81
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:81
no listening sockets available, shutting down

Don’t believe the difference is related to the update from CLI vs WUI.
When CU181 was released, I upgraded two vm’s and my production physical system and all three upgraded without any problems of apache not restarting.

I suspect there is some timing related thing here.