No space left on /

err0r · 8 January 2025 11:02

Happy new year everyone!

I have a problem since I made an update from 189 to 190.

openvpn was not working anymore. I turned off the service and wanted to turn it on again. This does not worked.
I want to turn on location block again, because I turned it off for TOR. It was also not possible to turn it on again.

I did some research here and someone mentioned to take a look into /var/log/httpd/error_log:

[Tue Jan 07 20:53:25.478641 2025] [log_config:warn] [pid 3354:tid 3389] (28)No space left on device: [client 192.168.1.99:58340] AH00646: Error writing to /var/log/httpd/ssl_request_log, referer: https://ip.home:444/

df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.8G  4.0K  3.8G   1% /dev
shm             256M     0  256M   0% /dev/shm
tmpfs           3.8G  792K  3.8G   1% /run
/dev/sda4       116G  116G     0 100% /
efivarfs        192K   49K  139K  27% /sys/firmware/efi/efivars
/dev/sdb1       4.6T  3.1T  1.3T  72% /mnt/data
/dev/sda1       488M   61M  391M  14% /boot
/dev/sda2        32M  278K   32M   1% /boot/efi
/var/lock       8.0M   16K  8.0M   1% /var/lock

du -h --max-depth=1 / | sort -h
du: cannot access '/proc/27124/task/27124/fd/4': No such file or directory
du: cannot access '/proc/27124/task/27124/fdinfo/4': No such file or directory
du: cannot access '/proc/27124/fd/3': No such file or directory
du: cannot access '/proc/27124/fdinfo/3': No such file or directory
0       /proc
0       /sys
4.0K    /dev
16K     /lost+found
16K     /media
20K     /tmp
56K     /root
164K    /home
592K    /opt
792K    /run
4.9M    /srv
7.8M    /bin
12M     /sbin
26M     /etc
61M     /boot
953M    /lib
1.2G    /usr
5.1G    /var
3.2T    /
3.2T    /mnt

The OS is installed on an 128GB NVME.
In /mnt are three HDDs mounted.

internal ~4TB HDD
internal ~2TB HDD
external 4TB HDD (USB)

What can I do now? Why can du not list the size of /proc?

arne_f · 8 January 2025 11:26

because on /proc is a virtual filesytem from the kernel mounted that displays procces informations. On / it is an empty directory for the mountpoint. (same is for /sys)
Sounds like your filesystem information are corrupted. Try to reboot with filesystem check or boot a livesystem and run fsck -y /dev/xxx on this disk.

nickh · 8 January 2025 11:29

Don’t worry about /proc. I am not sure of the correct term but it is a virtual file system that does not really exist on any disk.

Concentrate on the OS disk root partition, /dev/sda4 and try and find where the space has gone using the du command, but you will need to understand the filesystem structure a bit more with something like findmnt. Your /usr is a bit big compared to mine (885MB), but your /var is large compared to mine (218MB), but not fatal.

Have you used any bind mounts? What is the contents of /etc/fstab

err0r · 8 January 2025 11:37

What is the contents of /etc/fstab?

cat /etc/fstab 
UUID=b4dc2398-5af7-42d3-8197-ca2d7d44e7ac /boot    auto defaults,nodev,noexec,nosuid 1 2
UUID=0949-1562 /boot/efi auto defaults   1 2
UUID=9ddc9b93-046d-416e-8823-922fac459943 swap     swap defaults,pri=1 0 0
UUID=f3a626be-f431-4026-9605-6a05628a53f2 /        auto defaults   1 1
shm /dev/shm tmpfs defaults,size=256M 0 0

Where should I search? When I calculate the sizes from du -h together, it will not fill the whole NVME.

bonnietwin · 8 January 2025 11:42

That is why @arne_f suggested

nickh · 8 January 2025 12:34

OK, I don’t know IPF here because I cannot see how your extra disks are mounted as it is not through /etc/fstab. How have you mounted them?

I asked about bind mounts (which you don’t have in fstab), because they can mask a load of space e.g. if you have a 128GB disk and 120GB of it in /var/log (all hypothetical) then you add another disk like /mnt/ano and create a folder there, /mnt/ano/log and bind mount /mnt/ano/log into /var/log, then you will see a your new disk space in /var/log but you will have recovered no space from the 128GB disk as you did not empty the original /var/log before bind mounting. In that case you’d need to unmount the bind mount delete the files (or move them to /mnt/ano/log if you wanted to keep them) and redo the bind mount. I believe the same will happen with a normal mount into a folder.

What does findmnt show?

err0r · 8 January 2025 12:56

There was an test VM and ISO in /var/lib/libvirt/images. I deleted them.

I have now a little more space. But df -h shows no available space.

df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.8G  4.0K  3.8G   1% /dev
shm             256M     0  256M   0% /dev/shm
tmpfs           3.8G  792K  3.8G   1% /run
/dev/sda4       116G  111G     0 100% /
efivarfs        192K   49K  139K  27% /sys/firmware/efi/efivars
/dev/sda1       488M   61M  391M  14% /boot
/dev/sda2        32M  278K   32M   1% /boot/efi
/var/lock       8.0M   16K  8.0M   1% /var/lock

findmnt
TARGET                        SOURCE    FSTYPE   OPTIONS
/                             /dev/sda4 ext4     rw,relatime
├─/proc                       proc      proc     rw,nosuid,nodev,noexec,relatime
├─/sys                        sysfs     sysfs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup            none      cgroup2  rw,relatime,favordynmods
│ └─/sys/firmware/efi/efivars efivarfs  efivarfs rw,relatime
├─/dev                        devtmpfs  devtmpfs rw,nosuid,noexec,size=3955768k,nr_inodes=988942,mode=755
│ ├─/dev/pts                  devpts    devpts   rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ └─/dev/shm                  tmpfs     tmpfs    rw,nosuid,nodev,noexec
│   └─/dev/shm                shm       tmpfs    rw,relatime,size=262144k
├─/run                        tmpfs     tmpfs    rw,nosuid,nodev,noexec,mode=755
├─/boot                       /dev/sda1 ext4     rw,nosuid,nodev,noexec,relatime
│ └─/boot/efi                 /dev/sda2 vfat     rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro
└─/var/lock                   /var/lock tmpfs    rw,nosuid,nodev,relatime,size=8192k

arne_f · 8 January 2025 13:07

As i have already written this looks like internal fs data like the free blocks table is corrupted. Reboot and run fsck.

dr_techno · 8 January 2025 13:42

I also want to ask if this installation of ipfire is running in a VM?

Because not only doing that causes the network throughput to diminish, most VMs put size limits to files and will remove blocks free when a file reaches this limit.

These are just a few reasons why router OS should never be used in a VM. No matter who makes it.

I do recommend running fsck like @arne_f to fix this and if you are running on bare metal (like you should be) then I would boot a live os like an ubuntu desktop install disk and exit out of that setup to a desk environment and run fsck again, then look at /dev/sda4 and look at the other mount point directories to make sure they are empty.

err0r · 8 January 2025 13:46

As i have already written this looks like internal fs data like the free blocks table is corrupted. Reboot and run fsck.

Will do this.

I also want to ask if this installation of ipfire is running in a VM?

No, it is installed on bare metal.

dr_techno · 8 January 2025 14:37

Then the rest is housekeeping.
While booted from a live usb, check to make sure nothing is in /tmp other than empty directories.

then remove all the files in /var/cache/pakfire
remove all .gz files from /var/log/httpd
remove all .gz files from /var/log
remove all files from /var/log/openvpn

chron housekeeping is on my list of things I want to address. Its good to have logs, but they only have a limited window of usefulness so a lot should be deleted after so many days.

bbitsch · 8 January 2025 14:57

Householding is done already by logrotate!

It can be discussed, how long logs are retained.
Standard in IPFire is ( /etc/logrotate.conf )

rotate every week
keep 52 weeks ( one year )

err0r · 14 January 2025 08:29

Sorry for the late response.

I tried to recover the system with a live-USB but this does not worked for me. On every boot I got the same info and failure message:
“File system errors were found and have been corrected.” and “Error: Not enough free diskspace available.” Free disk space was available after deleting some VMs. But I was still able to boot the system.

Than I remembered, that I already had an problem with this NVME a few years ago and I already had an RMA with this NVME. I decided to boot an live-USB again and run some tests on this NVME.
The tests showed me several errors. But this was weird because S.M.A.R.T. showed me an OK on this NVME.

I decided to install another NVME from another brand and run the same tests on this device. Everything is OK on this NVME and I installed IPFire.

Thank you for your help!