Core 139 - Filesystem full

Just updated from core 138 to 139 and I ran into an Filesystem full issue. This error popped up on the Home/Main page:

This is what I see:

So I’ve run out of space on /dev/sda3. From the above disk usage I think I need to clean everywhere but not /boot and not /var. Does that sound right?

I’ve been using IPFire on the same device since the core 110s (110 to 119). Did this partition increase over the years? And it is time to start with a fresh build?

I’ve unplugged my IPFire SSD some time ago and re-partioned the file system. Since then I did not see this error anymore.

Made a backup of the SSD before taking any action, on my (Windows) PC.

I thought about repartitioning but it seems a bit scary to me.

I’ve found a few small files to delete but I haven’t been able to locate large files (or LOTs of small files) to delete.

Have you checked in /lib ? That is where all the myriads of kernel modules are stored.

If you have more than one /lib/4.* folder then the lower numbered folders are for superceded kernels and could be deleted.

Thank you for the hint. Nothing in /lib that begins with “4”. I looked in /usr/lib also.

[root@ipfire pmacct]# du /lib -d1 -h

5.4M /lib/udev
524K /lib/security
2.6M /lib/xtables
487M /lib/firmware
1.1M /lib/kbd
46M  /lib/modules
566M /lib

My error - /lib/modules is where there need be only the latest folder, having a name starting with “4”

The kernel update procedure should remove older folders, but if it has not done so, then it should be safe to do so manually. However, with only 46 M in /lib/modules this does not appear to be your problem

The / partition is normally about 75 % full.

Your /lib looks normal in size. If your /usr/lib is about 500 M then that too is normal.

Try running “du” against each other folder in /, in turn. All should be only a few M each.

Yep. That is what I have been doing. Part of the problem is my inexperience with Linux and not knowing what is A-OK to delete.

[root@ipfire ~]# du / -d1 -h | sort -r -h
. . .
15G 	/			<<<<  total used
13G 	/var		<<<<  ignore separate partition
1.1G	/usr
566M	/lib
29M 	/boot		<<<<  ignore separate partition
20M 	/etc
14M 	/srv
8.9M	/sbin
5.3M	/bin
3.5M	/root
1.8M	/opt
464K	/run
140K	/home
28K  	/mnt
16K  	/media
16K  	/lost+found
16K  	/dev
4.0K 	/tmp
. . .

 

I’ve been digging the the /usr/share area hoping I’ll see something that looks familiar.

[root@ipfire ~]# du /usr -d1 -h | sort -r -h
1.1G /usr
880M /usr/lib
123M /usr/share
54M /usr/bin
40M /usr/sbin
676K /usr/local
244K /usr/libexec
56K /usr/include
8.0K /usr/etc

It’s your /usr/lib that looks oversized. Mine is only 447 M, but that is on an ARM system and not an accurate comparison.

Ignoring the smaller subdirectories, I get:

447M /usr/lib
220M /usr/lib/locale
55M /usr/lib/perl5
15M /usr/lib/python2.7

That might give you an indication of where your excess files lie.

I have not done a fresh install on an x86 system since core 134 and don’t know what size partitions a new install would create.

I’m curious - what Size is the / on your ARM system? Mine is 2.0 GB.

[root@ipfire ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.9G  4.0K  1.9G   1% /dev
tmpfs           1.9G   12K  1.9G   1% /dev/shm
tmpfs           1.9G  452K  1.9G   1% /run
/dev/sda3       2.0G  1.7G  167M  91% /
/dev/sda1        59M   29M   26M  53% /boot
/dev/sda4        56G  4.3G   49G   9% /var
/var/lock       8.0M   12K  8.0M   1% /var/lock

On my / I’ve gone from 99% full down to 91% full by deleting addons that I don’t use all of the time.

Edit: Ive been digging the the /usr/lib directory looking for things I recognize and don’t need.

IPFire installations on ARM don’t use a separate /var. The / partition occupies most of the device, which could readily be up to 32 GB (u)SD card. Mine is actually 7.3 G on nano Pi - R1, of which 1.5 G is used - that includes 135 M of /var, without which / would have about 1.35 G used.

Are you using i586 or x86_64 version of IPFire ?

I am using x86_64

I’ll fire up an x86_64 installation and get back

I didn’t see anything worthy of deletion in /usr/lib.

I deleted ALL of my pakfire addons and I got down to 78% full

[root@ipfire locale]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.9G  4.0K  1.9G   1% /dev
tmpfs           1.9G   12K  1.9G   1% /dev/shm
tmpfs           1.9G  448K  1.9G   1% /run
/dev/sda3       2.0G  1.4G  411M  78% /          <<<<<
/dev/sda1        59M   29M   26M  53% /boot
/dev/sda4        56G  4.3G   49G   9% /var
/var/lock       8.0M   12K  8.0M   1% /var/lock

Seems a bit extreme, eh?

It transpires that I installed my x86_64 from a dot.IMG rather than dot.ISO, so it also puts /var on the same / partition. FWIW, that still uses only 1600 M, and /var contributes only 139 M to that - so not too different from ARM.

/lib is still only 557 M. That is likely where your problem lies.

I can’t readily copy & paste the output because the machine runs on a different LAN

The only significant addons I have are CUPS and Samba, but those are only there as fallbacks and not used. In any case, their data tends to go into /var.

Your most reliable option now could be to download your backup files to another PC then reinstall.

I agree! I’ll put this on this weekend’s to-do list! Thanks for your hints and help!

Thank you Michael @ms!! You are THE best!

https://lists.ipfire.org/pipermail/development/2020-January/006844.html

Is there a way for me to run this? Please know I know very little about git.

I grabbed the filesystem-cleanupsh out of the above message at https://lists.ipfire.org/pipermail/development/2020-January/006844.html and gave it a try (after a few tests). It deleted almost 60M with of files.

[root@ipfire filesystem-cleanup]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.9G  4.0K  1.9G   1% /dev
tmpfs           1.9G   12K  1.9G   1% /dev/shm
tmpfs           1.9G  448K  1.9G   1% /run
/dev/sda3       2.0G  1.4G  468M  75% /				<<<<
/dev/sda1        59M   29M   26M  53% /boot
/dev/sda4        56G  4.3G   49G   9% /var
/var/lock       8.0M   12K  8.0M   1% /var/lock

If you reinstall the addons hat you deleted then you might still be approaching the 2 G size of /.

My reading of the installer code for core 139 is that it no longer creates a separate /var. Almost all disk space is allocated to / and that would avoid the problem that you are facing.

Agreed, I found the same:
https://blog.ipfire.org/post/ipfire-2-21-core-update-121-is-available-for-testing
(see New partition layout near bottom of page)

I don’t wanna rebuild… I don’t wanna rebuild… I don’t wanna rebuild… waaaah! :angry:

[composure]

Its fairly easy to rebuild but I’ve added a few custom addons that I need to find the notes for… time to find my notes!

Yeah, I see that. At some point, we probably cannot avoid this, but we have taken on the problem right now and I have written that script together with Arne last week. There are some other things we can do to keep your box going for some time and it might be best to re-install at some point when you change hardware or so.