No more access to ipfire - Core 150 | Location-Filter

Getting Started with IPFire Updates Trouble
41

Hi,

…“for the records” and for everyone who runs into this problem with Core 150 - perhaps it helps:

Despite upgrading to Core 150 a few days ago, nothing was blocked. I waited.

Yesterday morning, 8:00, I suddenly couldn’t get any access to the GUI or via SSH. ‘ping’ was OK, but that was all. I had apparently joined the bug. Database update!? Ok, lets see.

What I did (memory protocol!):

Since there is no keyboard or monitor down in the cellar where my box is standing, I had to take her upstairs with me.

I attached monitor, keyboard and GREEN only. No RED available here…

Disabled Geo-IP-Blocking by editing /var/ipfire/firewall/locationblock, changing LOCATIONBLOCK_ENABLED=on
to
LOCATIONBLOCK_ENABLED=off.

Added the fix from:
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=c69c820025c21713cdb77eae3dd4fa61ca71b5fb to ./usr/lib/firewall/rules.pl

Rebooted. Access to GUI was OK. Hm.

I brought the box back in the cellar and attached GREEN and RED.

After a restart, GUI was still OK, but DNS and DNSSEC were completely broken.

I opened https://[GREEN_IPFIRE_IP]:444/cgi-bin/location-block.cgi (Enable Location based blocking: was unchecked as intended), changed NOTHING, just clicked on SAVE and reloaded all rules on https://[GREEN_IPFIRE_IP]:444/cgi-bin/firewall.cgi.

Then I activated Geo-IP-Blocking again and its still running OK.

IPTables / LOCATIONBLOCK on https://[GREEN_IPFIRE_IP]/cgi-bin/iptables.cgi have to look like this:

2020-10-17_iptables_location_block
2020-10-17_iptables_location_block823×285 6.13 KB

HTH,
Matthias

1 Like
42

Only can edit /var/ipfire/firewall/locationblock file and change “on” to “off” and restart firewall with /etc/init.d/firewall restart, everything goes back to normal. I change it connecting one monitor and keyboard.

That’s the solution until there’s a patch.

43

To explain this patch

The first iptables rule excludes local networks from being analysed by location-filter.
The second block of rules excludes private networks, which can exist on RED from the filter.
Thus the inconsistancy in the libloc tables is circumvented.
You can see this very clear in the post of Matthias Fischer above.

Hope this clarifies the case a bit, also the fact that only a part of systems is affected.

1 Like
44

FYI I have edited my fw rules.pl with the above changes, enabled GeoIP and applied fw change.
All working again as desired, after reboot too, to be sure. Well done on the quick fix.

One thing the location snafu has shown is the importance of running code only where required (i.e. Red interface in this case). Local addresses in database obviously a different matter. Location blocking should be rock solid from now on. :thinking:

45

I’m currently on the road (vacation) and had previously only disabled EU and DE in the location filter. It worked for 1 day but now I can’t access my Nextcloud or via VPN. It will probably be blocked by the error again.

:roll_eyes:

46

Well, the damage is done.

Now we have to think about a mechanism so that it does not happen again.

Before publishing the databases of the countries, it could be done a filter to detect / eliminate these errors.

I have several remote machines without access and explain the error to the end customer / distributor. You totally lose credibility. Besides being confiable, they have to be reliable.

This is a brutal denial of services as the connectivity of the machine is totally lost.

I do not want this to be taken as a negative review, but a comment to improve IPFire.

Thank you all.

2 Likes
47

This error could have been found by using the testing tree in systems that have private IPs as RED IP.
Just my opinion.

The source of the error is another story.

48

Hi all,

apart from exporting the networks in a way xt_geoip can handle overlaps safely, there also seem to be a few issues with dummy RIR data or records transferred from ARIN after establishing Regional Internet Registries.

We are working on detecting and excluding them while building the location database without loosing too much performance (string comparisons are involved, and it would be nice if we could work around regular expressions). So please continue holding the line… :slight_smile:

Thanks, and best regards,
Peter Müller

2 Likes
49

Hi,

apart from exporting the networks in a way xt_geoip can handle overlaps safely, there also seem to be a few issues with dummy RIR data or records transferred from ARIN after establishing Regional Internet Registries.

safeguards against the latter are now implemented (by using regular expressions, grmpf :expressionless: ). APNIC still writes lots of /8 network chunks into the database, but since they own those, it is technically correct to do so.

Some aspects regarding xt_geoip export are still to be clarified before I can submit a patchset to the location mailing list and we go into QA.

Thanks, and best regards,
Peter Müller

1 Like
50

Hi,

a patchset for sanitising RIR data (which fixes bugs #12500 and #12501) has been sent to the location mailing list. Although it did not show up any quirks on our Q/A system, I would be grateful if some of you could have a look at it and report anything suspicious here or on the mailing list (preferred). :slight_smile:

At the moment, we struggle with bug #12506, but once this is solved, I am confident of having a fixed version rolled out with upcoming Core Update 151, unforeseen incidents not taken into account.

Thanks, and best regards,
Peter Müler

3 Likes
51

Hi all,

meanwhile, @ms solved bug #12506 so we can generate and publish new location databases again.

The - at the time of writing - most recent one does not break any leftover Core Update 150 systems with the location filter enabled, as we reverted some changes introduced to make the database more accurate, but triggering the combination of bugs mentioned earlier.

This still leaves us with finding an elegant way of introduce the more accurate database again without breaking anything. Please refer to this thread on both development and location mailing lists for further information.

Apart from that, thanks again for your understanding and patience. There seems to be a light at the end of the tunnel, and this time we believe it’s not a train. :slight_smile:

Thanks, and best regards,
Peter Müller

2 Likes
52

Dear Peter,

thanks fro the info. is there anything we can do while we wait for Core 151?
Or it’s better to wait for it?

Best regards

UT1

53

Hi,

from my point of view, there is nothing to do apart from installing Core Update 151 as soon as we have released it (personal ETA: next week, but that is only a speculation :wink: ).

Your system will fetch new location databases with Core Update 150 as well, but they are not as accurate as they could be. It will probably take some time (Core Update 152/153) until we can finally publish accurate databases.

Thanks, and best regards,
Peter Müller

54

Got it!
Thanks.

55

My database has not updated since Oct 5th. Is that something to be expected with Core 150?
When I attempt to update it, I get

Downloaded database is outdated. Trying next mirror…
Could not download a new database

https://fireinfo.ipfire.org/profile/af4e7c1aa3b870cfd177edcf64593dd04b9607d5

56

Hi,

downloading a new database seems to work fine here using Core Update 150:

[root@maverick ~]# date
Sat Oct 24 03:58:13 PM CEST 2020
[root@maverick ~]# location update
Already on the latest version
[root@maverick ~]# location update --cron daily
The database has been updated recently (9 Hours, 32 Minutes, 44 Seconds ago)

Is DNS working correctly on your machine? Are there any outgoing firewall rules preventing the communication to the location mirror(s)?

Thanks, and best regards,
Peter Müller

57

DNS seems to be working fine – I have multiple servers using TLS
I see no DROP_OUTPUT entries in logs corresponding to “location update”
How do I find a list of location mirror(s)?

Thanks

58

The code accesses https://location.ipfire.org/databases/ but I’m in the same situation as @cbrown, database is outdated. Core 150, pings work, nothing else blocking ipfire.

[root@ipfire ~]# location update --cron daily
Downloaded database is outdated. Trying next mirror...
Could not download a new database
[root@ipfire ~]#
59

Check the output of dig, IPS, libloc and logs. A bit more thoroughly than usual, if you can

60

No IPS events, no DROP_OUTPUT in /var/log/messages; dig looks happy … I suppose?

dig location.ipfire.org

; <<>> DiG 9.11.21 <<>> location.ipfire.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46598
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;location.ipfire.org. IN A

;; ANSWER SECTION:
location.ipfire.org. 3197 IN CNAME fw01.ipfire.org.
fw01.ipfire.org. 3197 IN A 81.3.27.38

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Oct 24 18:35:30 CDT 2020
;; MSG SIZE rcvd: 83

Is there something else I can provide? Where do I find logs for libloc?