Location database does not update

Security Firewall Rules
21

I have the same problems.

[root@bs ~]# location --debug update
HTTP GET Request to location.ipfire.org
        URL: https://location.ipfire.org/databases/1/location.db.xz
        Headers:
                If-modified-since: Sat, 28 Nov 2020 04:29:57 GMT
                User-agent: location/0.9.4
HTTP Response: 410
        Headers:
                date: Sat, 28 Nov 2020 07:59:03 GMT
                content-length: 282
                content-type: text/html; charset=iso-8859-1
                strict-transport-security: max-age=31536000; includeSubDomains; preload
                connection: close
https://location.ipfire.org/databases/ reported: HTTP Error 410: Gone
Could not download a new database
[root@bs ~]#

New bug?.

UPDATE: I have tried on several IPFires that I have of i586 and x64 architectures and it gives the same error.

Regards.

1 Like
22

Same here, whereas curl doesn’t report the HTTP 410 failure:

# curl https://location.ipfire.org/databases/1/location.db.xz --output location.db.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4074k  100 4074k    0     0  9177k      0 --:--:-- --:--:-- --:--:-- 9177k

And wget:

# wget -S https://location.ipfire.org/databases/1/location.db.xz
--2020-11-28 10:17:42--  https://location.ipfire.org/databases/1/location.db.xz
Resolving location.ipfire.org... 81.3.27.38
Connecting to location.ipfire.org|81.3.27.38|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  date: Sat, 28 Nov 2020 09:17:44 GMT
  last-modified: Sat, 28 Nov 2020 04:31:38 GMT
  etag: "3fab4c-5b52340bb90d1"
  accept-ranges: bytes
  content-length: 4172620
  x-content-type-options: nosniff
  x-frame-options: deny
  referrer-policy: strict-origin
  x-xss-protection: 1; mode=block
  content-type: application/x-xz
  strict-transport-security: max-age=31536000; includeSubDomains; preload
Length: 4172620 (4.0M) [application/x-xz]
Saving to: 'location.db.xz'

location.db.xz                                  100%[===================>]   3.98M  14.0MB/s    in 0.3s    

2020-11-28 10:17:43 (14.0 MB/s) - 'location.db.xz' saved [4172620/4172620]
23

Hi all,

I guess it’s time for another short report from the engine room:

With upcoming Core Update 153, where no testing version is available yet, we plan to finally introduce the more precise location database, but this time without breaking systems as we did before. :expressionless:

Unfortunately, this requires a different output for the xt_geoip module, as it cannot handle tree, but only lists of networks, which must be flattened so there are no overlapping entries (xt_geoip does not look for the best match, but for the first one). @ms has worked several days on this to make it work fast enough even on dated hardware, and we believe we are somewhat there by now.

While Core Update 153 will contain the fix for the timezone problem (bug #12514), we have had to stop the generation scripts for the current databases, which is why every user now gets a 410 - gone HTTP error.

This is intended and unfortunately the only possibility we have. If we simply publish the more precise database, a number of systems running Core Update 152 or below would break again, and since all location update clients query the same URL, we cannot distinguish them at this level.

Worse, we cannot generate the more accurate and the less accurate version at this time, since the location clients query those by a DNS record, and if the generation time of the downloaded file do not match that record exactly, the files won’t be processed. It seems as we have built in so many safeguards into the updating procedure that we somewhat crippled ourselves when it comes to this problem. :slight_smile:

To keep it short: A testing version of Core Update 153 will be available soon (personal ETA: next 14 days) and as soon this will be released, any IPFire machine running it will get the more precise location database again, regardless of the time zone it is located in.

Sorry for all this confusion.

Thanks, and best regards,
Peter Müller

2 Likes
24

I was assuming this already… Thanks for the update and the work you’re putting in to this! :slight_smile:

1 Like
25

Hi,

there seems to be a glimmer of hope for this (at least from my point of view), see also:

Thanks, and best regards,
Peter Müller

2 Likes
26

So, activating Geo IP blocking with that old database could lead into the trouble, which was discussed in No more access to ipfire - Core 150 | Location-Filter or will it work as intended?

27

I suggested on bugzilla (probably better in developer mail list) that a less frequent location database update cron job (to daily or weekly), is all that’s necessary.
Alternatively a manual option/ability to switch auto update off - then you could ask users not to update until next IPFire release.
How often do countries and IP address ranges change that wee need hourly updates (I know you don’t currently change location db source that often).
Just thinking about stability.
Thanks

1 Like
28

Hi,

no, enabling the location filter will work as intended, as we have built in safeguards into Core Update 152.

Thanks, and best regards,
Peter Müller

29

Hi,

the default update interval of the location database is set to weekly:

Snippet from /var/spool/cron/root.orig:

# Update location database once a month.
%hourly,random * [ -f "/var/ipfire/red/active" ] && /usr/local/bin/update-location-database >/dev/null 2>&1

Snippets from /usr/local/bin/update-location-database:

UPDATE_INTERVAL="weekly"

# [...]

# Get the latest location database from server.
if /usr/bin/location update --cron=$UPDATE_INTERVAL; then
        # [...]
fi

This won’t solve our problem, which is actually more related to the xt_geoip module than the location database itself.

Well, things are stable at the moment (at least from my point of view): Systems running Core Update 152 and older will not break, and as soon as Core Update 153 is out and a machine is upgraded to it, a new, more precise location database will be used without breaking anything.

Just stay patient another few days please… :slight_smile:

Thanks, and best regards,
Peter Müller

1 Like
30

Yes, I activated Location based blocking, works so far. Although I observe a strange phenomenon on the location-block.cgi page: Countries list ranges only from letters A to E, all others are missing and cannot be found on iptables listing, too:

# iptables -L | grep country
DROP       all  --  anywhere             anywhere             -m geoip --source-country A1
DROP       all  --  anywhere             anywhere             -m geoip --source-country A2
DROP       all  --  anywhere             anywhere             -m geoip --source-country A3
DROP       all  --  anywhere             anywhere             -m geoip --source-country AD
...
DROP       all  --  anywhere             anywhere             -m geoip --source-country EG
DROP       all  --  anywhere             anywhere             -m geoip --source-country EH
DROP       all  --  anywhere             anywhere             -m geoip --source-country ER
#

Whats wrong with (my) data and how can I repair this? Might this be a side effect of restoring settings from core 148 backup with activated geo ip blocking onto clean core 152 system with newer geo ip blocking engine?

31

That is the known bug about perl access to libloc.

2 Likes
32

Dear @bbitsch, great - your modified /var/ipfire/location-functions.pl brings back all 256 countries in the list, just implemented it on my ‘weak’ Alix-board system with AMD Geode LX 500 MHz cpu. Thanks a lot! :+1:

1 Like
33

So, basically all will be fixed with core 153, right?

Just updated from core 144 to core 152 and disabled all GeoIP blocking beforehand because of these known bugs.
Is there a change to download the fix(es) before core 153 will be out in the wild?

Michael

34

Hi,

So, basically all will be fixed with core 153, right?

yes. :slight_smile:

Is there a change to download the fix(es) before core 153 will be out in the wild?

Not really as the changes involve re-compiling libloc so it is a bit more complicated than copying a script from next onto your machine.

Thanks, and best regards,
Peter Müller

1 Like
35

Thanks for your reply Peter!

As a conclusion - it’s best practice at the moment: do not use location blocking, correct?

37

I just saw the post with HTTP 410…

Is there any way IPFire users get announced about these changes that put down parts of the IPFire services?

38

Hi,

well, the location filter can be safely used if you are running Core Update 152, since we have built in various safeguards to ensure traffic aside from RED is no longer affected, etc.

In my opinion, you are safe to use the location filter if you like or have to.

Thanks, and best regards,
Peter Müller

3 Likes
39

Hi,

well, those changes were made a bit in a hurry, which is why we did not announce them in-depth. In addition, there unfortunately is nothing to see, yet. Core Update 153 is on tonight’s telephone conference agenda, and since libloc looks fine (we kept it in next extra long to ensure there are no bugs in the testing release anymore - at least none we are aware of :wink: ), I expect a testing version of it to be available soon.

Sorry for things being so fuzzy at the moment… :expressionless:

Thanks, and best regards,
Peter Müller

4 Likes
40

Hi all,

for the records:

Thanks, and best regards,
Peter Müller

2 Likes
41

Yes, core update 153 does it!

[root@ipfire ~]# location --version
location 0.9.5
[root@ipfire ~]# location --debug update
Already on the latest version

(although I had expected a more verbose output with “–debug” option, so showing ‘HTTP GET Request…’ and more)

2 Likes