Location database does not update

anon42188109 · 27 October 2020 14:28

I’ll start a new post for this issue only. Updated to core 151 …

[root@ipfire ~]# location version
Mon, 21 Sep 2020 05:38:06 GMT
[root@ipfire ~]# location update
Downloaded database is outdated. Trying next mirror…
Could not download a new database
[root@ipfire ~]#

But https://location.ipfire.org/databases/1/ shows a database dated 27 OCT 2020

@cbrown

pmueller · 27 October 2020 15:07

Hi,

thanks for starting a new topic for this issue.

Unfortunately, I cannot reproduce it two systems freshly updated to Core Update 151:

Last login: Tue Oct 27 15:54:36 2020 from 10.xxx.xxx.xxx
[root@fw1 ~]# location update
Downloaded new database from Tue, 27 Oct 2020 04:27:20 GMT
[root@fw1 ~]# location version
Tue, 27 Oct 2020 04:27:20 GMT

Last login: Tue Oct  6 20:36:41 2020 from 10.xxx.xxx.xxx
[root@fw2 ~]# location version
Fri, 23 Oct 2020 10:05:55 GMT
[root@fw2 ~]# location update
Downloaded new database from Tue, 27 Oct 2020 04:27:20 GMT
[root@fw2 ~]# location version
Tue, 27 Oct 2020 04:27:20 GMT

This is odd indeed, and at the moment, I am out of ideas why this happens on your machine. Just poking around: Could you please post the output of

cat /etc/os-release

here?

Thanks, and best regards,
Peter Müller

cbrown · 27 October 2020 15:24

Hmm, this is weird. I have updated to Core 151 and rebooted but
cat /etc/os-release shows:

[root@ipfire remote]# cat /etc/os-release
NAME=“IPFire”
VERSION=“2.25”
ID=ipfire
VERSION_ID=2
PRETTY_NAME=“IPFire 2.25 (x86_64) - core149”
ANSI_COLOR=“0:31”
[root@ipfire remote]#

Edit: Here’s my System Info

anon42188109 · 27 October 2020 15:30

system rebooted

[root@ipfire ~]# cat /etc/os-release
NAME=“IPFire”
VERSION=“2.25”
ID=ipfire
VERSION_ID=2
PRETTY_NAME=“IPFire 2.25 (x86_64) - core145”
ANSI_COLOR=“0:31”
[root@ipfire ~]#

I’m on 151 but mistakenly the pretty name reads, 145

cbrown · 27 October 2020 19:47

and …

[root@ipfire ~]# cat /var/ipfire/fireinfo/profile | grep release
“kernel_release”: “4.14.198-ipfire”,
“release”: “IPFire 2.25 (x86_64) - core151”,
[root@ipfire ~]#

anon42188109 · 27 October 2020 20:35

I d/l core151 iso and installed in Virtualbox to test. Did the setup configuration, rebooted. Accessed the gui. Enabled location block, selected all countries except A1,A2,A3, and US, Save, Applied firewall rules. Rebooted again. On console, I can successfully ping ipfire.org but …

location version: 21 Sep 2020, location update: Database is outdated.

cbrown · 27 October 2020 22:36

FWIW … update with --debug

[root@ipfire location]# location --debug update
HTTP GET Request to location.ipfire.org
URL: https://location.ipfire.org/databases/1/location.db.xz
Headers:
If-modified-since: Tue, 27 Oct 2020 04:27:20 GMT
User-agent: location/0.9.4
HTTP Response: 200
Headers:
date: Tue, 27 Oct 2020 22:34:51 GMT
last-modified: Tue, 27 Oct 2020 04:29:04 GMT
etag: “3eab10-5b29f7cad2476”
accept-ranges: bytes
content-length: 4107024
x-content-type-options: nosniff
x-frame-options: deny
referrer-policy: strict-origin
x-xss-protection: 1; mode=block
content-type: application/x-xz
strict-transport-security: max-age=31536000; includeSubDomains; preload
connection: close
Opening downloaded database at /var/lib/location/tmptx6zluhi
Downloaded database is outdated. Trying next mirror…
Could not download a new database
[root@ipfire location]#

Edit:
Interesting point: Opening downloaded database at /var/lib/location/tmptx6zluhi
The tmp file was not created in /var/lib/location/ or if it was, then it got deleted.

pmueller · 28 October 2020 16:18

Hi,

for the records: Since this seems to affect several people by now, I have just opened a ticket for it:

https://bugzilla.ipfire.org/show_bug.cgi?id=12514

Yes, this is a temporary one and the location downloader seems to cleanup orphaned files to avoid bloating the filesystem.

Thanks, and best regards,
Peter Müller

pmueller · 28 October 2020 16:33

Hi @cbrown @anon42188109,

just to ensure this is not because an outdated version of libloc itself: Could you please check if the output of the following command matches the one below?

[root@maverick ~]# location --version
location 0.9.4

Thanks, and best regards,
Peter Müller

cbrown · 28 October 2020 16:33

I thought it of interest that something did get downloaded. Yet there is the indication is that something was wrong preventing the database from getting updated. As I stated on the other thread, I can download the file via

curl https://location.ipfire.org/databases/1/location.db.xz --output location.db.xz

cbrown · 28 October 2020 16:34

[root@ipfire ~]# location --version
location 0.9.4
[root@ipfire ~]#

pmueller · 28 October 2020 16:54

Hi,

I see. Let’s figure this out.

The source code of the location command is available here, and the function for handling updates starts at line 399:


	def handle_update(self, db, ns):
		if ns.cron and db:
			now = datetime.datetime.utcnow()

[...]

In this script, we get to line 445 and exit with the error message mentioned above:


		# Try downloading a new database
		try:
			t = d.download(public_key=ns.public_key, timestamp=timestamp, tmpdir=tmpdir)

		# If no file could be downloaded, log a message
		except FileNotFoundError as e:
			log.error("Could not download a new database")
			return 1

t.download refers to an instance of the location downloader, whose source code is available here.

For some reason, this command in line 170 does return True, causing the error message “Downloaded database is outdated. Trying next mirror…”:

if not self._check_database(t, public_key, timestamp):
	[...]

In line 198 within the same file, the timestamp embedded into the downloaded database is expected to be newer than the timestamp published via the DNS record:

if timestamp and db.created_at < timestamp.timestamp():
	[...]

I believe this is why location update fails for both of you; we expect the database to be older than the timestamp published, but instead, it is newer or at least equally old.

This should not happen…

Thanks, and best regards,
Peter Müller

anon42188109 · 28 October 2020 17:26

same as cbrown, location 0.9.4

cbrown · 28 October 2020 17:46

Hi @pmueller, Is there anyway for you to send me some debug code to print out the values at the suspected areas of interest?
-cab

Edit: Did a search for python tutorial and stuck in a couple of prints just before the timestamp test:

[root@ipfire location]# location update
db.created_at : 1603859232
timestamp : 1603877232.0

Oddly, the timestamp on each execution with this debug is always an even 18000 greater than db.created_at

Edit: Would that 18000 delta be related to the -5hr timezone delta for my US/Central DST time?

[root@ipfire location]# date -u; date
Wed Oct 28 09:06:02 PM UTC 2020
Wed Oct 28 04:06:02 PM CDT 2020
[root@ipfire location]#

pmueller · 28 October 2020 21:14

Hi,

yes, that comparison is exactly where things go wrong here.

This suspiciously sounds like a time-zone issue indeed (I do not observe such issues within CET/CEST, which is +1/+2 from UTC). I will ask @ms about this the other day - it appears as the output of the db.created_at command is not in UTC, while the timestamp fetched from DNS is.

Thanks, and best regards,
Peter Müller

anon42188109 · 29 October 2020 13:48

(My python skills are basic) … I d/l the new database from location.ipfire.org and uncompressed it.
I still have the old database dated September, the new database dated October.

from datetime import datetime
import location

# current db in my system
db = location.Database("/var/lib/location/database.db")
db_obj = datetime.fromtimestamp(db.created_at)
print("old:", db_obj)

# newdb downloaded from ipfire site
newdb = location.Database("/var/lib/location/location.db")
newdb_obj = datetime.fromtimestamp(newdb.created_at)
print("new:", newdb_obj)

[root@ipfire ~]# python3 test.py 
old: 2020-09-20 22:38:06
new: 2020-10-26 21:27:20

anon42188109 · 30 October 2020 13:55

I installed (again) a fresh core 151. This time, as timezone I selected Europe/Athens instead of the usual America/Phoenix. Guess what, location update worked, I got a database 30 OCT 2020 04:27:17 GMT

cbrown · 30 October 2020 14:25

Ditto (sort of) here also: switch tz to CET and successfully downloaded db; switched back to US CDT and got the failure again

pmueller · 30 October 2020 14:49

Hi all,

for the records: This is indeed a bug in the location importer affecting all systems in time zones behind UTC.

A fix has been developed and merged into the libloc repository, and will probably be released with Core Update 153 since #152 is already closed. Please refer to these two commits for further information:

Sorry for things being so fuzzy at the moment…

Thanks, and best regards,
Peter Müller

dark0ipfire · 27 November 2020 21:41

I have a slightly different effect, but same issue (location database does not update):

# location --debug update
HTTP GET Request to location.ipfire.org
        URL: https://location.ipfire.org/databases/1/location.db.xz
        Headers:
                If-modified-since: Thu, 26 Nov 2020 04:29:36 GMT
                User-agent: location/0.9.4
HTTP Response: 410
        Headers:
                date: Thu, 26 Nov 2020 21:56:02 GMT
                content-length: 282
                content-type: text/html; charset=iso-8859-1
                strict-transport-security: max-age=31536000; includeSubDomains; preload
                connection: close
https://location.ipfire.org/databases/ reported: HTTP Error 410: Gone
Could not download a new database
#

Any idea / solution / suggestion? System is running on i586 with latest and clean installed core 152 (did a settings recovery from a backup made with core 148).