Speedtest-cli ERROR: Unable to connect to servers to test latency when run from fcrontab

1

This post has been re-written now the problem is understood

Hello,

I have a problem where an fcron job to run speedtest-cli fails when run from fcron if a root shell hasn’t been open in the past ~15 minutes.

  • If a root shell is open or has been only disconnected recently, it works without problem
  • This happens with both the Pakfire speedtest-cli add-on version AND a copy directly from GitHub - sivel/speedtest-cli: Command line interface for testing internet bandwidth using speedtest.net
  • The problem is not with the fcron syntax or the file permissions
  • The problem occurs when I do not nominate a speed test server
  • fcrontab syntax is 30 01 * * * /usr/local/bin/<myscript>.sh
  • The script runs without problem and it calls /usr/bin/python3 /usr/bin/speedtest-cli --no-upload --secure --csv but through debugging I can see that it always fails with error: Result is: ERROR: Unable to connect to servers to test latency.

I’m unable to find why this script works when I have an active session, but fails when there isn’t any.

If you have any ideas please let me know!

Thank you!

2

The problem seems to be that the cron job doesn’t run if there isn’t a root session available. This is why, whenever I test it, it works just fine.

Does anyone know why this would occur?

3

I think this was an “own goal”. :man_facepalming:

The fcron job was running, however as debug was off I didn’t know this.
My script produces output to the log (using logger) but only after a function attempted to run a command as another user (for security reasons).

I’d incorrectly been running:

RAW_SPEED=$( sudo --user=speedtest /home/speedtest/speedtest-cli --no-upload --csv | cut -d',' -f7 )

but the speedtest user deliberately doesn’t have a shell set!
(I had picked up the --shell option as best-practice somewhere)

So I simply removed that and the script still works

RAW_SPEED=$( sudo --shell --user=speedtest /home/speedtest/speedtest-cli --no-upload --csv | cut -d',' -f7 )

and now correctly produces some output when run from cron.

2024/12/20 EDIT: It turns out that I still haven’t resolved the problem. Cron is working fine, but for some reason the sudo syntax above only works if I set a cron session and log out before it runs. At 01:30 in the morning the sudo command fails to run.

6

Looking back at your first post you say that the issue is explicitly with the time 01:30 because if you change the time in the fcrontab to a few minutes in advance of your now time then the script is run successfully by fcron.

Can you confirm that the above is the issue that you are experiencing, that the script without changes works if you have a different time than 01:30

7

Hello and thank you for responding @bonnietwin !

When debugging I added an early echo statement and can now prove that cron isn’t the problem.

I have a script running as root, which needs to switch to another user for one command. There’s something preventing this from running when there are no interactive shells open on my IPFire system.

Background
I have a dedicated local user “speedtest” which exists to run a speedtest-cli script I found online. I’d prefer to not run the script directly as root so have tried a few different iterations of sudo syntax. The latest is this:

sudo --set-home --shell --user=speedtest /home/speedtest/speedtest-cli --no-upload --csv | cut -d',' -f7

and I explicitly set a shell for the speedtest user (before it didn’t have one but I could run cron jobs as the user):

# getent passwd speedtest
speedtest:x:1001:1001::/home/speedtest:/bin/bash

however there’s still something preventing the speedtest-cli from running. I’m not sure what I’m doing wrong - from my research those sudo options should cover my needs. Do you have any ideas please?

8

I just realised there’s a pakfire version of the speedtest-cli so I can trust that’s updated and should probably just run it as root. I’m probably over-engineering this, but now I’ve started I’d really like to get this sudo command working :smiley:

EDIT: I switch to running the pakfire version of speedtest-cli in my script and it appears to work from cron, when there’s no interactive sessions. I’ll find out tomorrow if it ran at 01:30.

2 Likes
9

It would be but the package itself has not been updated for some time.

I also use that package in a script to collect up and down speed data 4 times a day for the last couple of years.

I use the following script (speed_test.sh) to do it.

#!/bin/sh

# Runs the speedtest command and feeds the output
# in csv format to a file

this_month=$(date +%Y_%m)
speedtest --csv --secure >> /home/fcronuser/data/speedtest/"$this_month".csv

The script has owner and group fcronuser and permissions
rwxr-x---

The user is defined as
fcronuser:x:998:998:non root fcrontab user:/:/bin/false
so the fcrontab user has no login shell and is only used for running this script.

I then have that script run in a user fcrontab called fcronuser specified as

#
# crontab for fcronuser
#


# Run speedtest at 02:10, 06:10, 10:10, 14:10, 18:10 & 22:10
10 2,6,10,14,18,22 * * * /home/fcronuser/scripts/speed_test.sh

I created the fcronuser fcrontab using the wiki info

https://www.ipfire.org/docs/pkgs/fcron

In terms of your current situation, the only thing I can think of with regard to your problem is checking the permissions and the ownerships and also what you have placed in the sudoers file for the user you are using for the script. You need to allow that user to use sudo without a password otherwise it will expect one to be provided which doesn’t work in a non-interactive script.

I have created separate sudo files in sudoers.d for additional users such as one called 02-fcronuser which then contains

## Allow fcronuser to use sudo without a password
fcronuser       ALL=(ALL) NOPASSWD:ALL

I create the separate fcrontabs and sudoers files so that I can then add them to my include.user file for backing up and also if fcrontab gets updated then I don’t lose my additional entries.

1 Like
10

Thank you again.

Oh, that’s a pity, I had hoped the speedtest-cli version would be maintained. I had a quick look and noticed it was totally different to this version on GitHub but presumed it was an IPFire specific adaptation. (That version on GitHub hasn’t actually changed since April 2021 anyway!)

So, I’m running the cron job as root because I need root’s privileges to change QoS profiles (my crazy ISP offers me a few days a month at a higher speed, rather than just giving me a $2 discount!).

I am running sudo from root (to switch to another user) so as far as I was aware no password should be required. I briefy tried using su but it seemed harder to use.

Thanks - I did see that wiki documentation about the fcron user but the only cron job I ever ran as a non-root user was as my speedtest user, which I’d created long before the wiki page was written. I no longer need to run tasks as an unprivileged user from cron, but it’s a very helpful page.

For reference, when I needed to run a cron task as another user I’d used fcron to use the user specifically with this syntax:

59 1,5,9,13,17,21 * * * speedtest /usr/bin/speedtest-cli --csv >> "/home/speedtest/results.csv"

but in the case of this thread I need to switch QoS profiles and run the qosctrl IPFire script. (It seems worse to allow the unprivileged user to run these commands through sudo)

11

That is exactly the one we are using.

What makes you say it is totally different to the one in IPFire?

I just checked the b2sum hash for the source tarball from github with the b2sum hash that we have for the source tarball we have downloaded in the past and they match.

12

For your speedtest user what owner:group do you have set up and what are the permissions?

13

Here’s the info you asked about. I can’t find any problem here:

[root@ipfire ~]# id speedtest
uid=1001(speedtest) gid=1001(speedtest) groups=1001(speedtest)

[root@ipfire ~]# getent passwd speedtest
speedtest:x:1001:1001::/home/speedtest:/bin/bash

[root@ipfire ~]# ls -la /home/speedtest
total 600
drwxr-x--- 2 root      speedtest     86 Dec 21 16:20 .
drwxr-xr-x 8 root      root         114 Mar 27  2023 ..
-rw-rw-r-- 1 speedtest speedtest 475233 Jun 14  2022 results.csv
-rwxr-xr-x 1 root      root       65335 Dec  3 10:51 speedtest-cli

Previously, when running scripts directly from fcron as this user, it did not need to have a shell set. I’ve only added bash since trying to troubleshoot this problem.

14

As a separate issue, the script did not not complete when run at 01:30 (that is, without an interactive shell) even when the IPFire version of speedtest-cli is used as root. This is interesting…

15

The script still fails to run when there’s no open root session, so I’ve added debug output through it to confirm which step is failing.

16

edited with latest info:

The problem is running the speedtest-cli python script from a shell script started by fcron.

Even if I use the version provided with IPFire (a Pakfire Add-On) speedtest-cli fails. Even if I run the command directly as root and preface it with /usr/bin/python3 it fails.

However if I schedule cron less than ~5 minutes after I’ve exited a shell session on my IPFire system, it succeeds.

17

I’ve found the problem only occurs when speedtest-cli is run from cron when a root shell session is not open (in fact ~10 minutes after such a session has been exited).

The speedtest script is called with the following options:

/usr/bin/speedtest-cli --no-upload --secure --csv

but always returns this if a root shell hasn’t been opened recently:

Result is: ERROR: Unable to connect to servers to test latency.

I’m calling speedtest-cli from a bash script, but I’m 99% sure this extra step isn’t the cause.

Updated original post to include this detail

PS: I am attempting to work-around the problem by specifying a server. However, I ended up with all the code below instead of an elegant one-liner because the speedtest server does not let you use the same server a few minutes apart (the --list command returns a different list each time). Note that I already have a local unprivileged speedtest user set up which has a copy of the github script in it’s homedir.

RAW=$( sudo --user=speedtest /usr/bin/python3 /home/speedtest/speedtest-cli --server <NNNNN> --no-upload --csv 2>&1 )
  if [ ${?} -ne 0 ]
  then
    logger -p local0.info -t qos-automatic.sh[$$] "Speedtest failed. Retrying. Result: ${RAW}"
    RAW=$( sudo --user=speedtest /usr/bin/python3 /home/speedtest/speedtest-cli --server <NNNNN> --no-upload --csv 2>&1 )
    if [ ${?} -ne 0 ]
    then
      logger -p local0.info -t qos-automatic.sh[$$] "Speedtest failed. Result: ${RAW}"
      exit 1
    fi
RAW_SPEED=$( echo "${RAW}" | cut -d',' -f7 )
18

Could you provide the fcrontab line(s) that you are using and the content, ownership and permissions of the script you are running.

This may all be in the thread above but it is not so easy to be sure I have picked up exactly the correct stuff.

I will try and setup the same in my system and see if I can reproduce what you are experiencing.

If I can then that gives me something to work on.
If not :person_shrugging:

1 Like
19

Thank you again @bonnietwin .
I had also decided to put that detail in the first post of this thread when I rewrote it yesterday, as I realise this has become a saga.

Cron
30 01 * * * /usr/local/bin/qos-automatic.sh

Script
-rwxr-xr-x 1 root root 3655 Jan 2 21:26 /usr/local/bin/qos-automatic.sh

I could send the whole shell script but it’s long and I can prove it is not the cause of this problem. Here are some relevant parts:

#!/bin/bash
export PATH=/sbin:/bin:/usr/sbin:/usr/bin

RAW=$( /usr/bin/python3 /usr/bin/speedtest-cli --no-upload --secure --csv 2>&1 )
# I've tested both the above line and the below, with the same result
# RAW=$( sudo --user=speedtest /usr/bin/python3 /home/speedtest/speedtest-cli --server XXXXX --no-upload --csv 2>&1 )
if [ ${?} -ne 0 ]
then
  logger -p local0.info -t qos-automatic.sh[$$] "Speedtest failed. Retrying. Result: ${RAW}"
  RAW=$( /usr/bin/python3 /usr/bin/speedtest-cli --no-upload --secure --csv 2>&1 )
  #  RAW=$( sudo --user=speedtest /usr/bin/python3 /home/speedtest/speedtest-cli --server YYYYY --no-upload --csv 2>&1 )
  if [ ${?} -ne 0 ]
  then
    logger -p local0.info -t qos-automatic.sh[$$] "Speedtest failed. Result: ${RAW}"
    exit 1
  fi
fi

RAW_SPEED=$( echo "${RAW}" | cut -d',' -f7 )
logger -p local0.info -t qos-automatic.sh[$$] "DEBUG - RAW_SPEED = ${RAW_SPEED}"

My attempt to use specific speed test servers failed, returning an error Result: ERROR: No matched servers: XXXXX but I suspect this is due to a change in the way the speedtest servers work.

I realise the entire script output could be redirect to a single logfile with set -x, but the logger method I wrote above seemed adequate.

PS: If you can think of an alternate way to get an approximate downlink speed then I need not even use this speedtest-cli script!

Thank you again!

1 Like
20

Pleased to, hopefully, be of some help.

I have set the script up in your defined location and set the permissions for it.

I then tested running it manually and it worked. It added a line into the messages log which I was tailing.

I have added the entry into the root fcrontab and will now wait till tomorrow morning to see if it does or doesn’t run on my system at 01:30.

I also added 11:30 and 12:30 into the fcrontab so I get a quicker response with an earlier time but with the root access not having been used for at least 25 minutes - so longer than the 10 mins that @dnl found stopped the script working.

So will be interesting to see what happens with all three of those time settings.

2 Likes