Root password not working on ssh

Everyone,

I’m in a bit of a panick mode this morning. When trying to login to my IPFire Corporate system the root password is not working for ssh. I can still login to the GUI, but the root password is not working via putty from my desk or on the local console. If I reboot to try to get into single user mode that will take down the entire network of 60+ systems.

Every other service (DHCP, DNS, Proxy, etc) seems to be working, I just can’t login.

I want to resolve the situation without taking down the network, and I want to be absolutely positively sure that our system was not broken into.

I’m looking for advice and “what would you do” feedback. Thanks.

if root password does not work on the ipfire console, that’s a problem.

(I assume ssh is enabled and active either on port 22 or 222)

Correct, ssh is active, I’m prompted for a username and password, but Access denied is what is returned every time.

My only thought is to set up another system as quickly as possible with the same configuration, and swap out boxes, thoughts?

how about cycle-thru ssh access, that is, disable ssh access from the gui, save, then enable ssh access, save. Maybe the process got stale for some reason … the only way to change root pw is via setup on console. (maybe bad keyboard, caps lock stuck, …

Is there anything in the System Logs for section ssh ? you can see that from the gui.

Already tried cycling through ssh. I confirmed after disabling ssh that I don’t get prompted at all for a username from Putty - as expected. When I turn it back on I get the same result.
ssh logs for today show:

9:40:40 sshd[31777]: Disconnecting authenticating user root 10.5.1.75 port 55061: Too many authentica tion failures [preauth]
09:40:40 sshd[31777]: error: maximum authentication attempts exceeded for root from 10.5.1.75 port 550 61 ssh2 [preauth]
09:40:40 sshd[31777]: Failed password for root from 10.5.1.75 port 55061 ssh2
09:40:37 sshd[31777]: Failed password for root from 10.5.1.75 port 55061 ssh2
09:40:34 sshd[31777]: Failed password for root from 10.5.1.75 port 55061 ssh2
09:40:32 sshd[31777]: Failed password for root from 10.5.1.75 port 55061 ssh2
09:40:30 sshd[31777]: Failed password for root from 10.5.1.75 port 55061 ssh2
09:40:27 sshd[31777]: Failed password for root from 10.5.1.75 port 55061 ssh2
09:39:40 sshd[31300]: Connection closed by authenticating user root 10.5.1.75 port 55052 [preauth]
09:39:36 sshd[31300]: Failed password for root from 10.5.1.75 port 55052 ssh2
09:39:34 sshd[31300]: Failed password for root from 10.5.1.75 port 55052 ssh2
09:39:31 sshd[31300]: Failed password for root from 10.5.1.75 port 55052 ssh2
09:39:07 sshd[31130]: Server listening on 0.0.0.0 port 222.
09:38:50 sshd[30769]: Received signal 15; terminating.
09:38:29 sshd[30769]: Server listening on 0.0.0.0 port 222.
09:38:22 sshd[20770]: Received signal 15; terminating.
09:38:03 sshd[30338]: Disconnecting authenticating user root 10.5.1.75 port 54998: Too many authentica tion failures [preauth]
09:38:03 sshd[30338]: error: maximum authentication attempts exceeded for root from 10.5.1.75 port 549 98 ssh2 [preauth]
09:38:03 sshd[30338]: Failed password for root from 10.5.1.75 port 54998 ssh2
09:38:02 sshd[30338]: Failed password for root from 10.5.1.75 port 54998 ssh2
09:37:46 sshd[30338]: Failed password for root from 10.5.1.75 port 54998 ssh2
09:36:58 sshd[29917]: Failed password for root from 10.5.1.75 port 54985 ssh2
09:36:56 sshd[29917]: Failed password for root from 10.5.1.75 port 54985 ssh2
09:36:54 sshd[29917]: Failed password for root from 10.5.1.75 port 54985 ssh2
09:36:51 sshd[29917]: Failed password for root from 10.5.1.75 port 54985 ssh2
09:34:17 sshd[28611]: Failed password for root from 10.5.1.75 port 54947 ssh2
09:34:13 sshd[28611]: Failed password for root from 10.5.1.75 port 54947 ssh2
09:34:10 sshd[28611]: Failed password for root from 10.5.1.75 port 54947 ssh2
09:22:31 sshd[24474]: Failed password for root from 10.5.1.75 port 54695 ssh2
09:22:28 sshd[24474]: Failed password for root from 10.5.1.75 port 54695 ssh2
09:18:12 sshd[22733]: Failed password for root from 10.5.1.75 port 54589 ssh2
09:18:09 sshd[22733]: Failed password for root from 10.5.1.75 port 54589 ssh2
09:08:59 sshd[20949]: Failed password for root from 10.5.1.75 port 54472 ssh2
09:08:24 sshd[20815]: Disconnecting authenticating user root 10.5.1.75 port 54462: Too many authentica tion failures [preauth]
09:08:24 sshd[20815]: error: maximum authentication attempts exceeded for root from 10.5.1.75 port 544 62 ssh2 [preauth]
09:08:24 sshd[20815]: Failed password for root from 10.5.1.75 port 54462 ssh2
09:08:21 sshd[20815]: Failed password for root from 10.5.1.75 port 54462 ssh2
09:08:17 sshd[20815]: Failed password for root from 10.5.1.75 port 54462 ssh2
09:08:14 sshd[20815]: Failed password for root from 10.5.1.75 port 54462 ssh2
09:08:11 sshd[20815]: Failed password for root from 10.5.1.75 port 54462 ssh2
09:08:08 sshd[20815]: Failed password for root from 10.5.1.75 port 54462 ssh2
09:07:58 sshd[20770]: Server listening on 0.0.0.0 port 222.
09:07:54 sshd[3794]: Received signal 15; terminating.
09:07:38 sshd[20642]: Disconnecting authenticating user root 10.5.1.75 port 54451: Too many authentica tion failures [preauth]
09:07:38 sshd[20642]: error: maximum authentication attempts exceeded for root from 10.5.1.75 port 544 51 ssh2 [preauth]
09:07:33 sshd[20642]: Failed password for root from 10.5.1.75 port 54451 ssh2
09:07:26 sshd[20642]: Failed password for root from 10.5.1.75 port 54451 ssh2

My internal ip is .75, I looked back at every log entry since the beginning of the month and there’s no activity other than my login, and I login almost daily. I would think if someone broke in they would have wiped the whole log but I’m not concluding anything yet. is listening on 0.0.0.0:222 a normal entry?

yes, the last checkbox on ssh access is not checked so ssh listens on 222

image

Hi,

all right, let’s ask the usual questions:

  • Is your keyboard layout configured correctly?
  • The root password might differ from the admin password. Are you using the correct one?
  • Are there any log messages indicating successful SSH logins within the last days (those can be viewed via the web interface as well)?

Thanks, and best regards,
Peter Müller

Keyboard is configured correctly, I confirmed root password is going correctly to the console by typing the root password temporarily into the username field without hitting enter. The admin and root passwords are different, and I’m 100% sure I’m using the right one, I’ve tried perhaps 25 times in the last hour, including both versions and other admin passwords used on the network. I’ve gone back and looked through all log entries so far to the beginning of June I don’t see any attempted login attempts over ssh from any other IP other than my own LAN IP, but I used the system this week multiple times without issue, so I know this just happened recently. As I said before I’ve tried from my local machine and from the direct console - a different keyboard) both layouts are correct and I’m typing the password correctly, it’s not working.

Thanks for your reply. Do you have physical access to that IPFire machine? If yes, are you able to log in at the console with the same password?

(If so, we are possibly dealing with a bug. If not, your system might indeed be compromised.)

Yes, When I say I have access to the console and I’m trying to login, I’m referring to typing on a KVM that’s wired into the physical box in the computer room. I’ve tried logging in at the physical device perhaps 15 times today, and it’s not working.

I see. Are you capable of creating a snapshot of that machine, in case it is compromised?

It’s a physical machine the border device on the network between the internet and the rest of the LAN, it’s not in a vmware environment that I can do any kind of snapshot backup. I can run the backup process from the GUI as I can still login there but I can’t do any kind of rsync backup or anything else with an external program, as I have no access to the command line.

Howzit Chris

If you cannot figure out why the password changed, then I suggest you treat the machine as compromised.
Common practise, is to remove log entries that show questionable activity and leave the rest, this avoids suspicion and lets the system give the resemblance of being normal. The only way to check if this was done needs command line access, which you don’t have… catch22 :upside_down_face:

If you however wish to keep said system and only want to reset the root password… let’s open Pandora’s box :smiling_imp:
It’s been some time since I needed to reset root pwd, so bare with me.
If memory serves me correctly you should have the option at boot to edit grub.

What you are looking for is the line starting with linux, mine as an example looks like

 linux	/vmlinuz-4.14.173-ipfire root=UUID=*long string of ascii* ro panic=10 

You may even have more than one of those lines.
You need to change the image from read-only to read/write, so change the ro part at the end to rw
Exit edit, and type mount to see what the root (/) partition is and if its set to rw

# mount
 /dev/sda3 on / type ext4 (rw)
 /dev/sda1 on /boot type ext4 (rw)
 /dev/sda4 on /var type ext4 (rw)

In the above example ./ is /dev/sda3 and is rw

If it says ro, you need to redo step one.
If its rw you can now type passwd root which will let you reset the root password.
If it moans with an error

passwd: Authentication token manipulation error
passwd: password unchanged

Then the partition is still ro and not rw
See if it will let you mount the / partition manually (not sure if the flag is -o or -n -o, you may want to check that)

mount -n -o remount,rw /
passwd root

If that worked you can type exec /sbin/init
Let the system boot and check if you can now login as root with your new password.

Alternative would be if you have a Linux live CD to boot from (I think Slax, Mint, Ubuntu, Debian, Slackware let you run off the CD before install or get a copy of Kali), open command prompt and check what the root partition is, type

fdisk -l  

Lets assume it’s /dev/sda3 as the above example, then do the following

mkdir /mnt/rec
mount /dev/sda3 /mnt/rec
chroot /mnt/rec
passwd root
exit
umount /mnt/rec
exit

Eject the live CD and let the system restart.
If I am not as rusty as i think I am you should now have resolved the issue, check your logs entries :stuck_out_tongue_winking_eye:

Hope this helps, good luck

On a side note, I add in public key(s), once I have set up ssh, and then disable password based access. No need to worry about keyboard language or layout there after…and it’s considerably more secure.

Forgot to answer your “what would I do”… simple, reinstall a machine form scratch, don’t trust the current one.
However if you go the above route and don’t redo your system, then at least take the time to check for questionable entries, as an example you should start here

cat /etc/passwd
cat /etc/group
cat /etc/shadow
cat /root/.ssh/authorized_keys
cat /root/.ssh/known_hosts

have you tried using “root” as login name, then set CAPS LOCK on and type the root password, as you know it.

It’s also possible that if NUM LOCK was on when root password was set then some characters would have been omitted.

Hi,

based on the information you gave us, it looks like the machine was compromised indeed. Personally, I would recommend to take a backup of its disk (e. g. by dd if=/dev/sdX of=/dev/sdY bs=1M status=progress) so you can analyze it further. Afterwards, please re-install your system.

In case you came across a security vulnerability in IPFire itself, please report it to security@ipfire.org or create a ticket at https://bugzilla.ipfire.org/ and request to make it private. Thank you very much in advance for doing so.

Thanks, and best regards,
Peter Müller

1 Like

Thanks for all the advice everyone I ended up redoing the machine and restoring from an earlier backup and changing all passwords.

Chris

1 Like