Rescue at Hetzner

Hetnzer has got some nice servers often at their auctions. The issue is that those servers are totally unmanaged. You need to be pretty clued in how they are setup and you should never lose the thin umblical chord to it

We were connected to one such server and someone did something silly on SSHd and suddenly we found ourselves with the dreaded “SSH Too Many Authentication Failures”

Now if you search for that on Google, you would often find good advice – but there is crucial flaw in their advice. They assume that you are able to connect to that erring server in some other way. But what do you do with an unmanaged Hetzner server which you are managing from India (in our case) using SSHD?

Worry not! This is what needs to be done. It’s clunky. But it works!

Note: This is for Ubuntu! If you used some other version of Linux, you would need to modify the instructions below accordingly.

Firstly you need to have Java installed on your local machine. Then go over to the Hetzner support and ask them provide you with an emergency KVM console

You will get an automated email immediately stating that you are in queue. After a few more minutes (or hours, depending on what their queue was) you would be given access to a console that directly connects to your server.

They will give a link, a username and a password.

On clicking that link, you would get a JNLP file – which is to be run via Java. If it does not start automatically on your windows computer; locate the correct javaws.exe executable (and not the java.exe) Thta one runs the JNLP file.

Now for the headache! By default, the terminal window opens up would be in Italian or German or some other keyboard. There is an options button at the top – and you can theoretically switch it over to the keyboard you have. Try it. In our case, that also did not work. The keyboard kept sending out strange characters. For e.g. the / key was somewhere else!

So we opted for the on-screen keyboard. That works.

Now, you need to send a Ctrl-Alt-Del to reboot your machine; else you would be waiting at a screen which has not proceeded further due to the error. Moreover the screen display is all out of whack in that Java application. It would take a few trials and errors to get display to look right

Now when the system is rebooting, follow carefully the instructions appearing on the screen – and interrupt the boot process so that you get a set of options where the option to use the Ubuntu Rescue mode. I guess this mode does not load all the usual services that normally are auto-started in the beginning.

Phew! Now you have root access to the rescue server. Which is basically your own server minus the initial system services.

We managed to load the errant sshd config with nano

nano /etc/ssh_config

And removed the access to the additional user there. But wait! It didnt happen so easily! Even the onscreen keyboard didnt work well this time inside nano. The only thing that did work was the backspace.

Luckily we were able to backspace the offending line into a comment that luckily was available before that.

Yay! That was nice! Now we saved the ssh_config moved over to the offending users home directory, located the .ssh folder there and removed all the authentication keys

In the command line, the onscreen keyboard worked fine – and were able to restart sshd with the new configuration!

Then one ctrl-alt-del and the server rebooted; and this time it worked! Yippeeee!