How to Troubleshoot SSH Protocol Issues
SSH is the primary method available for managing DigitalOcean Droplets. Dealing with SSH errors or failures can be frustrating because the errors themselves often prohibit you from accessing your servers.
There are two prerequisites to troubleshooting SSH issues:
- Should I troubleshoot SSH? Determine whether troubleshooting is the right decision or if migration/redeployment is more appropriate.
- What should I do before troubleshooting SSH?. Make sure the issue is truly with SSH, then review the information and skills necessary to resolve SSH issues, like having root access to the server and understanding how to access and edit files.
When to Consider Migration or Redeployment
To resolve your issue quickly, first determine whether troubleshooting the connection is the right solution for your problem or if you should instead focus on recovering your data for redeployment.
Some issues, such as an accidental recursive rm or chmod command or incorrect network configuration, can lock you out of a Droplet permanently. Other issues may seem like connection problems, but are actually more complex issues with no clear resolution, like corrupted file systems, incorrect file permissions and ownership, and broken system packages and required libraries.
You can typically identify boot errors through the Droplet console startup output. File system issues and startup failures that prevent a working console login session are signs that troubleshooting your network configuration may not be the better option. In situations like this, the best approach is to salvage what you can. In some cases, a good backup or snapshot strategy is the fastest way back to your previous working environment.
What to Do Before Troubleshooting
If you’ve decided that troubleshooting is right for your situation, go through the following steps:
-
Check the control panel. Before anything else, make sure there are no ongoing issues, like an outage in the region impacting your Droplet.
-
Check if Droplet is disabled because of abuse. Droplets are sometimes disabled due to the detection of abusive activity. If your Droplet has been disabled, an email has been sent to the email address linked to your DigitalOcean account with the title
Networking Disabled: <your-droplet-name>. You can also log in to the support portal to see if any support tickets have been created for your resources.If your Droplet has been disabled due to suspected abuse, contact our support team for further information.
-
Recover root access. If you do not have the current root password, reset it using the reset root password function in the control panel.
-
Access the Recovery Console. If you cannot log in to the Droplet, the Recovery Console is another way to gain access (as long as your Droplet is running and you have a working root password).
-
Reboot your Droplet. Many connectivity problems can be resolved after a reboot. If you’re experiencing connectivity issues, try rebooting the Droplet and see if this resolves the issue.
Before rebooting your Droplet, we highly recommend taking a snapshot of it. This allows you to redeploy your Droplet in its current configuration if rebooting the Droplet causes more serious problems.
To reboot your Droplet, log in to it and run the following command:
sudo reboot -
Review file management and permissions. Some of these solutions may require you to review or edit files on the system or manage permissions.
-
Check logs. Once you can get into the Droplet, check the system’s log files for more information to identify the error so you can then look up a solution.
You can learn more about the logs on your server with this Linux logging tutorial and this journalctl and systemd logging tutorial.
-
Use verbose SSH output. The level of detail an SSH client provides about the SSH session is generally quiet by default. It’s helpful to have more information when debugging an issue.
For the OpenSSH client, you can use the
-voption with multipleventries to increase the verbosity of the output, as inssh -v [email protected]. While most issues are revealed with a singlev, some issues may benefit from-vvv.The PuTTY client supports an Event Log accessible from the context icon in the application window bar. There’s also an option for configuring session logging from the settings page when initiating the connection.
After you decide to troubleshoot an SSH issue instead of migrating or redeploying, you can identify and resolve specific SSH errors based on which phase of a successful SSH connection you need to debug.
Once a connection is successfully established, the SSH protocol negotiates an encrypted connection with the client based on establishing trust. Before you can even verify who you are on the system, the system attempts to establish an encryption method with your client. There are a few unique issues that can occur in this scope, and this tutorial covers how to identify them, how to resolve them, and additional resources to prevent them.
Errors
Host Key Verification Issues
When you connect to a server through an SSH client, the server attempts to identify itself using a host key. In situations where the host key changes, you may see a warning such as:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
...
In PuTTY, you get an equally ominous warning:
WARNING - POTENTIAL SECURITY BREACH!
The server's host key does not match the one PuTTY has
cached in the registry. This means that either the
Server administrator has changed the host key, or you
...
Common causes for this issue include:
- Rebuilding the Droplet from a snapshot or backup
- Moving a Reserved IP to another Droplet
- Reinstalling the SSH server through the package manager
To resolve this, you can clear your host keys.
Connection Closed or Reset By Peer
In some situations, connectivity is established at the socket level but disconnects during the host key verification phase.
OpenSSH clients may show this error:
Connection closed by 203.0.113.0 port 22
PuTTY may show this error:
Server Unexpectedly closed network connection
Some common issues that may cause this error include:
- The SSH service has crashed or is experiencing bugs.
- The SSH service cannot initialize a connection due missing server host keys.
In a situation like this, a more thorough review of the protocol output is necessary. In most cases, you need to investigate this from the server through the Recovery Console. From there, make sure that the service is currently running and bound to the expected port.
Unable to Negotiate with Host
When initiating the SSH protocol, a shared secret is generated through a cipher negotiated between the client and the host.
An OpenSSH client may show an error like this:
Unable to negotiate with 203.0.113.0: no matching key exchange method found.
Their offer: diffie-hellman-group1-sha1
PuTTY may show an error like this:
Couldn't agree a client-to-server cipher (available: aes128-ctr, aes192-ctr, aes256-ctr)
Some common issues that may cause this error include:
- You have a modern OpenSSH client implementation and an older host that does not support a common cipher with the client.
- You have customized your SSH client cipher list to only allow a selection that the server does not support.
To resolve this issue, you need to customize the supported ciphers in your SSH client.
Solutions
Clearing Out Host Keys from Known Hosts
Host keys are typically stored in the ~/.ssh/known_hosts file on OpenSSH client implementations. PuTTY typically stores these in the Windows Registry (HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys).
In PuTTY, use the window that opens when you connect to select Yes to allow the connection and update the registry. Alternatively, you can manually remove the entry.
On an OpenSSH client, you can find the host entry in the ~/.ssh/known_hosts file and manually remove it. Another option is to use the ssh-keygen command:
ssh-keygen -R 203.0.113.0
This attempts to access and clear the matching host entry in the known_hosts file:
# Host 203.0.113.0 found: line 2
/home/user/.ssh/known_hosts updated.
Original contents retained as /home/user/.ssh/known_hosts.old
You can then try to reconnect to your server normally.
Checking and Regenerating SSH Host Keys
If your SSH host doesn’t have its own private key to generate a shared secret, the connection is aborted. To check that the host has a private key, look in the /etc/ssh directory for a group of files named like sshd_host_*_key with corresponding .pub files.
If the files don’t exist, you need to regenerate them. You can do this using ssh-keygen as root or with sudo:
ssh-keygen -A
The output reports the actions taken, including a list of the keys generated:
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
You can then try to reconnect to your server normally.
Customizing Supported SSH Ciphers
You can customize the supported SSH ciphers on your client machine when you need support for a deprecated cipher like SHA1. This is not a very common issue. It typically happens when you use a modern SSH client to connect to an old SSH server that hasn’t yet disabled weaker ciphers.
In OpenSSH, you can configure SHA1 using the KexAlgorithms option on the command line:
openssh -oKexAlgorithms=+diffie-hellman-group1-sha1 [email protected]
PuTTY should already include the Diffie-Hellman group 1 option in the Connection > SSH > Kex configuration.
Conclusion
If you need further help, you can open a support ticket. Make sure to include the following information:
- The username, host, and port you are using to connect.
- The authentication mechanism you expect to use.
- The full output of the errors linked to the stage of error, including verbose output of the SSH client
- All of the information you’ve gathered from troubleshooting so far.
- Anything you were unclear about while referencing this article.
Including all the above diagnostic information and clarifying where you are encountering the issue when trying to connect can help us quickly get up to speed with where your need on the issue is.
Related Topics
