Linux: Slow response time using sshd to log in

The MOS document "Slow Login Response From sshd" via article ID 2000698.1 was not what I thought it would be, as I have experienced slow logins with ssh when certain events happen such as server load or network/firewall issues, but I had not figured in how DNS (or RDNS) may take out our ability to log into a server or set of servers.  Read up on this as you never know when it could help you out some day!

How the NTP service can be your weakest link

If you are not familiar with the Network Time Protocol and the services related to supporting it, do not feel bad; I did not know either until we had a problem with it!  Why could this be your weakest link?  I am glad you asked!  Recently we had our Exadata DBs start reporting different server times because it seemed that they were getting the server time from their switches but the switches were not correctly getting their time from our network time server.  That is obviously a problem, but imagine my surprise when I tried to log in and received "Permission denied, please try again" when trying to SSH into our server.  Why does this relate to NTP?  Well, it seems that if the time drifts too much on a server the SSH login attempt is rejected!

Why am I making a big deal out of this?  What happens when you are alerted that your CPU is starting to spike and you want to log in to take a look, but are rejected?  Do you have time to waste on getting a system administrator to restart the ntpd service so you can get into your machine?  I doubt it!  Now, it is on to making an alert to check our system time against the network time server!