Here’s (roughly in order) what I would look for to help troubleshoot ongoing web server issues. Although lots of this stuff is pretty well known and basic, it’s good to have it written down as reference when you’re in a panic and not thinking straight.
Check some basic stuff
uptime who # who else is logged into the server currently last | head | sort -r # shows who the last logged in users were last <username> | head -n 20 | sort -r # shows last 20 times a specific user connected to the server
Check a user’s authentication / password expiration. This is helpful if a user can’t get into a box or if you have some crons not running under another user.
tail -f /var/log/secure tail -f /var/log/auth.log chage -l <username>
Check the logs
ls -ltr /var/log
Check for listening ports. You should probably have *:80, :443, *:22, *:25
sudo netstat -plntu
Check if you’re out of resources
htop free -m
Check for recently modified or created files
find /path -type f -mtime 5
Check disk space
df -h du -sh /path
Is a web server running?
service --status-all service httpd status ps aux | grep httpd sudo netstat -plunt | grep httpd
Can you curl a page on the affected machine from inside and outside the network?
curl https://website.com # try from a box inside and outside the network curl -kIL https://website.com # to see response / headers
Is there a file that is growing out of control?
find /path -size +1G -type f du -a / | sort -n -r | head -n 20 #Find largest directories of disk utilization
Check if new patches were applied
Is there a SAN that needs to be mounted?
cat /etc/fstab # check if there's anything that needs to be mounted mount # show all currently mounted file systems
Is there a database that needs to be reached?
ping databasehost.com mysql -h databasehost.com -u username -p
Check the crontab for something that just went off
crontab -l sudo crontab -u <username> -l
Are the permissions on resources set correctly?
ls -la /path