Troubleshooting web servers

July 3, 2018

Here’s (roughly in order) what I would look for to help troubleshoot ongoing web server issues. Although lots of this stuff is pretty well known and basic, it’s good to have it written down as reference when you’re in a panic and not thinking straight.

Check some basic stuff

who # who else is logged into the server currently
last | head | sort -r # shows who the last logged in users were
last <username> | head -n 20 | sort -r # shows last 20 times a specific user connected to the server

Check a user’s authentication / password expiration. This is helpful if a user can’t get into a box or if you have some crons not running under another user.

tail -f /var/log/secure
tail -f /var/log/auth.log
chage -l <username>

Check the logs

ls -ltr /var/log

Check for listening ports. You should probably have *:80, :443, *:22, *:25

sudo netstat -plntu

Check if you’re out of resources

free -m

Check for recently modified or created files

find /path -type f -mtime 5

Check disk space

df -h
du -sh /path

Is a web server running?

service --status-all
service httpd status
ps aux | grep httpd
sudo netstat -plunt | grep httpd

Can you curl a page on the affected machine from inside and outside the network?

curl # try from a box inside and outside the network
curl -kIL # to see response / headers

Is there a file that is growing out of control?

find /path -size +1G -type f
du -a / | sort -n -r | head -n 20 #Find largest directories of disk utilization

Check if new patches were applied

less /var/log/yum.log

Is there a SAN that needs to be mounted?

cat /etc/fstab # check if there's anything that needs to be mounted
mount # show all currently mounted file systems

Is there a database that needs to be reached?

mysql -h -u username -p

Check the crontab for something that just went off

crontab -l
sudo crontab -u <username> -l

Are the permissions on resources set correctly?

ls -la /path