Troubleshooting Steps For When Your Hard Disk Is Giving You Trouble

When we’re talking about server performance, one of the more difficult issues we run into is the disk performance troubleshooting. While CPU load and memory usage can both can be monitored quite easily, disk overload can have load peaks that are hard to see and over time greatly affect overall server performance. Before you go deeper into your server storage system performance, it's a good idea to first look to the basics: does the server have sufficient free storage space and inodes count? That can be checked with the commands ‘df -h’ and ‘df -ih’ as shown below:

root@serversuit:~# df -h
Filesystem                 Size  Used Avail Use% Mounted on
rootfs                      30G  1.3G   27G   5% /
udev                        10M     0   10M   0% /dev
tmpfs                      101M  144K  101M   1% /run
/dev/disk/by-label/DOROOT   30G  1.3G   27G   5% /
tmpfs                      5.0M     0  5.0M   0% /run/lock
tmpfs                      201M     0  201M   0% /run/shm

root@serversuit:~# df -ih
Filesystem                Inodes IUsed IFree IUse% Mounted on
rootfs                      1.9M   36K  1.9M    2% /
udev                        124K   275  124K    1% /dev
tmpfs                       126K   195  126K    1% /run
/dev/disk/by-label/DOROOT   1.9M   36K  1.9M    2% /
tmpfs                       126K     1  126K    1% /run/lock
tmpfs                       126K     2  126K    1% /run/shm

ServerSuit also has a widget you can add to track disk space usage.

So, when is it a good time to check on your server disk performance? If get sudden lag spikes or high load average numbers on your server, or you can see a high wait average metric from your ‘top’ output, you should probably check advanced disk performance information. We can start with ‘iostat’ command at bash console (you’ll need the ‘sysstat’ package installed). Let’s look for the output:

root@serversuit:~# iostat -xcd
Linux 3.2.0-4-amd64 (serversuit)   04/28/2016      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          0.4 7   0.0   0.22    0.46     0.02     98.83

Device: rrqm/s  wrqm/s  r/s   w/s   rkB/s  wkB/s  avgrq-sz avgqu-sz await  svctm  %util
vda     20.70   47.44   0.91  12.01 26.18  237.91 40.89    2.35     181.49 7.76   10.03

This output will get you averaged stats from the time of boot of the server, which is not a silver bullet but can give you a basic understanding of what's been going on. To look for real-time data you’ll need to run ‘iostat -xcd -t 10’ command, which will return 10-seconds averages. You should pay attention to at least these metrics: rrqm/s - reads per second requested from your apps; wrqm/s - writes per second requested from the apps, which will give you your IOPS summary; r/s - actual reads from the storage device; w/s - actual writes from the storage device; await - average latency of all requests. If we’ll look at the numbers, we can draw some conclusions. Read requests are cached and were effectively merged: for 20.7 read IO requests only 1 IO request to the device was actually executed. Write requests were either random or can’t be cached, so for 47 write IO queries - 12 actual IO requests was executed. Average throughput was not huge, so there was probably some random IO and some low-level storage device underneath. Average latency is not good, so looks like we have a problem here. You’ll need to run ‘iostat’ with ‘-t’ key for some time to have real-time data about your storage load, so you can see IO peaks too, not just averages. When you have all the data you need, you can look for your current storage activity and probably have some graphs which will show you storage load peaks. It can be hard because you can’t say for sure which services are using disk resources with this tool. That’s where you can use the ‘iotop’ utility: this tool can help you to look for processes and their IO activity in real-time:

Total DISK READ:       0.00 B/s | Total DISK WRITE:      23.42 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  137 be/3 root        0.00 B/s   15.62 K/s  0.00 %  0.05 % [jbd2/vda1-8]
30040 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % ssh 
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init [2]
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
 2052 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % sh /usr/bin/mysqld_safe
    5 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u:0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]

Overall disk performance troubleshooting is relatively difficult for every system administrator, but hopefully this handy guide gave you some useful options for troubleshooting. As mentioned before, to ease your administrative burden ServerSuit has basic disk monitoring available. However, we're currently working a a feature set to fully troubleshoot disk performance, so look for that in an upcoming release in the near future! 

Until next time!