Disk scheduler for EqualLogic SAN and MySQL

by Martijn Otto ,  Emiel Bruijntjes

At Copernica we use a cluster of Dell EqualLogic SAN devices as storage backend for our database servers. To get the best possible performance from MySQL running on a SAN we have spent quite some time on configuring the database servers for low latency and high throughput - and we discovered that the official Dell advice to use the noop disk scheduler does not always give the best performance

This disk scheduler is an OS level configuration setting that determines what exactly happens when a read or write request is sent to a block device. Linux has a choice offering of three schedulers:

  • cfq (completely fair queue)
  • noop (no operation)
  • deadline

What is the difference between these schedulers?

The cfq scheduler is a scheduler that gives processes equal access to a block device. If one process is completely hogging a disk, and another process tries to access the same disk, the cfq scheduler will make sure they both get their fair share by throttling the most active process.

The noop scheduler is the simplest scheduler, since it will just forward the requests in the order it receives them. It is also the scheduler that uses the least CPU-time.

The deadline scheduler is the most complex scheduler linux has. It queues incoming read and write requests up until a certain deadline. When the deadline is reached, it will merge all the requests it has in the queue based on their position on disk and sends out fewer, but bigger, IO requests.

Why are these schedulers relevant?

Consider a simple, old-fashioned spinning hard disk. When it receives a request to retrieve or store a block, it needs to position the head in the correct position and wait until the disk has rotated to the right place. It can only do this so many times a second (for old hard disks, this is about 100 times per second, newer ones get about 200). If the drive receives more than that, the requests get queued, at which point performance degrades.

When requests get merged and ordered, however, the drive needs to do less seek operations to get all the requested data out. In addition, because the requests are lined up, the drive need not wait a whole round to get the next block. This increases the overall throughput at the expense of some CPU-time and some added latency.

Based on this, the deadline scheduler sounds the best, why is it not the default everybody uses?

In some cases, request merging and ordering does not make sense. One example of this are SSD devices. Since they have no moving parts, they do not have to rotate (seek) to the requested block. Retrieving three sequential blocks takes the same time as retrieving three blocks scattered over the entire disk. Another example are high-end RAID-controllers that have these merging capabilities built-in.

When it comes to desktop computers, the cfq scheduler is nice because it guarantees that none of your applications will be completely starved by another, creating a more responsive desktop.

The Dell EqualLogic and schedulers

The Dell EqualLogic has a controller that intelligently routes requests to different hard disks or SSDs by itselves. It also does request merging and reordering. For this reason, Dell recommends to use the 'noop' scheduler at the OS when working with the EqualLogic. This makes sense as queueing and merging requests twice (by the OS and by the SAN) only introduces extra latency.

In practice, this seems to work well until you reach about 1500 to 2000 IOPS (input/output operations per second). When you exceed this number, the controller starts to have trouble handling all the requests and becomes a bottleneck.

Working on improving our database performance, we ignored the official advice from Dell and switched to the deadline scheduler instead and noticed an immediate improvicement. The IO-rates went up as the IOPS went down. This reduced the strain on the EqualLogic controller causing requests to receive a response much quicker, which more than makes up for the added latency introduced by switching to the deadline scheduler.

In our case, the IO throughput has more than doubled and no adverse reactions to this change have been detected. For us, the deadline scheduler is the best scheduler to use when running a MySQL database on a SAN.

How to change the IO scheduler?

You can find out what scheduler is now in place by reading out the "file" /sys/block/$block/queue/scheduler. The scheduler in use is displayed between square brackets. To change the scheduler on a running system, simply write the scheduler name to this file, for example by executing the following command:

echo deadline | sudo tee /sys/block/$block/queue/scheduler



Where you change $block with the name of the block device you are trying to update the scheduler for (e.g. sda)

However, you will need to enter this command every time that the server reboots. To make the setting persist across boots as well, the easiest way is to create a udev rule. Place the following in a file under /etc/udev/rules.d/

ACTION=="add|change", SUBSYSTEM=="block", ATTRS{vendor}=="EQLOGIC", ATTR{queue/scheduler}="deadline"


To immediately trigger these udev rules without a reboot, you can run the following:

sudo udevadm control --reload-rules
sudo udevadm trigger