Trick to Simulate Higher Latency IOs on Linux

When running tests, it is sometimes useful to slow down IOs to clearly show the impact of reading from the disk.  Modern laptops and servers usually have fast local SSDs, and most cloud vendors provide, in addition to fast local disks, relatively fast network block devices (EBS for AWS and Persistent Disk for GCP).  Even when using magnetic network block devices, IOs are not guaranteed to be slow because they might hit the cache of the storage appliance (EBS volumes sometime have sub-millisecond latency).  This blog post describes the solution I am using for simulating high latency IOs on Linux. TL&DR: device-mapper’s delay target.