Linux 4.20 IO Scheduler Tests

Linux 4.20 I/O scheduler tests by Michael Larabel on NVMe SSD storage for a future article on Phoronix.

HTML result view exported from: https://openbenchmarking.org/result/1812038-SK-LINUX420I24&obr_sor=y&obr_rro=y&rdt&grt.

Linux 4.20 IO Scheduler TestsProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerOpenGLVulkanCompilerFile-SystemScreen ResolutionNoneMQ DeadlineKyberBFQ low_latencyBFQAMD Ryzen Threadripper 2990WX 32-Core @ 3.00GHz (32 Cores / 64 Threads)ASUS ROG ZENITH EXTREME (1501 BIOS)AMD Family 17h4 x 8192 MB DDR4-1600MT/s F4-3200C14-8GFXSamsung SSD 970 EVO 500GBAMD Radeon RX Vega 8GBRealtek ALC1220ASUS VP28UIntel I211 Gigabit Connection + Qualcomm Atheros QCA6174 802.11ac WirelessUbuntu 18.104.20.0-999-generic (x86_64) 20181201GNOME Shell 3.30.1X Server4.5 Mesa 18.2.2 (LLVM 7.0.0)1.1.70GCC 8.2.0ext43840x2160OpenBenchmarking.orgCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v Disk Details- None: NONE / errors=remount-ro,relatime,rw- MQ Deadline: MQ-DEADLINE / errors=remount-ro,relatime,rw- Kyber: KYBER / errors=remount-ro,relatime,rw- BFQ low_latency: BFQ / errors=remount-ro,relatime,rw- BFQ: BFQ / errors=remount-ro,relatime,rwProcessor Details- Scaling Governor: acpi-cpufreq ondemandSecurity Details- __user pointer sanitization + Full AMD retpoline IBPB: conditional STIBP: disabled RSB filling + SSB disabled via prctl and seccomp

Linux 4.20 IO Scheduler Testsstartup-time: Only Seq Reads - xterm - Default Test Directorystartup-time: Seq Reads + Writes - xterm - Default Test Directorystartup-time: Only Seq Reads - GNOME Terminal - Default Test Directorystartup-time: Only Seq Reads - LibreOffice Writer - Default Test Directorystartup-time: Seq Reads + Writes - GNOME Terminal - Default Test Directorystartup-time: Seq Reads + Writes - LibreOffice Writer - Default Test Directoryblogbench: Readblogbench: Writefio: Rand Read - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfs-mark: 1000 Files, 1MB Sizefs-mark: 5000 Files, 1MB Size, 4 Threadsfs-mark: 4000 Files, 32 Sub Dirs, 1MB Sizefs-mark: 1000 Files, 1MB Size, No Sync/FSyncpgbench: Buffer Test - Normal Load - Read Writesqlite: Timed SQLite InsertionsNoneMQ DeadlineKyberBFQ low_latencyBFQ0.540.752.131.773.222.497266381431833226662362101110309562402102223420396.032320428535.950.541.122.171.734.443.4092902112993336276423698861034962240093922414991.912319438137.950.541.062.111.794.593.5292929313856336275423699441029953239494922715284.282315460937.960.170.180.300.250.550.34488353191353342716237171210337612401712232216125.082305547736.210.551.122.241.784.453.17475934182393368752236571010337802394721229244124.662289515137.31OpenBenchmarking.org

Application Start-up Time

Background I/O Mix: Only Sequential Reads - Application To Start: xterm - Disk Target: Default Test Directory

OpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.4.0Background I/O Mix: Only Sequential Reads - Application To Start: xterm - Disk Target: Default Test DirectoryBFQKyberMQ DeadlineNoneBFQ low_latency0.12380.24760.37140.49520.619SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 9SE +/- 0.00, N = 3SE +/- 0.00, N = 90.550.540.540.540.17

Application Start-up Time

Background I/O Mix: Sequential Reads + Writes - Application To Start: xterm - Disk Target: Default Test Directory

OpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.4.0Background I/O Mix: Sequential Reads + Writes - Application To Start: xterm - Disk Target: Default Test DirectoryBFQMQ DeadlineKyberNoneBFQ low_latency0.2520.5040.7561.0081.26SE +/- 0.06, N = 6SE +/- 0.07, N = 9SE +/- 0.02, N = 3SE +/- 0.01, N = 5SE +/- 0.01, N = 91.121.121.060.750.18

Application Start-up Time

Background I/O Mix: Only Sequential Reads - Application To Start: GNOME Terminal - Disk Target: Default Test Directory

OpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.4.0Background I/O Mix: Only Sequential Reads - Application To Start: GNOME Terminal - Disk Target: Default Test DirectoryBFQMQ DeadlineNoneKyberBFQ low_latency0.5041.0081.5122.0162.52SE +/- 0.02, N = 3SE +/- 0.04, N = 3SE +/- 0.03, N = 9SE +/- 0.03, N = 3SE +/- 0.00, N = 32.242.172.132.110.30

Application Start-up Time

Background I/O Mix: Only Sequential Reads - Application To Start: LibreOffice Writer - Disk Target: Default Test Directory

OpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.4.0Background I/O Mix: Only Sequential Reads - Application To Start: LibreOffice Writer - Disk Target: Default Test DirectoryKyberBFQNoneMQ DeadlineBFQ low_latency0.40280.80561.20841.61122.014SE +/- 0.00, N = 3SE +/- 0.03, N = 5SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 31.791.781.771.730.25

Application Start-up Time

Background I/O Mix: Sequential Reads + Writes - Application To Start: GNOME Terminal - Disk Target: Default Test Directory

OpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.4.0Background I/O Mix: Sequential Reads + Writes - Application To Start: GNOME Terminal - Disk Target: Default Test DirectoryKyberBFQMQ DeadlineNoneBFQ low_latency1.03282.06563.09844.13125.164SE +/- 0.21, N = 9SE +/- 0.16, N = 9SE +/- 0.08, N = 3SE +/- 0.03, N = 3SE +/- 0.12, N = 94.594.454.443.220.55

Application Start-up Time

Background I/O Mix: Sequential Reads + Writes - Application To Start: LibreOffice Writer - Disk Target: Default Test Directory

OpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.4.0Background I/O Mix: Sequential Reads + Writes - Application To Start: LibreOffice Writer - Disk Target: Default Test DirectoryKyberMQ DeadlineBFQNoneBFQ low_latency0.7921.5842.3763.1683.96SE +/- 0.14, N = 9SE +/- 0.06, N = 9SE +/- 0.09, N = 9SE +/- 0.04, N = 3SE +/- 0.02, N = 93.523.403.172.490.34

BlogBench

Test: Read

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.0Test: ReadBFQBFQ low_latencyNoneMQ DeadlineKyber200K400K600K800K1000KSE +/- 8443.11, N = 9SE +/- 13128.49, N = 9SE +/- 16687.41, N = 9SE +/- 55608.68, N = 9SE +/- 56002.71, N = 94759344883537266389290219292931. (CC) gcc options: -O2 -pthread

BlogBench

Test: Write

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.0Test: WriteMQ DeadlineKyberNoneBFQBFQ low_latency4K8K12K16K20KSE +/- 940.27, N = 3SE +/- 871.74, N = 3SE +/- 403.05, N = 3SE +/- 188.75, N = 3SE +/- 406.72, N = 312993138561431818239191351. (CC) gcc options: -O2 -pthread

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryNoneBFQ low_latencyMQ DeadlineKyberBFQ7001400210028003500SE +/- 16.84, N = 3SE +/- 12.86, N = 3SE +/- 9.71, N = 3SE +/- 18.46, N = 3SE +/- 23.46, N = 3332233423362336233681. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryNoneBFQ low_latencyBFQKyberMQ Deadline160320480640800SE +/- 1.67, N = 3SE +/- 2.91, N = 3SE +/- 1.00, N = 3SE +/- 6.23, N = 3SE +/- 1.86, N = 36667167527547641. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryNoneBFQ low_latencyBFQKyberMQ Deadline40K80K120K160K200KSE +/- 333.33, N = 3SE +/- 881.92, N = 3SE +/- 333.33, N = 3SE +/- 1527.53, N = 3SE +/- 333.33, N = 31703331836671926671930001956671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryNoneBFQMQ DeadlineKyberBFQ low_latency5001000150020002500SE +/- 2.52, N = 3SE +/- 2.08, N = 3SE +/- 2.08, N = 3SE +/- 2.85, N = 3236223652369236923711. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQBFQ low_latencyMQ DeadlineKyberNone2004006008001000SE +/- 3.79, N = 3SE +/- 4.73, N = 3SE +/- 7.02, N = 3SE +/- 4.04, N = 3SE +/- 16.18, N = 371071288694410111. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQBFQ low_latencyMQ DeadlineKyberNone60K120K180K240K300KSE +/- 1201.85, N = 3SE +/- 1201.85, N = 3SE +/- 1666.67, N = 3SE +/- 1000.00, N = 3SE +/- 4163.33, N = 31816671823332266672420002590001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryKyberNoneBFQ low_latencyBFQMQ Deadline2004006008001000SE +/- 0.33, N = 3SE +/- 0.33, N = 3SE +/- 1.00, N = 3102910301033103310341. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQ low_latencyBFQKyberNoneMQ Deadline2004006008001000SE +/- 12.46, N = 12SE +/- 2.40, N = 3SE +/- 10.12, N = 3SE +/- 1.53, N = 3SE +/- 9.54, N = 37617809539569621. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQ low_latencyBFQKyberNoneMQ Deadline50K100K150K200K250KSE +/- 3254.66, N = 12SE +/- 666.67, N = 3SE +/- 2603.42, N = 3SE +/- 333.33, N = 3SE +/- 2516.61, N = 31947501993332436672446672460001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryKyberBFQMQ DeadlineBFQ low_latencyNone5001000150020002500SE +/- 4.06, N = 3SE +/- 0.88, N = 3SE +/- 0.67, N = 3SE +/- 1.00, N = 3239423942400240124021. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQ low_latencyBFQMQ DeadlineKyberNone2004006008001000SE +/- 10.12, N = 12SE +/- 4.81, N = 3SE +/- 0.88, N = 3SE +/- 14.53, N = 3SE +/- 11.02, N = 371272193994910221. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQ low_latencyBFQMQ DeadlineKyberNone60K120K180K240K300KSE +/- 2571.58, N = 12SE +/- 1201.85, N = 3SE +/- 333.33, N = 3SE +/- 3785.94, N = 3SE +/- 2728.45, N = 31824171846672403332430002616671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lgfapi -lglusterfs -lnuma -libverbs -lrt -laio -lz -lm -lpthread -ldl

FS-Mark

Test: 1000 Files, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeMQ DeadlineKyberBFQBFQ low_latencyNone50100150200250SE +/- 3.62, N = 3SE +/- 2.10, N = 3SE +/- 0.96, N = 3SE +/- 1.22, N = 3SE +/- 0.81, N = 32242272292322341. (CC) gcc options: -static

FS-Mark

Test: 5000 Files, 1MB Size, 4 Threads

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 5000 Files, 1MB Size, 4 ThreadsMQ DeadlineKyberNoneBFQ low_latencyBFQ50100150200250SE +/- 14.88, N = 6SE +/- 19.67, N = 9SE +/- 13.88, N = 11SE +/- 4.91, N = 12SE +/- 7.23, N = 121491522032162441. (CC) gcc options: -static

FS-Mark

Test: 4000 Files, 32 Sub Dirs, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 4000 Files, 32 Sub Dirs, 1MB SizeKyberMQ DeadlineNoneBFQBFQ low_latency306090120150SE +/- 2.72, N = 12SE +/- 5.20, N = 12SE +/- 6.11, N = 12SE +/- 7.44, N = 12SE +/- 7.06, N = 1284.2891.9196.03124.66125.081. (CC) gcc options: -static

FS-Mark

Test: 1000 Files, 1MB Size, No Sync/FSync

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB Size, No Sync/FSyncBFQBFQ low_latencyKyberMQ DeadlineNone5001000150020002500SE +/- 4.67, N = 3SE +/- 9.10, N = 3SE +/- 7.14, N = 3SE +/- 6.99, N = 3SE +/- 5.99, N = 3228923052315231923201. (CC) gcc options: -static

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read WriteNoneMQ DeadlineKyberBFQBFQ low_latency12002400360048006000SE +/- 66.90, N = 12SE +/- 27.41, N = 3SE +/- 94.23, N = 12SE +/- 72.89, N = 5SE +/- 93.38, N = 3428543814609515154771. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -lcrypt -ldl -lm

SQLite

Timed SQLite Insertions

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.22Timed SQLite InsertionsKyberMQ DeadlineBFQBFQ low_latencyNone918273645SE +/- 0.24, N = 3SE +/- 0.22, N = 3SE +/- 0.03, N = 3SE +/- 0.06, N = 3SE +/- 0.07, N = 337.9637.9537.3136.2135.951. (CC) gcc options: -O2 -lz -ldl -lpthread

Phoronix Test Suite v10.8.5