Linux 4.17 IO Scheduler Testing

Tests by Michael Larabel for a future article.

HTML result view exported from: https://openbenchmarking.org/result/1806021-AR-LINUX417I05&grw&sor.

ProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopCompilerFile-SystemScreen ResolutionHDDSSD BFQ BFQ low_latency CFQ DEADLINE Kyber MQ-DEADLINE Noop BFQ BFQ low_latency CFQ DEADLINE Kyber MQ-DEADLINE Noop2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads)TYAN S7106 (V1.00 BIOS)Intel Sky Lake-E DMI3 Registers12 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1256GB Samsung SSD 850 + 2000GB Seagate ST2000DM006-2DM1 + 2 x 120GB TOSHIBA-TR150ASPEED ASPEED FamilyVE228Intel I210 Gigabit ConnectionUbuntu 18.044.17.0-999-generic (x86_64) 20180529GNOME Shell 3.28.1GCC 7.3.0ext41920x1080OpenBenchmarking.orgCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-as=/usr/bin/x86_64-linux-gnu-as --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-ld=/usr/bin/x86_64-linux-gnu-ld --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v Disk Details- HDD: BFQ: BFQ / relatime,rw- HDD: BFQ low_latency: BFQ / relatime,rw- HDD: CFQ: CFQ / relatime,rw- HDD: DEADLINE: DEADLINE / relatime,rw- HDD: Kyber: KYBER / relatime,rw- HDD: MQ-DEADLINE: MQ-DEADLINE / relatime,rw- HDD: Noop: NOOP / relatime,rw- SSD: BFQ: BFQ / relatime,rw- SSD: BFQ low_latency: BFQ / relatime,rw- SSD: CFQ: CFQ / relatime,rw- SSD: DEADLINE: DEADLINE / relatime,rw- SSD: Kyber: KYBER / relatime,rw- SSD: MQ-DEADLINE: MQ-DEADLINE / relatime,rw- SSD: Noop: NOOP / relatime,rwProcessor Details- Scaling Governor: intel_pstate powersaveSecurity Details- KPTI + __user pointer sanitization + Full generic retpoline Protection

dbench: 12fio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfs-mark: 1000 Files, 1MB Sizefs-mark: 5000 Files, 1MB Size, 4 Threadsfs-mark: 4000 Files, 32 Sub Dirs, 1MB Sizeiozone: 1MB - 8GB - Write Performancesqlite: Timed SQLite Insertionspgbench: Buffer Test - Normal Load - Read Writestartup-time: Seq Reads + Writes - GNOME Terminalstartup-time: Only Seq Reads - GNOME TerminalHDDSSD BFQ BFQ low_latency CFQ DEADLINE Kyber MQ-DEADLINE Noop BFQ BFQ low_latency CFQ DEADLINE Kyber MQ-DEADLINE Noop30.431.571.0618817.4048.4017.37179.09705.15301.025.026.7529.601.571.0860.6817.2348.4017.30181.58790.19354.985.246.6730.531.571.0519617.4747.8717.33182.55769.04320.9646.075.4730.561.581.0519617.4048.6717.33184.40707.30245.0939.275.4827.481.561.0416817.3747.7317.37174.82698.82511.815.226.5127.811.561.0418517.3348.5317.37183.72716.51522.595.667.0430.491.551.0619817.3748.6717.40183.62708.38253.0456.565.48486.6823397.5341655.7349.4741.22102.5945.132301.405.236.86486.7121324041747.624740.7494.1850.681961.435.266.95560.4022428141658.0550.4041.4297.1241.592264.9640.885.50538.55227175.7841155.4547.6741.05103.7542.542289.3254.705.34517.2322418241255.8249.3340.9896.0044.511970.865.506.86547.1923617141456.2045.8035.8098.6942.941757.425.487.07570.9320613340051.1548.5340.98105.5943.442329.0953.685.23OpenBenchmarking.org

Dbench

Client Count: 12

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 12NoopCFQMQ-DEADLINEDEADLINEKyberBFQ low_latencyBFQ120240360480600SE +/- 23.68, N = 6SE +/- 23.57, N = 6SE +/- 16.76, N = 6SE +/- 21.64, N = 6SE +/- 14.65, N = 6SE +/- 14.01, N = 6SE +/- 11.71, N = 6SE +/- 0.59, N = 6SE +/- 0.20, N = 3SE +/- 0.44, N = 5SE +/- 0.55, N = 3SE +/- 0.31, N = 3SE +/- 0.38, N = 3SE +/- 0.41, N = 3570.93560.40547.19538.55517.23486.71486.6830.5630.5330.4930.4329.6027.8127.481. (CC) gcc options: -lpopt -O2

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryMQ-DEADLINEBFQDEADLINEKyberCFQBFQ low_latencyNoop50100150200250SE +/- 0.88, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3236.00233.00227.00224.00224.00213.00206.001.581.571.571.571.561.561.551. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryCFQBFQ low_latencyKyberDEADLINEMQ-DEADLINENoopBFQ60120180240300SE +/- 0.67, N = 3SE +/- 27.29, N = 6SE +/- 17.24, N = 6SE +/- 21.77, N = 6SE +/- 26.68, N = 6SE +/- 3.68, N = 6SE +/- 10.50, N = 6SE +/- 0.05, N = 6SE +/- 0.06, N = 6SE +/- 0.03, N = 6SE +/- 0.04, N = 6SE +/- 0.05, N = 6SE +/- 0.04, N = 6SE +/- 0.04, N = 6281.00240.00182.00175.78171.00133.0097.531.081.061.061.051.051.041.041. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQ low_latencyCFQBFQMQ-DEADLINEKyberDEADLINENoop90180270360450SE +/- 1.00, N = 3SE +/- 2.00, N = 3SE +/- 6.25, N = 5SE +/- 6.23, N = 6SE +/- 5.70, N = 6SE +/- 2.79, N = 6SE +/- 2.33, N = 3SE +/- 8.75, N = 6SE +/- 5.23, N = 6417.00416.00416.00414.00412.00411.00400.00198.00196.00196.00188.00185.00168.0060.681. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

FS-Mark

Test: 1000 Files, 1MB Size

SSDHDDOpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeCFQMQ-DEADLINEKyberBFQDEADLINENoopBFQ low_latency1326395265SE +/- 11.06, N = 6SE +/- 11.35, N = 6SE +/- 10.65, N = 6SE +/- 10.95, N = 6SE +/- 10.83, N = 6SE +/- 12.52, N = 6SE +/- 10.66, N = 6SE +/- 0.03, N = 3SE +/- 0.06, N = 3SE +/- 0.06, N = 3SE +/- 0.03, N = 3SE +/- 0.07, N = 3SE +/- 0.09, N = 3SE +/- 0.03, N = 358.0556.2055.8255.7355.4551.1547.6217.4717.4017.4017.3717.3717.3317.231. (CC) gcc options: -static

FS-Mark

Test: 5000 Files, 1MB Size, 4 Threads

SSDHDDOpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 5000 Files, 1MB Size, 4 ThreadsCFQBFQKyberNoopDEADLINEMQ-DEADLINEBFQ low_latency1122334455SE +/- 0.40, N = 3SE +/- 0.27, N = 3SE +/- 0.48, N = 3SE +/- 0.84, N = 6SE +/- 1.74, N = 6SE +/- 0.67, N = 3SE +/- 0.94, N = 6SE +/- 0.13, N = 3SE +/- 0.13, N = 3SE +/- 0.13, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.93, N = 3SE +/- 0.48, N = 350.4049.4749.3348.5347.6747.0045.8048.6748.6748.5348.4048.4047.8747.731. (CC) gcc options: -static

FS-Mark

Test: 4000 Files, 32 Sub Dirs, 1MB Size

SSDHDDOpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 4000 Files, 32 Sub Dirs, 1MB SizeCFQBFQDEADLINENoopKyberBFQ low_latencyMQ-DEADLINE918273645SE +/- 0.65, N = 6SE +/- 0.63, N = 5SE +/- 0.56, N = 6SE +/- 0.56, N = 6SE +/- 0.60, N = 5SE +/- 0.62, N = 5SE +/- 3.00, N = 6SE +/- 0.00, N = 3SE +/- 0.03, N = 3SE +/- 0.03, N = 3SE +/- 0.03, N = 3SE +/- 0.03, N = 3SE +/- 0.03, N = 3SE +/- 0.00, N = 341.4241.2241.0540.9840.9840.7435.8017.4017.3717.3717.3717.3317.3317.301. (CC) gcc options: -static

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

HDDSSDOpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write PerformanceDEADLINEMQ-DEADLINENoopCFQBFQ low_latencyBFQKyber4080120160200SE +/- 0.56, N = 3SE +/- 0.67, N = 3SE +/- 0.27, N = 3SE +/- 0.66, N = 3SE +/- 0.36, N = 3SE +/- 0.46, N = 3SE +/- 0.46, N = 3SE +/- 2.02, N = 6SE +/- 2.83, N = 6SE +/- 2.09, N = 6SE +/- 3.97, N = 6SE +/- 7.29, N = 6SE +/- 4.48, N = 6SE +/- 6.06, N = 6184.40183.72183.62182.55181.58179.09174.82105.59103.75102.5998.6997.1296.0094.181. (CC) gcc options: -O3

SQLite

Timed SQLite Insertions

SSDHDDOpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.22Timed SQLite InsertionsCFQDEADLINEMQ-DEADLINENoopKyberBFQBFQ low_latency2004006008001000SE +/- 1.02, N = 6SE +/- 1.06, N = 6SE +/- 1.15, N = 6SE +/- 1.30, N = 6SE +/- 1.30, N = 6SE +/- 0.75, N = 4SE +/- 1.37, N = 6SE +/- 10.48, N = 3SE +/- 11.30, N = 6SE +/- 27.02, N = 6SE +/- 7.08, N = 3SE +/- 2.21, N = 3SE +/- 60.15, N = 6SE +/- 27.29, N = 641.5942.5442.9443.4444.5145.1350.68698.82705.15707.30708.38716.51769.04790.191. (CC) gcc options: -O2 -lz -ldl -lpthread

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Write

SSDHDDOpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read WriteNoopBFQDEADLINECFQKyberBFQ low_latencyMQ-DEADLINE5001000150020002500SE +/- 13.76, N = 3SE +/- 55.20, N = 6SE +/- 16.17, N = 3SE +/- 8.35, N = 3SE +/- 62.04, N = 6SE +/- 39.09, N = 3SE +/- 94.63, N = 6SE +/- 8.51, N = 6SE +/- 15.05, N = 6SE +/- 30.15, N = 6SE +/- 4.46, N = 6SE +/- 4.63, N = 5SE +/- 2.69, N = 3SE +/- 2.62, N = 32329.092301.402289.322264.961970.861961.431757.42522.59511.81354.98320.96301.02253.04245.091. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -lcrypt -ldl -lm

Application Start-up Time

Background I/O Mix: Sequential Reads + Writes - Application To Start: GNOME Terminal

HDDSSDOpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.3.0Background I/O Mix: Sequential Reads + Writes - Application To Start: GNOME TerminalBFQKyberBFQ low_latencyMQ-DEADLINEDEADLINECFQNoop1326395265SE +/- 0.13, N = 6SE +/- 0.09, N = 4SE +/- 0.11, N = 3SE +/- 0.06, N = 3SE +/- 2.58, N = 6SE +/- 4.33, N = 6SE +/- 1.83, N = 6SE +/- 0.07, N = 3SE +/- 0.01, N = 3SE +/- 0.05, N = 3SE +/- 0.08, N = 3SE +/- 1.02, N = 6SE +/- 1.76, N = 6SE +/- 2.23, N = 65.025.225.245.6639.2746.0756.565.235.265.485.5040.8853.6854.70

Application Start-up Time

Background I/O Mix: Only Sequential Reads - Application To Start: GNOME Terminal

SSDHDDOpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.3.0Background I/O Mix: Only Sequential Reads - Application To Start: GNOME TerminalNoopDEADLINECFQKyberBFQ low_latencyBFQMQ-DEADLINE246810SE +/- 0.07, N = 3SE +/- 0.02, N = 3SE +/- 0.02, N = 3SE +/- 0.02, N = 3SE +/- 0.07, N = 3SE +/- 0.03, N = 3SE +/- 0.08, N = 3SE +/- 0.02, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.04, N = 3SE +/- 0.09, N = 3SE +/- 0.11, N = 6SE +/- 0.07, N = 35.235.345.506.866.866.957.075.475.485.486.516.676.757.04

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBFQ low_latencyCFQBFQMQ-DEADLINEKyberDEADLINENoop20K40K60K80K100KSE +/- 333.33, N = 3SE +/- 333.33, N = 3SE +/- 666.67, N = 3SE +/- 1641.46, N = 5SE +/- 1589.08, N = 6SE +/- 635.96, N = 3SE +/- 290.59, N = 3SE +/- 1472.64, N = 6SE +/- 681.99, N = 6SE +/- 554.78, N = 3SE +/- 2204.39, N = 6SE +/- 1342.80, N = 6106667106333106333106120106000105000102250505335026750000478674726742900155331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryMQ-DEADLINEBFQDEADLINECFQKyberBFQ low_latencyNoop13K26K39K52K65KSE +/- 208.17, N = 3SE +/- 33.33, N = 3605005970058000573005723354600527001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryCFQBFQ low_latencyKyberDEADLINEMQ-DEADLINENoopBFQ15K30K45K60K75KSE +/- 133.33, N = 3SE +/- 7128.44, N = 6SE +/- 4413.84, N = 6SE +/- 5582.04, N = 6SE +/- 6847.96, N = 6SE +/- 964.51, N = 6SE +/- 2655.36, N = 6720676130046600450504365033817247831. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Phoronix Test Suite v10.8.4