05112023-TCST PVE-NVME-R10-8DISK-ZFS2.2 Genoa PVE-NVME-R10-4DISK-ZFS2.1: Processor: AMD EPYC 7543 32-Core (12 Cores), Motherboard: QEMU Standard PC (Q35 + ICH9 2009) (3.20230228-4 BIOS), Chipset: Intel 82G33/G31/P35/P31 + ICH9, Memory: 16GB, Disk: 34GB QEMU HDD, Graphics: bochs-drmdrmfb, Audio: Intel 82801I, Monitor: QEMU Monitor, Network: Red Hat Virtio device OS: Ubuntu 22.04, Kernel: 5.15.0-86-generic (x86_64), Vulkan: 1.3.238, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 1280x800, System Layer: KVM PVE-NVME-R10-4DISK-ZFS2.2: Processor: AMD EPYC 7543 32-Core (12 Cores), Motherboard: QEMU Standard PC (Q35 + ICH9 2009) (3.20230228-4 BIOS), Chipset: Intel 82G33/G31/P35/P31 + ICH9, Memory: 16GB, Disk: 34GB QEMU HDD, Graphics: bochs-drmdrmfb, Audio: Intel 82801I, Monitor: QEMU Monitor, Network: Red Hat Virtio device OS: Ubuntu 22.04, Kernel: 5.15.0-88-generic (x86_64), Vulkan: 1.3.238, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 1280x800, System Layer: KVM PVE-NVME-R10-4DISK-LVM: Processor: AMD EPYC 7543 32-Core (12 Cores), Motherboard: QEMU Standard PC (Q35 + ICH9 2009) (3.20230228-4 BIOS), Chipset: Intel 82G33/G31/P35/P31 + ICH9, Memory: 16GB, Disk: 34GB QEMU HDD, Graphics: bochs-drmdrmfb, Audio: Intel 82801I, Monitor: QEMU Monitor, Network: Red Hat Virtio device OS: Ubuntu 22.04, Kernel: 5.15.0-88-generic (x86_64), Vulkan: 1.3.238, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 1280x800, System Layer: KVM PVE-NVME-R10-4DISK-LVM-THIN: Processor: AMD EPYC 9354 32-Core (12 Cores), Motherboard: QEMU Standard PC (Q35 + ICH9 2009) (4.2023.08-2 BIOS), Chipset: Intel 82G33/G31/P35/P31 + ICH9, Memory: 16GB, Disk: 34GB QEMU HDD, Graphics: bochs-drmdrmfb, Audio: Intel 82801I, Monitor: QEMU Monitor, Network: Red Hat Virtio device OS: Ubuntu 22.04, Kernel: 5.15.0-84-generic (x86_64), Vulkan: 1.3.238, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 1280x800, System Layer: KVM XEN-NVME-R10-4DISK-EXT4: Processor: AMD EPYC 9354 32-Core (12 Cores), Motherboard: Xen HVM domU v4.13, Chipset: Intel 440FX 82441FX PMC, Memory: 16GB, Disk: 30GB, Graphics: bochs-drmdrmfb OS: Ubuntu 22.04, Kernel: 5.15.0-89-generic (x86_64), Vulkan: 1.3.238, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 1024x768, System Layer: Xen HVM domU 4.13 PVE-NVME-R10-8DISK-ZFS2.2: Processor: AMD EPYC 9354 32-Core (16 Cores), Motherboard: QEMU Standard PC (Q35 + ICH9 2009) (4.2023.08-2 BIOS), Chipset: Intel 82G33/G31/P35/P31 + ICH9, Memory: 16GB, Disk: 34GB QEMU HDD, Graphics: bochs-drmdrmfb, Audio: Intel 82801I, Monitor: QEMU Monitor, Network: Red Hat Virtio device OS: Ubuntu 22.04, Kernel: 5.15.0-91-generic (x86_64), Vulkan: 1.3.238, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 1280x800, System Layer: KVM Flexible IO Tester 3.35 Type: Random Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 4033 |========= PVE-NVME-R10-4DISK-ZFS2.2 ... 3814 |======== PVE-NVME-R10-4DISK-LVM ...... 18808 |========================================== PVE-NVME-R10-4DISK-LVM-THIN . 19004 |========================================== XEN-NVME-R10-4DISK-EXT4 ..... 1638 |==== PVE-NVME-R10-8DISK-ZFS2.2 ... 8708 |=================== Flexible IO Tester 3.35 Type: Random Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 127708 |========================== PVE-NVME-R10-4DISK-ZFS2.2 ... 118691 |======================== PVE-NVME-R10-4DISK-LVM ...... 174062 |==================================== PVE-NVME-R10-4DISK-LVM-THIN . 200093 |========================================= XEN-NVME-R10-4DISK-EXT4 ..... 114841 |======================== PVE-NVME-R10-8DISK-ZFS2.2 ... 175996 |==================================== Flexible IO Tester 3.35 Type: Random Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 60526 |================== PVE-NVME-R10-4DISK-ZFS2.2 ... 55802 |================ PVE-NVME-R10-4DISK-LVM ...... 132418 |======================================= PVE-NVME-R10-4DISK-LVM-THIN . 140015 |========================================= XEN-NVME-R10-4DISK-EXT4 ..... 26468 |======== PVE-NVME-R10-8DISK-ZFS2.2 ... 111129 |================================= Flexible IO Tester 3.35 Type: Random Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 2554 |============= PVE-NVME-R10-4DISK-ZFS2.2 ... 2643 |============== PVE-NVME-R10-4DISK-LVM ...... 6922 |==================================== PVE-NVME-R10-4DISK-LVM-THIN . 6392 |================================= XEN-NVME-R10-4DISK-EXT4 ..... 1504 |======== PVE-NVME-R10-8DISK-ZFS2.2 ... 8225 |=========================================== Flexible IO Tester 3.35 Type: Random Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 88678 |======================= PVE-NVME-R10-4DISK-ZFS2.2 ... 88991 |======================= PVE-NVME-R10-4DISK-LVM ...... 117717 |=============================== PVE-NVME-R10-4DISK-LVM-THIN . 152386 |======================================== XEN-NVME-R10-4DISK-EXT4 ..... 110091 |============================= PVE-NVME-R10-8DISK-ZFS2.2 ... 156992 |========================================= Flexible IO Tester 3.35 Type: Random Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 31437 |============= PVE-NVME-R10-4DISK-ZFS2.2 ... 10978 |===== PVE-NVME-R10-4DISK-LVM ...... 97990 |========================================== PVE-NVME-R10-4DISK-LVM-THIN . 77542 |================================= XEN-NVME-R10-4DISK-EXT4 ..... 21766 |========= PVE-NVME-R10-8DISK-ZFS2.2 ... 88149 |====================================== Flexible IO Tester 3.35 Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 3804 |======== PVE-NVME-R10-4DISK-ZFS2.2 ... 3606 |======== PVE-NVME-R10-4DISK-LVM ...... 20038 |========================================== PVE-NVME-R10-4DISK-LVM-THIN . 19692 |========================================= XEN-NVME-R10-4DISK-EXT4 ..... 1487 |=== PVE-NVME-R10-8DISK-ZFS2.2 ... 6695 |============== Flexible IO Tester 3.35 Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 131558 |============================== PVE-NVME-R10-4DISK-ZFS2.2 ... 122330 |============================ PVE-NVME-R10-4DISK-LVM ...... 164873 |====================================== PVE-NVME-R10-4DISK-LVM-THIN . 178629 |========================================= XEN-NVME-R10-4DISK-EXT4 ..... 118749 |=========================== PVE-NVME-R10-8DISK-ZFS2.2 ... 173927 |======================================== Flexible IO Tester 3.35 Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 52119 |================ PVE-NVME-R10-4DISK-ZFS2.2 ... 48157 |============== PVE-NVME-R10-4DISK-LVM ...... 127996 |====================================== PVE-NVME-R10-4DISK-LVM-THIN . 136745 |========================================= XEN-NVME-R10-4DISK-EXT4 ..... 24210 |======= PVE-NVME-R10-8DISK-ZFS2.2 ... 85740 |========================== Flexible IO Tester 3.35 Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 2420 |============ PVE-NVME-R10-4DISK-ZFS2.2 ... 1378 |======= PVE-NVME-R10-4DISK-LVM ...... 6807 |=================================== PVE-NVME-R10-4DISK-LVM-THIN . 6109 |=============================== XEN-NVME-R10-4DISK-EXT4 ..... 1495 |======== PVE-NVME-R10-8DISK-ZFS2.2 ... 8430 |=========================================== Flexible IO Tester 3.35 Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 93001 |====================== PVE-NVME-R10-4DISK-ZFS2.2 ... 91556 |===================== PVE-NVME-R10-4DISK-LVM ...... 124815 |============================= PVE-NVME-R10-4DISK-LVM-THIN . 175268 |========================================= XEN-NVME-R10-4DISK-EXT4 ..... 126259 |============================== PVE-NVME-R10-8DISK-ZFS2.2 ... 161774 |====================================== Flexible IO Tester 3.35 Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory IOPS > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 27142 |=========== PVE-NVME-R10-4DISK-ZFS2.2 ... 10264 |==== PVE-NVME-R10-4DISK-LVM ...... 100259 |========================================= PVE-NVME-R10-4DISK-LVM-THIN . 89246 |==================================== XEN-NVME-R10-4DISK-EXT4 ..... 20852 |========= PVE-NVME-R10-8DISK-ZFS2.2 ... 89795 |===================================== Flexible IO Tester 3.35 Type: Random Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 4032.60 |======== PVE-NVME-R10-4DISK-ZFS2.2 ... 3814.21 |======== PVE-NVME-R10-4DISK-LVM ...... 18807.88 |======================================= PVE-NVME-R10-4DISK-LVM-THIN . 19003.62 |======================================= XEN-NVME-R10-4DISK-EXT4 ..... 1637.85 |=== PVE-NVME-R10-8DISK-ZFS2.2 ... 8708.00 |================== Flexible IO Tester 3.35 Type: Random Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 510833.62 |======================== PVE-NVME-R10-4DISK-ZFS2.2 ... 474764.96 |======================= PVE-NVME-R10-4DISK-LVM ...... 696247.30 |================================= PVE-NVME-R10-4DISK-LVM-THIN . 800373.64 |====================================== XEN-NVME-R10-4DISK-EXT4 ..... 459364.69 |====================== PVE-NVME-R10-8DISK-ZFS2.2 ... 703983.63 |================================= Flexible IO Tester 3.35 Type: Random Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 3782.85 |================= PVE-NVME-R10-4DISK-ZFS2.2 ... 3487.64 |================ PVE-NVME-R10-4DISK-LVM ...... 8276.15 |====================================== PVE-NVME-R10-4DISK-LVM-THIN . 8750.94 |======================================== XEN-NVME-R10-4DISK-EXT4 ..... 1654.23 |======== PVE-NVME-R10-8DISK-ZFS2.2 ... 6945.55 |================================ Flexible IO Tester 3.35 Type: Random Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 2554.28 |============ PVE-NVME-R10-4DISK-ZFS2.2 ... 2643.08 |============= PVE-NVME-R10-4DISK-LVM ...... 6922.49 |================================== PVE-NVME-R10-4DISK-LVM-THIN . 6392.34 |=============================== XEN-NVME-R10-4DISK-EXT4 ..... 1504.33 |======= PVE-NVME-R10-8DISK-ZFS2.2 ... 8225.43 |======================================== Flexible IO Tester 3.35 Type: Random Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 354712.28 |===================== PVE-NVME-R10-4DISK-ZFS2.2 ... 355963.94 |====================== PVE-NVME-R10-4DISK-LVM ...... 470866.56 |============================ PVE-NVME-R10-4DISK-LVM-THIN . 609544.81 |===================================== XEN-NVME-R10-4DISK-EXT4 ..... 440366.12 |=========================== PVE-NVME-R10-8DISK-ZFS2.2 ... 627966.71 |====================================== Flexible IO Tester 3.35 Type: Random Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 1964.79 | PVE-NVME-R10-4DISK-ZFS2.2 ... 702583.68 |====================================== PVE-NVME-R10-4DISK-LVM ...... 6124.38 | PVE-NVME-R10-4DISK-LVM-THIN . 4846.36 | XEN-NVME-R10-4DISK-EXT4 ..... 1360.37 | PVE-NVME-R10-8DISK-ZFS2.2 ... 5509.34 | Flexible IO Tester 3.35 Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 3803.88 |======= PVE-NVME-R10-4DISK-ZFS2.2 ... 3606.58 |======= PVE-NVME-R10-4DISK-LVM ...... 20037.94 |======================================= PVE-NVME-R10-4DISK-LVM-THIN . 19691.68 |====================================== XEN-NVME-R10-4DISK-EXT4 ..... 1486.96 |=== PVE-NVME-R10-8DISK-ZFS2.2 ... 6695.37 |============= Flexible IO Tester 3.35 Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 526231.73 |============================ PVE-NVME-R10-4DISK-ZFS2.2 ... 489319.73 |========================== PVE-NVME-R10-4DISK-LVM ...... 659491.14 |=================================== PVE-NVME-R10-4DISK-LVM-THIN . 714517.97 |====================================== XEN-NVME-R10-4DISK-EXT4 ..... 474996.81 |========================= PVE-NVME-R10-8DISK-ZFS2.2 ... 695706.62 |===================================== Flexible IO Tester 3.35 Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 3257.47 |=============== PVE-NVME-R10-4DISK-ZFS2.2 ... 3009.81 |============== PVE-NVME-R10-4DISK-LVM ...... 7999.78 |===================================== PVE-NVME-R10-4DISK-LVM-THIN . 8546.58 |======================================== XEN-NVME-R10-4DISK-EXT4 ..... 1513.12 |======= PVE-NVME-R10-8DISK-ZFS2.2 ... 5358.76 |========================= Flexible IO Tester 3.35 Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 1MB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 2419.83 |=========== PVE-NVME-R10-4DISK-ZFS2.2 ... 1378.71 |======= PVE-NVME-R10-4DISK-LVM ...... 6807.31 |================================ PVE-NVME-R10-4DISK-LVM-THIN . 6108.63 |============================= XEN-NVME-R10-4DISK-EXT4 ..... 1494.97 |======= PVE-NVME-R10-8DISK-ZFS2.2 ... 8429.73 |======================================== Flexible IO Tester 3.35 Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 4KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 372002.83 |==================== PVE-NVME-R10-4DISK-ZFS2.2 ... 366223.52 |==================== PVE-NVME-R10-4DISK-LVM ...... 499262.03 |=========================== PVE-NVME-R10-4DISK-LVM-THIN . 701071.06 |====================================== XEN-NVME-R10-4DISK-EXT4 ..... 505037.27 |=========================== PVE-NVME-R10-8DISK-ZFS2.2 ... 647097.87 |=================================== Flexible IO Tester 3.35 Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: No - Block Size: 64KB - Job Count: 1 - Disk Target: Default Test Directory MiB/s > Higher Is Better PVE-NVME-R10-4DISK-ZFS2.1 ... 1696.39 | PVE-NVME-R10-4DISK-ZFS2.2 ... 656928.12 |====================================== PVE-NVME-R10-4DISK-LVM ...... 6266.22 | PVE-NVME-R10-4DISK-LVM-THIN . 5577.91 | XEN-NVME-R10-4DISK-EXT4 ..... 1303.25 | PVE-NVME-R10-8DISK-ZFS2.2 ... 5612.17 |