AMD EPYC Turin AI/ML Tuning Guide AMD EPYC 9655P following AMD tuning guide for AI/ML workloads - https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/58467_amd-epyc-9005-tg-bios-and-workload.pdf Benchmarks by Michael Larabel for a future article. Stock: Processor: AMD EPYC 9655P 96-Core @ 2.60GHz (96 Cores / 192 Threads), Motherboard: Supermicro Super Server H13SSL-N v1.01 (3.0 BIOS), Chipset: AMD 1Ah, Memory: 12 x 64GB DDR5-6000MT/s Micron MTC40F2046S1RC64BDY QSFF, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Broadcom NetXtreme BCM5720 PCIe OS: Ubuntu 24.10, Kernel: 6.12.0-rc7-linux-pm-next-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 1024x768 AI/ML Tuning Recommendations: Processor: AMD EPYC 9655P 96-Core @ 2.60GHz (96 Cores / 192 Threads), Motherboard: Supermicro Super Server H13SSL-N v1.01 (3.0 BIOS), Chipset: AMD 1Ah, Memory: 12 x 64GB DDR5-6000MT/s Micron MTC40F2046S1RC64BDY QSFF, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Broadcom NetXtreme BCM5720 PCIe OS: Ubuntu 24.10, Kernel: 6.12.0-rc7-linux-pm-next-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 1024x768 LiteRT 2024-10-15 Model: Mobilenet Float Microseconds < Lower Is Better Stock ........................ 4438.52 |======================================= AI/ML Tuning Recommendations . 4335.67 |====================================== LiteRT 2024-10-15 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 236 MAX: 263 AI/ML Tuning Recommendations . MIN: 124 AVG: 283 MAX: 304 LiteRT 2024-10-15 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 102 AVG: 395 MAX: 407 AI/ML Tuning Recommendations . MIN: 100 AVG: 421 MAX: 461 LiteRT 2024-10-15 Model: NASNet Mobile Microseconds < Lower Is Better Stock ........................ 733737 |======================================== AI/ML Tuning Recommendations . 689396 |====================================== LiteRT 2024-10-15 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 4 AVG: 221 MAX: 239 AI/ML Tuning Recommendations . MIN: 99 AVG: 261 MAX: 280 LiteRT 2024-10-15 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 97 AVG: 339 MAX: 365 AI/ML Tuning Recommendations . MIN: 98 AVG: 391 MAX: 415 LiteRT 2024-10-15 Model: SqueezeNet Microseconds < Lower Is Better Stock ........................ 7035.74 |======================================= AI/ML Tuning Recommendations . 6926.88 |====================================== LiteRT 2024-10-15 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 4 AVG: 237 MAX: 264 AI/ML Tuning Recommendations . MIN: 87 AVG: 280 MAX: 304 LiteRT 2024-10-15 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 373 MAX: 408 AI/ML Tuning Recommendations . MIN: 100 AVG: 442 MAX: 461 LiteRT 2024-10-15 Model: Inception V4 Microseconds < Lower Is Better Stock ........................ 43898.9 |======================================= AI/ML Tuning Recommendations . 43824.9 |======================================= LiteRT 2024-10-15 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 244 MAX: 273 AI/ML Tuning Recommendations . MIN: 101 AVG: 304 MAX: 326 LiteRT 2024-10-15 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 400 MAX: 424 AI/ML Tuning Recommendations . MIN: 98 AVG: 454 MAX: 496 Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better Stock ........................ 45.84 |======================================== AI/ML Tuning Recommendations . 46.45 |========================================= Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 236 MAX: 335 AI/ML Tuning Recommendations . MIN: 57 AVG: 321 MAX: 401 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 390 MAX: 558 AI/ML Tuning Recommendations . MIN: 99 AVG: 536 MAX: 650 Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better Stock ........................ 72.99 |======================================= AI/ML Tuning Recommendations . 76.41 |========================================= Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 7 AVG: 222 MAX: 264 AI/ML Tuning Recommendations . MIN: 127 AVG: 295 MAX: 333 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 366 MAX: 444 AI/ML Tuning Recommendations . MIN: 100 AVG: 456 MAX: 526 Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better Stock ........................ 96.99 |====================================== AI/ML Tuning Recommendations . 101.77 |======================================== Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 17 AVG: 232 MAX: 261 AI/ML Tuning Recommendations . MIN: 123 AVG: 289 MAX: 326 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 383 MAX: 438 AI/ML Tuning Recommendations . MIN: 98 AVG: 451 MAX: 514 Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better Stock ........................ 144.53 |====================================== AI/ML Tuning Recommendations . 152.92 |======================================== Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 11 AVG: 248 MAX: 290 AI/ML Tuning Recommendations . MIN: 114 AVG: 288 MAX: 316 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 410 MAX: 502 AI/ML Tuning Recommendations . MIN: 100 AVG: 454 MAX: 544 Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better Stock ........................ 92.82 |======================================== AI/ML Tuning Recommendations . 95.42 |========================================= Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 11 AVG: 165 MAX: 270 AI/ML Tuning Recommendations . MIN: 86 AVG: 203 MAX: 305 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 97 AVG: 256 MAX: 411 AI/ML Tuning Recommendations . MIN: 100 AVG: 324 MAX: 465 Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better Stock ........................ 154.60 |======================================== AI/ML Tuning Recommendations . 155.73 |======================================== Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 15 AVG: 244 MAX: 300 AI/ML Tuning Recommendations . MIN: 97 AVG: 292 MAX: 350 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 371 MAX: 450 AI/ML Tuning Recommendations . MIN: 97 AVG: 434 MAX: 515 Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better Stock ........................ 306.51 |======================================== AI/ML Tuning Recommendations . 308.32 |======================================== Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 13 AVG: 255 MAX: 302 AI/ML Tuning Recommendations . MIN: 96 AVG: 307 MAX: 351 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 411 MAX: 500 AI/ML Tuning Recommendations . MIN: 98 AVG: 468 MAX: 534 Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better Stock ........................ 48.07 |======================================== AI/ML Tuning Recommendations . 48.73 |========================================= Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 6 AVG: 239 MAX: 332 AI/ML Tuning Recommendations . MIN: 57 AVG: 293 MAX: 400 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 428 MAX: 552 AI/ML Tuning Recommendations . MIN: 99 AVG: 462 MAX: 644 Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better Stock ........................ 72.40 |======================================= AI/ML Tuning Recommendations . 77.05 |========================================= Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 10 AVG: 224 MAX: 270 AI/ML Tuning Recommendations . MIN: 95 AVG: 289 MAX: 335 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 372 MAX: 415 AI/ML Tuning Recommendations . MIN: 100 AVG: 442 MAX: 513 Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better Stock ........................ 97.19 |====================================== AI/ML Tuning Recommendations . 101.71 |======================================== Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 232 MAX: 266 AI/ML Tuning Recommendations . MIN: 92 AVG: 285 MAX: 338 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 375 MAX: 447 AI/ML Tuning Recommendations . MIN: 99 AVG: 445 MAX: 524 Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better Stock ........................ 149.00 |======================================== AI/ML Tuning Recommendations . 150.29 |======================================== Llama.cpp b4154 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 248 MAX: 291 AI/ML Tuning Recommendations . MIN: 86 AVG: 288 MAX: 323 Llama.cpp b4154 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 416 MAX: 507 AI/ML Tuning Recommendations . MIN: 99 AVG: 463 MAX: 547 Numpy Benchmark Score > Higher Is Better Stock ........................ 885.50 |======================================== AI/ML Tuning Recommendations . 887.75 |======================================== Numpy Benchmark CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3.0 AVG: 78.8 MAX: 85.7 AI/ML Tuning Recommendations . MIN: 47.9 AVG: 80.7 MAX: 85.3 Numpy Benchmark System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 171 MAX: 178 AI/ML Tuning Recommendations . MIN: 99 AVG: 177 MAX: 188 oneDNN 3.6 Harness: Convolution Batch Shapes Auto - Engine: CPU ms < Lower Is Better Stock ........................ 0.341475 |====================================== AI/ML Tuning Recommendations . 0.321833 |==================================== oneDNN 3.6 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 147 MAX: 242 AI/ML Tuning Recommendations . MIN: 79 AVG: 177 MAX: 274 oneDNN 3.6 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 289 MAX: 452 AI/ML Tuning Recommendations . MIN: 99 AVG: 245 MAX: 409 oneDNN 3.6 Harness: Deconvolution Batch shapes_1d - Engine: CPU ms < Lower Is Better Stock ........................ 6.70897 |======================================= AI/ML Tuning Recommendations . 6.65482 |======================================= oneDNN 3.6 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 2 AVG: 174 MAX: 251 AI/ML Tuning Recommendations . MIN: 101 AVG: 202 MAX: 263 oneDNN 3.6 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 97 AVG: 310 MAX: 410 AI/ML Tuning Recommendations . MIN: 98 AVG: 332 MAX: 436 oneDNN 3.6 Harness: Deconvolution Batch shapes_3d - Engine: CPU ms < Lower Is Better Stock ........................ 0.718484 |====================================== AI/ML Tuning Recommendations . 0.677050 |==================================== oneDNN 3.6 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 107 MAX: 148 AI/ML Tuning Recommendations . MIN: 103 AVG: 116 MAX: 136 oneDNN 3.6 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 97 AVG: 220 MAX: 359 AI/ML Tuning Recommendations . MIN: 97 AVG: 187 MAX: 377 oneDNN 3.6 Harness: IP Shapes 1D - Engine: CPU ms < Lower Is Better Stock ........................ 0.535874 |====================================== AI/ML Tuning Recommendations . 0.507355 |==================================== oneDNN 3.6 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 4 AVG: 146 MAX: 204 AI/ML Tuning Recommendations . MIN: 72 AVG: 170 MAX: 231 oneDNN 3.6 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 97 AVG: 255 MAX: 348 AI/ML Tuning Recommendations . MIN: 98 AVG: 269 MAX: 401 oneDNN 3.6 Harness: IP Shapes 3D - Engine: CPU ms < Lower Is Better Stock ........................ 0.265564 |====================================== AI/ML Tuning Recommendations . 0.254123 |==================================== oneDNN 3.6 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 0 AVG: 155 MAX: 226 AI/ML Tuning Recommendations . MIN: 86 AVG: 188 MAX: 252 oneDNN 3.6 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 275 MAX: 366 AI/ML Tuning Recommendations . MIN: 97 AVG: 303 MAX: 420 oneDNN 3.6 Harness: Recurrent Neural Network Training - Engine: CPU ms < Lower Is Better Stock ........................ 425.72 |======================================== AI/ML Tuning Recommendations . 406.45 |====================================== oneDNN 3.6 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 204 MAX: 265 AI/ML Tuning Recommendations . MIN: 104 AVG: 227 MAX: 275 oneDNN 3.6 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 97 AVG: 337 MAX: 449 AI/ML Tuning Recommendations . MIN: 98 AVG: 365 MAX: 471 oneDNN 3.6 Harness: Recurrent Neural Network Inference - Engine: CPU ms < Lower Is Better Stock ........................ 276.38 |======================================== AI/ML Tuning Recommendations . 262.52 |====================================== oneDNN 3.6 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 0 AVG: 197 MAX: 290 AI/ML Tuning Recommendations . MIN: 128 AVG: 222 MAX: 267 oneDNN 3.6 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 329 MAX: 467 AI/ML Tuning Recommendations . MIN: 99 AVG: 361 MAX: 498 ONNX Runtime 1.19 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better Stock ........................ 276.08 |======================================= AI/ML Tuning Recommendations . 285.10 |======================================== ONNX Runtime 1.19 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better Stock ........................ 3.62341 |======================================= AI/ML Tuning Recommendations . 3.50697 |====================================== ONNX Runtime 1.19 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 204 MAX: 221 AI/ML Tuning Recommendations . MIN: 82 AVG: 237 MAX: 257 ONNX Runtime 1.19 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 328 MAX: 350 AI/ML Tuning Recommendations . MIN: 99 AVG: 375 MAX: 396 ONNX Runtime 1.19 Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better Stock ........................ 6.09788 |===================================== AI/ML Tuning Recommendations . 6.35626 |======================================= ONNX Runtime 1.19 Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better Stock ........................ 165.09 |======================================== AI/ML Tuning Recommendations . 158.53 |====================================== ONNX Runtime 1.19 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 249 MAX: 280 AI/ML Tuning Recommendations . MIN: 77 AVG: 271 MAX: 301 ONNX Runtime 1.19 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 430 MAX: 477 AI/ML Tuning Recommendations . MIN: 99 AVG: 460 MAX: 510 OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU FPS > Higher Is Better Stock ........................ 140497.34 |==================================== AI/ML Tuning Recommendations . 146329.80 |===================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU ms < Lower Is Better Stock ........................ 0.45 |========================================== AI/ML Tuning Recommendations . 0.43 |======================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 262 MAX: 292 AI/ML Tuning Recommendations . MIN: 4 AVG: 276 MAX: 306 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 436 MAX: 471 AI/ML Tuning Recommendations . MIN: 100 AVG: 457 MAX: 491 OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU FPS > Higher Is Better Stock ........................ 716.20 |======================================= AI/ML Tuning Recommendations . 730.73 |======================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU ms < Lower Is Better Stock ........................ 66.89 |========================================= AI/ML Tuning Recommendations . 65.56 |======================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 63 AVG: 325 MAX: 356 AI/ML Tuning Recommendations . MIN: 79 AVG: 340 MAX: 369 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 100 AVG: 594 MAX: 647 AI/ML Tuning Recommendations . MIN: 100 AVG: 615 MAX: 672 OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better Stock ........................ 14035.76 |=================================== AI/ML Tuning Recommendations . 15437.79 |====================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better Stock ........................ 6.72 |========================================== AI/ML Tuning Recommendations . 6.09 |====================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 83 AVG: 271 MAX: 295 AI/ML Tuning Recommendations . MIN: 95 AVG: 305 MAX: 329 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 102 AVG: 440 MAX: 472 AI/ML Tuning Recommendations . MIN: 105 AVG: 502 MAX: 525 OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU FPS > Higher Is Better Stock ........................ 8270.99 |===================================== AI/ML Tuning Recommendations . 8691.00 |======================================= OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU ms < Lower Is Better Stock ........................ 5.77 |========================================== AI/ML Tuning Recommendations . 5.49 |======================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 49 AVG: 290 MAX: 319 AI/ML Tuning Recommendations . MIN: 132 AVG: 314 MAX: 337 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 100 AVG: 485 MAX: 522 AI/ML Tuning Recommendations . MIN: 99 AVG: 521 MAX: 554 OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better Stock ........................ 6720.98 |===================================== AI/ML Tuning Recommendations . 7144.84 |======================================= OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better Stock ........................ 7.09 |========================================== AI/ML Tuning Recommendations . 6.66 |======================================= OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 33 AVG: 255 MAX: 282 AI/ML Tuning Recommendations . MIN: 79 AVG: 276 MAX: 300 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 442 MAX: 469 AI/ML Tuning Recommendations . MIN: 99 AVG: 462 MAX: 497 OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better Stock ........................ 852.83 |======================================= AI/ML Tuning Recommendations . 881.22 |======================================== OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better Stock ........................ 56.24 |========================================= AI/ML Tuning Recommendations . 54.43 |======================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 17 AVG: 313 MAX: 351 AI/ML Tuning Recommendations . MIN: 79 AVG: 335 MAX: 365 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 100 AVG: 556 MAX: 612 AI/ML Tuning Recommendations . MIN: 99 AVG: 584 MAX: 636 OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU FPS > Higher Is Better Stock ........................ 23587.02 |==================================== AI/ML Tuning Recommendations . 25050.47 |====================================== OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU ms < Lower Is Better Stock ........................ 3.94 |========================================== AI/ML Tuning Recommendations . 3.71 |======================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 14 AVG: 282 MAX: 312 AI/ML Tuning Recommendations . MIN: 90 AVG: 309 MAX: 331 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 101 AVG: 473 MAX: 504 AI/ML Tuning Recommendations . MIN: 101 AVG: 511 MAX: 537 OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU FPS > Higher Is Better Stock ........................ 3559.87 |====================================== AI/ML Tuning Recommendations . 3649.63 |======================================= OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU ms < Lower Is Better Stock ........................ 26.94 |========================================= AI/ML Tuning Recommendations . 26.28 |======================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 7 AVG: 317 MAX: 352 AI/ML Tuning Recommendations . MIN: 117 AVG: 335 MAX: 362 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 549 MAX: 578 AI/ML Tuning Recommendations . MIN: 100 AVG: 560 MAX: 600 OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU FPS > Higher Is Better Stock ........................ 2517.89 |===================================== AI/ML Tuning Recommendations . 2630.42 |======================================= OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU ms < Lower Is Better Stock ........................ 18.99 |========================================= AI/ML Tuning Recommendations . 18.16 |======================================= OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 5 AVG: 278 MAX: 306 AI/ML Tuning Recommendations . MIN: 82 AVG: 300 MAX: 323 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 486 MAX: 529 AI/ML Tuning Recommendations . MIN: 101 AVG: 512 MAX: 562 OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU FPS > Higher Is Better Stock ........................ 10525.86 |===================================== AI/ML Tuning Recommendations . 10800.12 |====================================== OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU ms < Lower Is Better Stock ........................ 4.53 |========================================== AI/ML Tuning Recommendations . 4.41 |========================================= OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 0 AVG: 302 MAX: 333 AI/ML Tuning Recommendations . MIN: 82 AVG: 321 MAX: 342 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 101 AVG: 506 MAX: 536 AI/ML Tuning Recommendations . MIN: 100 AVG: 523 MAX: 552 OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU FPS > Higher Is Better Stock ........................ 6747.48 |====================================== AI/ML Tuning Recommendations . 6891.64 |======================================= OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU ms < Lower Is Better Stock ........................ 13.68 |========================================= AI/ML Tuning Recommendations . 13.38 |======================================== OpenVINO 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 27 AVG: 283 MAX: 309 AI/ML Tuning Recommendations . MIN: 107 AVG: 299 MAX: 322 OpenVINO 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 523 MAX: 567 AI/ML Tuning Recommendations . MIN: 99 AVG: 545 MAX: 589 OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better Stock ........................ 55.63 |======================================== AI/ML Tuning Recommendations . 56.46 |========================================= OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better Stock ........................ 24.17 |======================================= AI/ML Tuning Recommendations . 25.66 |========================================= OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better Stock ........................ 17.98 |========================================= AI/ML Tuning Recommendations . 17.71 |======================================== OpenVINO GenAI 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 191 MAX: 280 AI/ML Tuning Recommendations . MIN: 69 AVG: 249 MAX: 349 OpenVINO GenAI 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 332 MAX: 454 AI/ML Tuning Recommendations . MIN: 98 AVG: 381 MAX: 543 OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better Stock ........................ 51.01 |========================================= AI/ML Tuning Recommendations . 51.12 |========================================= OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better Stock ........................ 29.07 |======================================= AI/ML Tuning Recommendations . 30.70 |========================================= OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better Stock ........................ 19.60 |========================================= AI/ML Tuning Recommendations . 19.56 |========================================= OpenVINO GenAI 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 6 AVG: 239 MAX: 310 AI/ML Tuning Recommendations . MIN: 53 AVG: 294 MAX: 364 OpenVINO GenAI 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 427 MAX: 510 AI/ML Tuning Recommendations . MIN: 98 AVG: 490 MAX: 586 OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU tokens/s > Higher Is Better Stock ........................ 37.84 |========================================= AI/ML Tuning Recommendations . 38.00 |========================================= OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better Stock ........................ 36.13 |========================================= AI/ML Tuning Recommendations . 35.43 |======================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better Stock ........................ 26.43 |========================================= AI/ML Tuning Recommendations . 26.31 |========================================= OpenVINO GenAI 2024.5 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 14 AVG: 236 MAX: 293 AI/ML Tuning Recommendations . MIN: 46 AVG: 291 MAX: 341 OpenVINO GenAI 2024.5 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 100 AVG: 420 MAX: 487 AI/ML Tuning Recommendations . MIN: 98 AVG: 483 MAX: 552 PyTorch 2.2.1 Device: CPU - Batch Size: 256 - Model: ResNet-50 batches/sec > Higher Is Better Stock ........................ 51.61 |======================================== AI/ML Tuning Recommendations . 53.13 |========================================= PyTorch 2.2.1 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 235 MAX: 291 AI/ML Tuning Recommendations . MIN: 55 AVG: 281 MAX: 336 PyTorch 2.2.1 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 393 MAX: 455 AI/ML Tuning Recommendations . MIN: 98 AVG: 440 MAX: 516 PyTorch 2.2.1 Device: CPU - Batch Size: 256 - Model: ResNet-152 batches/sec > Higher Is Better Stock ........................ 20.78 |======================================= AI/ML Tuning Recommendations . 21.79 |========================================= PyTorch 2.2.1 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 265 MAX: 295 AI/ML Tuning Recommendations . MIN: 53 AVG: 312 MAX: 343 PyTorch 2.2.1 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 97 AVG: 426 MAX: 463 AI/ML Tuning Recommendations . MIN: 96 AVG: 486 MAX: 526 PyTorch 2.2.1 Device: CPU - Batch Size: 512 - Model: ResNet-50 batches/sec > Higher Is Better Stock ........................ 51.48 |======================================== AI/ML Tuning Recommendations . 53.34 |========================================= PyTorch 2.2.1 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 2 AVG: 238 MAX: 291 AI/ML Tuning Recommendations . MIN: 54 AVG: 285 MAX: 336 PyTorch 2.2.1 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 396 MAX: 457 AI/ML Tuning Recommendations . MIN: 99 AVG: 442 MAX: 517 PyTorch 2.2.1 Device: CPU - Batch Size: 512 - Model: ResNet-152 batches/sec > Higher Is Better Stock ........................ 20.60 |======================================= AI/ML Tuning Recommendations . 21.71 |========================================= PyTorch 2.2.1 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 52 AVG: 270 MAX: 295 AI/ML Tuning Recommendations . MIN: 58 AVG: 313 MAX: 344 PyTorch 2.2.1 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 427 MAX: 461 AI/ML Tuning Recommendations . MIN: 99 AVG: 480 MAX: 528 TensorFlow 2.16.1 Device: CPU - Batch Size: 256 - Model: ResNet-50 images/sec > Higher Is Better Stock ........................ 204.33 |======================================= AI/ML Tuning Recommendations . 207.38 |======================================== TensorFlow 2.16.1 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 4 AVG: 241 MAX: 256 AI/ML Tuning Recommendations . MIN: 53 AVG: 253 MAX: 265 TensorFlow 2.16.1 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 443 MAX: 476 AI/ML Tuning Recommendations . MIN: 99 AVG: 461 MAX: 494 TensorFlow 2.16.1 Device: CPU - Batch Size: 512 - Model: ResNet-50 images/sec > Higher Is Better Stock ........................ 231.18 |======================================= AI/ML Tuning Recommendations . 235.35 |======================================== TensorFlow 2.16.1 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 255 MAX: 268 AI/ML Tuning Recommendations . MIN: 54 AVG: 265 MAX: 276 TensorFlow 2.16.1 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 101 AVG: 472 MAX: 519 AI/ML Tuning Recommendations . MIN: 102 AVG: 488 MAX: 535 Whisper.cpp 1.6.2 Model: ggml-small.en - Input: 2016 State of the Union Seconds < Lower Is Better Stock ........................ 221.63 |======================================== AI/ML Tuning Recommendations . 214.63 |======================================= Whisper.cpp 1.6.2 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 1 AVG: 220 MAX: 236 AI/ML Tuning Recommendations . MIN: 58 AVG: 254 MAX: 273 Whisper.cpp 1.6.2 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 358 MAX: 407 AI/ML Tuning Recommendations . MIN: 99 AVG: 391 MAX: 464 Whisper.cpp 1.6.2 Model: ggml-medium.en - Input: 2016 State of the Union Seconds < Lower Is Better Stock ........................ 454.29 |======================================== AI/ML Tuning Recommendations . 449.70 |======================================== Whisper.cpp 1.6.2 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 244 MAX: 262 AI/ML Tuning Recommendations . MIN: 47 AVG: 283 MAX: 299 Whisper.cpp 1.6.2 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 389 MAX: 469 AI/ML Tuning Recommendations . MIN: 99 AVG: 441 MAX: 504 Whisperfile 20Aug24 Model Size: Tiny Seconds < Lower Is Better Stock ........................ 31.89 |========================================= AI/ML Tuning Recommendations . 31.44 |======================================== Whisperfile 20Aug24 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 137 MAX: 193 AI/ML Tuning Recommendations . MIN: 0 AVG: 155 MAX: 217 Whisperfile 20Aug24 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 256 MAX: 339 AI/ML Tuning Recommendations . MIN: 99 AVG: 285 MAX: 376 Whisperfile 20Aug24 Model Size: Small Seconds < Lower Is Better Stock ........................ 90.67 |========================================= AI/ML Tuning Recommendations . 88.04 |======================================== Whisperfile 20Aug24 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 6 AVG: 181 MAX: 208 AI/ML Tuning Recommendations . MIN: 47 AVG: 200 MAX: 231 Whisperfile 20Aug24 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 99 AVG: 328 MAX: 372 AI/ML Tuning Recommendations . MIN: 98 AVG: 351 MAX: 405 Whisperfile 20Aug24 Model Size: Medium Seconds < Lower Is Better Stock ........................ 200.48 |======================================== AI/ML Tuning Recommendations . 197.24 |======================================= Whisperfile 20Aug24 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 3 AVG: 189 MAX: 207 AI/ML Tuning Recommendations . MIN: 52 AVG: 204 MAX: 222 Whisperfile 20Aug24 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 98 AVG: 341 MAX: 388 AI/ML Tuning Recommendations . MIN: 98 AVG: 362 MAX: 410 XNNPACK b7b048 Model: FP32MobileNetV1 us < Lower Is Better Stock ........................ 4539 |========================================== AI/ML Tuning Recommendations . 4471 |========================================= XNNPACK b7b048 Model: FP32MobileNetV2 us < Lower Is Better Stock ........................ 9203 |========================================== AI/ML Tuning Recommendations . 9062 |========================================= XNNPACK b7b048 Model: FP32MobileNetV3Small us < Lower Is Better Stock ........................ 10488 |========================================= AI/ML Tuning Recommendations . 10387 |========================================= XNNPACK b7b048 Model: FP16MobileNetV1 us < Lower Is Better Stock ........................ 4634 |========================================== AI/ML Tuning Recommendations . 4513 |========================================= XNNPACK b7b048 Model: FP16MobileNetV2 us < Lower Is Better Stock ........................ 9092 |========================================== AI/ML Tuning Recommendations . 9028 |========================================== XNNPACK b7b048 Model: FP16MobileNetV3Small us < Lower Is Better Stock ........................ 10966 |========================================= AI/ML Tuning Recommendations . 10323 |======================================= XNNPACK b7b048 Model: QS8MobileNetV2 us < Lower Is Better Stock ........................ 10042 |========================================= AI/ML Tuning Recommendations . 9813 |======================================== XNNPACK b7b048 CPU Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 2 AVG: 241 MAX: 264 AI/ML Tuning Recommendations . MIN: 0 AVG: 280 MAX: 305 XNNPACK b7b048 System Power Consumption Monitor Watts < Lower Is Better Stock ........................ MIN: 100 AVG: 378 MAX: 404 AI/ML Tuning Recommendations . MIN: 100 AVG: 424 MAX: 459