Background

On november 4, 2019, we were benchmarking a number of virtual machines on a new 12-node DELL 740XD-24 appliance under Nutanix AHV (and RDMA enabled). I was getting weird results and I tweeted the following:

At the moment we were configuring both the Nutanix appliance and the Linux kernel parameters most suitable for a postgresql 9.5 specific workload. That was not a good idea and produced confusing test results since the actual I/O subsystem benchmark needed a few appliance-specific tuning first.

The second bad idea was to make insufficiently long tests. This is why we observed surprisingly that using hugepages adversely impacted performance. More sustained tests confirmed the benefits of using hugepages for postgresql on linux. This is not true.

Disclaimer

The following test results are not intended to give definitive numbers on the tested platform. These are only illustrative of the methodology used to test the platform for postgresql workloads. The results are an initial result sample that improved once the platform configuration was stabilized.

I/O and filesystem benchmarks

We used 2 tools here: fio and We tested an ext4 filesystem for the data directory in 2 scenarios :

  • plain block devices (vdisks)
  • volume groups (lvm)

FIO tests :

The following command was used to generate a random I/O workload with 75% reads and 25% writes: fio –randrepeat=1 –ioengine=libaio –direct=1 –gtod_reduce=1 –name=test –filename=test –bs=4k –iodepth=64 –size=4G –readwrite=randrw –rwmixread=75

  • RDMA without LVM: Random Read/Write performance: read: IOPS=42.8k, BW=167MiB/s (175MB/s)(3070MiB/18376msec) write: IOPS=14.3k, BW=55.8MiB/s (58.5MB/s)(1026MiB/18376msec)
  • No RDMA without LVM: Random Read/Write performance: read: IOPS=37.0k, BW=145MiB/s (152MB/s)(3070MiB/21237msec)  write: IOPS=12.4k, BW=48.3MiB/s (50.7MB/s)(1026MiB/21237msec)
  • RDMA with LVM: Random Read/Write performance: read: IOPS=38.4k, BW=150MiB/s (157MB/s)(3070MiB/20441msec) write: IOPS=12.8k, BW=50.2MiB/s (52.6MB/s)(1026MiB/20441msec

Pgio tests:

  • With lvm:
Date: Wed Oct 30 17:15:50 CET 2019
Database connect string: "pgio". 
Shared buffers: 1GB. 
Testing 16 schemas with 8 thread(s) accessing 8G (1048576 blocks) of each schema.
Running iostat, vmstat and mpstat on current host--in background.
Launching sessions. 16 schema(s) will be accessed by 8 thread(s) each.
Sessions launched. NOTE: Launching the sessions took 1 seconds.
pg_stat_database stats:
    datname| blks_hit|blks_read|tup_returned|tup_fetched|tup_updated
BEFORE:pgio| 49017763| 9063793 |   16791017 |      9865 |         66
AFTER: pgio| 49130549|20283450 |   18055536 |   1274333 |         66
DBNAME:pgio. 16 schemas, 8 threads(each). Run time: 123 seconds. RIOPS >9915< CACHE_HITS/s >935<
  • Without lvm:
Date: Thu Oct 31 09:49:43 CET 2019
Database connect string: "pgio". 
Shared buffers: 1GB. 
Testing 16 schemas with 8 thread(s) accessing 8G (1048576 blocks) of each schema.
Running iostat, vmstat and mpstat on current host--in background.
Launching sessions. 16 schema(s) will be accessed by 8 thread(s) each.
pg_stat_database stats:
    datname| blks_hit|blks_read|tup_returned|tup_fetched|tup_updated
BEFORE:pgio|49383615 |22763686 |   20454623 |   3648699 |         72
AFTER: pgio|49516016 |24965866 |   22705702 |   5899519 |         72
DBNAME:  pgio. 16 schemas, 8 threads(each). Run time: 121 seconds. RIOPS >18199< CACHE_HITS/s >1122<

These tests were obviously focused on physical I/O so that we tried to minimize buffered I/O. We used pgio, a simple tool to generate I/O with database workload as we have been using SLOB for Oracle I/O benchmarking for years.

Production workload replay

We ran a sample production workload on this environment and compared the performance between the current production environment and this (test) environment. A sample production workload was captured for a few hours and replayed using the venerable pgreplay.

We measured the performance in production and in the test environment using pgwatch2, a handy tool to fetch performance metrics and plot them into grafana dashboards.

Workload replay comparison

Production

DB Overview
Query performance overview

Test environment:

DB Overview
Query performance overview

The above workload is illustrative. While pgreplay is limited and there is a percentage of errors to take into account, it gives a good idea of what to expect on a new environment and detect potential performance issues. The SQL execution dashboard included in pgwatch2 helps in identifying query regressions and the DB Overview dashboard includes overall instance metrics.

Sustained database benchmark with different linux kernel parameters

We tested different linux kernel tuning recommendations that we will summarize as follows :

Nutanix best practices:
(https://www.nutanix.com/de/go/postgresql-best-practice ) 

elevator=noop scsi_mod.use_blk_mq=1 dm_mod.use_blk_mq=y transparent_hugepage=nevervm.swappiness=0 # NOTE: GRUB configuration
vm.overcommit_memory=1
vm.dirty_background_ratio=5
vm.dirty_ratio=15

percona :
(https://www.percona.com/blog/2018/08/29/tune-linux-kernel-parameters-for-postgresql-optimization/ )

vm.swappiness=1
vm.overcommit_memory=2
vm.dirty_background_ratio=5
vm.dirty_ratio=15

dbi :
(https://blog.dbi-services.com/stay-tuned-with-kernel-parameters/

(note: although this post was targeted at Oracle databases, we just took the following vm tunables)

vm.swappiness=1
vm.dirty_background_ratio=3
vm.dirty_ratio=80

Other considerations :
The Percona suggested parameter vm.overcommit_memory=2 caused the machine to crash. We configured hugepages in all our tests.

We also tested a combination of vm.dirty_background_bytes and vm.dirty_bytes. We used tuned to create different tuning profiles for each test set and ran a series of pgbench-tools tests with the following characteristics:
scale: 10, 100, 1000
clients: 8, 16, 32, 64, 128

For each test set, pgbench-tools will test a TPC-B-like test iteratively over each scale factor (database size) and concurrency (number of concurrent clients). It will rebuild the database at each test and will execute each iteration test a configured number of times. Here are the overall TPS and latency results for different scaling factors and concurrency levels.

Postgresql on Nutanix Test Results

Averages across all test sets:

Test sets comparison:

Nutanix – SWP=1

Averages for test set 2 by scale:

set scale tps avg_latency 90%< max_latency
2 10 1169 40.056 89.042 835.771
2 100 3614 11.475 17.309 153.588
2 1000 2901 14.086 18.374 166.652

Averages for test set 2 by clients:
set clients tps avg_latency 90%< max_latency
2 8 1026 7.798 10.202 99.6
2 16 1596 10.708 15.872 96.437
2 32 2511 15.684 27.851 182.117
2 64 3558 26.024 51.182 454.39
2 128 4114 49.148 102.768 1094.141

Averages for test set 2 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
2 10 8   1040 7.687 13.255 182.4
2 10 16   1074 14.883 27.286 145.158
2 10 32   1194 26.773 55.259 340.146
2 10 64   1270 50.282 112.359 927.543
2 10 128   1266 100.655 237.052 2583.611
2 100 8   1048 7.63 8.096 58.573
2 100 16   1920 8.322 10.492 74.811
2 100 32   3404 9.383 15.101 113.958
2 100 64   5412 11.788 18.792 198.208
2 100 128   6284 20.255 34.064 322.389
2 1000 8   990 8.078 9.256 57.826
2 1000 16   1792 8.92 9.839 69.343
2 1000 32   2936 10.897 13.192 92.248
2 1000 64   3994 16.001 22.394 237.419
2 1000 128   4791 26.533 37.188 376.424

Detail for test set 2:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
2 91 10 8   1027 401.401 0 0 0 0 4663 0 0 5775360 205220536 00:00:01.134547
2 96 10 8   1029 73.793 0 0 0 0 1601 0 0 10272768 197841120 00:00:00.953355
2 101 10 8   1064 72.006 0 0 0 0 1619 0 0 9068544 201456792 00:00:01.958402
2 92 10 16   1067 162.547 0 0 0 0 1766 0 0 10461184 202734344 00:00:00.977637
2 97 10 16   1086 131.76 0 0 0 0 1658 0 0 7397376 192408192 00:00:00.903263
2 102 10 16   1069 141.166 0 0 0 0 1601 0 0 8355840 207545400 00:00:01.672924
2 93 10 32   1186 327.439 0 0 0 0 1708 0 0 8933376 200768320 00:00:00.90643
2 98 10 32   1189 307.538 0 0 0 0 1679 0 0 4222976 208003072 00:00:00.980448
2 103 10 32   1206 385.46 0 0 0 0 1713 0 0 7610368 214440896 00:00:01.012527
2 94 10 64   1289 816.55 0 0 0 0 1905 0 0 6070272 216960352 00:00:00.973361
2 99 10 64   1271 951.16 0 0 0 0 1763 0 0 10395648 214708344 00:00:01.050308
2 104 10 64   1249 1014.918 0 0 0 0 1857 0 0 5042176 216529440 00:00:01.173612
2 95 10 128   1268 2467.384 0 0 0 0 1815 0 0 4431872 201786424 00:00:00.990879
2 100 10 128   1264 2306.169 0 0 0 0 1950 0 0 10129408 219729392 00:00:01.061743
2 105 10 128   1267 2977.279 0 0 0 0 1807 0 0 4984832 219243224 00:00:01.060785
2 106 100 8   1030 66.883 0 0 0 0 27113 0 0 10452992 639323424 00:00:04.969729
2 111 100 8   1069 53.422 0 0 0 0 3520 0 0 7909376 572866880 00:00:03.597518
2 116 100 8   1044 55.414 0 0 0 0 2071 0 0 7712768 536375800 00:00:03.617157
2 107 100 16   1917 82.253 0 0 0 0 5271 0 0 7274496 1158110168 00:00:06.25713
2 112 100 16   1906 73.686 0 0 0 0 2570 0 0 9187328 865286008 00:00:04.465264
2 117 100 16   1938 68.495 0 0 0 0 2595 0 0 8949760 869032456 00:00:04.484084
2 108 100 32   3339 106.808 0 0 0 0 3142 0 0 9388032 1458084032 00:00:05.808569
2 113 100 32   3443 105.286 0 0 0 0 3101 0 0 8671232 1223079624 00:00:04.893603
2 118 100 32   3428 129.78 0 0 0 0 2946 0 0 9916416 1228242384 00:00:04.947918
2 109 100 64   5269 236.869 0 0 0 0 3810 0 0 5320704 1632378768 00:00:05.284386
2 114 100 64   5491 173.14 0 0 0 0 3788 0 0 9318400 1543258800 00:00:05.145104
2 119 100 64   5476 184.616 0 0 0 0 3791 0 0 9818112 1534591288 00:00:04.837266
2 110 100 128   6187 357.632 0 0 0 0 3837 0 0 12095488 1692169376 00:00:04.837857
2 115 100 128   6370 305.634 0 0 0 0 4259 0 0 10223616 1665829856 00:00:05.273714
2 120 100 128   6296 303.901 0 0 0 0 4288 0 0 9613312 1637966584 00:00:05.285589
2 121 1000 8   1006 56.041 0 0 0 0 117552 0 0 6565888 936354560 00:00:33.960196
2 126 1000 8   996 57.657 0 0 9 0 37999 0 0 4653056 1289572856 00:00:42.607478
2 131 1000 8   967 59.779 0 0 0 0 34535 0 0 5996544 1258169208 00:00:39.092409
2 122 1000 16   1856 63.002 0 0 0 0 180628 0 0 8523776 2054479872 00:00:38.537672
2 127 1000 16   1771 69.642 0 0 20 0 64378 0 0 7458816 2187086136 00:00:42.879895
2 132 1000 16   1750 75.386 0 0 51 0 62073 0 0 7327744 2170146616 00:00:38.346188
2 123 1000 32   3127 86.331 0 0 0 0 236014 0 0 10878976 3323978800 00:00:45.604883
2 128 1000 32   2866 106.706 0 0 76 0 114604 0 0 10301440 3421137048 00:00:53.884147
2 133 1000 32   2814 83.707 0 0 74 0 111723 0 0 10235904 3369281072 00:01:00.967181
2 124 1000 64   4260 230.154 0 0 0 0 238447 0 0 9703424 4487708672 00:01:01.316089
2 129 1000 64   3930 177.928 0 0 23 0 162808 0 0 9703424 4503944440 00:01:08.072835
2 134 1000 64   3791 304.176 0 0 54 0 157005 0 0 7868416 4367736736 00:01:06.013256
2 125 1000 128   4846 350.794 0 0 7634 0 207268 68 0 22306816 5211840560 00:01:08.650471
2 130 1000 128   4831 402.499 0 0 102 0 199080 0 0 12734464 5379879560 00:01:09.863973
2 135 1000 128   4697 375.979 0 0 46 0 195731 0 0 9203712 5251457024 00:01:10.590457

Nutanix – SWP1/VM

Averages for test set 3 by scale:

set scale tps avg_latency 90%< max_latency
3 10 1081 43.218 94.455 717.472
3 100 3134 13.079 20.84 172.158
3 1000 2888 14.163 18.82 159.626

Averages for test set 3 by clients:
set clients tps avg_latency 90%< max_latency
3 8 978 8.179 11.204 69.483
3 16 1493 11.485 18.448 105.265
3 32 2417 16.584 30.27 194.831
3 64 3152 28.521 56.255 419.958
3 128 3797 52.663 107.347 959.221

Averages for test set 3 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
3 10 8   964 8.295 13.915 81.887
3 10 16   992 16.121 30.366 168.247
3 10 32   1104 28.958 61.204 390.85
3 10 64   1169 54.68 122.636 815.33
3 10 128   1179 108.035 244.153 2131.044
3 100 8   1004 7.961 9.857 70.626
3 100 16   1755 9.106 13.885 70.729
3 100 32   3152 10.134 15.948 102.536
3 100 64   4367 14.598 23.583 257.071
3 100 128   5391 23.595 40.926 359.827
3 1000 8   966 8.28 9.841 55.937
3 1000 16   1734 9.229 11.094 76.818
3 1000 32   2996 10.661 13.658 91.108
3 1000 64   3921 16.284 22.544 187.474
3 1000 128   4822 26.359 36.961 386.792

Detail for test set 3:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
3 136 10 8   940 84.468 0 0 0 0 3393 0 0 3117056 179782352 00:00:01.102532
3 141 10 8   959 85.185 0 0 0 0 374 0 0 5877760 167339104 00:00:00.875153
3 146 10 8   993 76.008 0 0 0 0 389 0 0 5341184 173904128 00:00:00.916166
3 137 10 16   1004 178.094 0 0 0 0 419 0 0 6385664 177822328 00:00:00.861491
3 142 10 16   980 172.429 0 0 0 0 384 0 0 2883584 174031096 00:00:00.783348
3 147 10 16   991 154.218 0 0 0 0 391 0 0 483328 181668096 00:00:00.894519
3 138 10 32   1092 332.347 0 0 0 0 472 0 0 3678208 167320704 00:00:00.891234
3 143 10 32   1142 378.936 0 0 0 0 455 0 0 6090752 190501728 00:00:00.965376
3 148 10 32   1078 461.266 0 0 0 0 465 0 0 421888 187224568 00:00:00.841851
3 139 10 64   1208 912.423 0 0 0 0 549 0 0 7000064 193891928 00:00:00.921156
3 144 10 64   1103 767.271 0 0 0 0 456 0 0 4276224 201175152 00:00:00.950972
3 149 10 64   1196 766.296 0 0 0 0 555 0 0 466944 192447664 00:00:00.794497
3 140 10 128   1187 1537.608 0 0 0 0 669 0 0 5226496 196873664 00:00:00.986962
3 145 10 128   1180 1967.646 0 0 0 0 540 0 0 6742016 209018880 00:00:01.052592
3 150 10 128   1168 2887.879 0 0 0 0 657 0 0 552960 195824048 00:00:00.847009
3 151 100 8   1001 68.683 0 0 0 0 25692 0 0 3727360 606611552 00:00:03.918596
3 156 100 8   999 68.422 0 0 0 0 566 0 0 8491008 547328968 00:00:03.647128
3 161 100 8   1012 74.773 0 0 0 0 587 0 0 6656000 548811088 00:00:03.975654
3 152 100 16   1725 74.93 0 0 0 0 3987 0 0 5050368 1091913072 00:00:04.869103
3 157 100 16   1782 68.301 0 0 0 0 1088 0 0 6692864 848881328 00:00:04.348871
3 162 100 16   1758 68.956 0 0 0 0 1103 0 0 8269824 822101864 00:00:04.575881
3 153 100 32   3133 108.495 0 0 0 0 1853 0 0 6983680 1398756736 00:00:05.011016
3 158 100 32   3096 88.791 0 0 0 0 2002 0 0 7307264 1187916512 00:00:04.839242
3 163 100 32   3227 110.321 0 0 0 0 1664 0 0 6107136 1232103528 00:00:05.203882
3 154 100 64   4316 271.544 0 0 0 0 2472 0 0 9695232 1555227344 00:00:04.860892
3 159 100 64   4501 232.171 0 0 0 0 2217 0 0 9781248 1468994456 00:00:04.877038
3 164 100 64   4284 267.497 0 0 0 0 2314 0 0 5414912 1426404968 00:00:04.856215
3 155 100 128   5365 353.464 0 0 0 0 2627 0 0 7696384 1645373552 00:00:04.928387
3 160 100 128   5497 383.41 0 0 0 0 2214 0 0 7970816 1627731464 00:00:04.834517
3 165 100 128   5310 342.607 0 0 0 0 2732 0 0 7340032 1581002928 00:00:05.342896
3 166 1000 8   941 60.425 0 0 0 0 109356 0 0 7823360 868452360 00:00:33.51471
3 171 1000 8   972 55.229 0 0 5 0 39123 0 0 4046848 1223985616 00:00:41.520004
3 176 1000 8   984 52.156 0 0 12 0 33969 0 0 4288512 1217515040 00:00:46.811764
3 167 1000 16   1663 68.47 0 0 0 0 163305 0 0 7942144 1878757200 00:00:39.659402
3 172 1000 16   1792 60.846 0 0 47 0 67331 0 0 6905856 2197558872 00:00:34.759853
3 177 1000 16   1745 101.138 0 0 50 0 59810 0 0 9256960 2136085504 00:00:38.477648
3 168 1000 32   2938 87.257 0 0 0 0 227562 0 0 10293248 3088614928 00:00:44.836438
3 173 1000 32   3023 63.392 0 0 390 0 140226 0 0 8491008 3535926976 00:00:58.519543
3 178 1000 32   3028 122.675 0 0 63 0 118963 0 0 5890048 3530411336 00:01:03.718875
3 169 1000 64   3742 188.448 0 0 0 0 219553 0 0 9289728 4089559408 00:00:57.145272
3 174 1000 64   4005 206.046 0 0 38 0 167901 0 0 8646656 4559075064 00:01:08.796204
3 179 1000 64   4015 167.929 0 0 291 0 163407 0 0 8781824 4539702048 00:01:06.753549
3 170 1000 128   4688 323.067 0 0 2478 0 209802 24 0 84316160 4924307720 00:01:11.303982
3 175 1000 128   4904 444.514 0 0 208 0 193597 0 0 8515584 5355819832 00:01:08.654924
3 180 1000 128   4873 392.795 0 0 965 0 199757 0 0 7434240 5389553952 00:01:12.896901

Nutanix – best practices

Averages for test set 4 by scale:

set scale tps avg_latency 90%< max_latency
4 10 1182 39.365 88.138 787.896
4 100 3518 11.721 17.968 170.601
4 1000 3067 13.313 16.832 165.793

Averages for test set 4 by clients:
set clients tps avg_latency 90%< max_latency
4 8 1038 7.702 9.926 63.048
4 16 1605 10.631 15.813 111.423
4 32 2573 15.364 27.823 211.484
4 64 3513 25.879 51.09 435.873
4 128 4215 47.755 100.246 1051.989

Averages for test set 4 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
4 10 8   1035 7.72 13.183 74.105
4 10 16   1084 14.749 26.905 178.843
4 10 32   1208 26.44 56.71 434.356
4 10 64   1281 49.805 111.948 916.073
4 10 128   1299 98.11 231.944 2336.105
4 100 8   1057 7.567 7.961 60.626
4 100 16   1906 8.385 11.107 77.675
4 100 32   3388 9.427 15.073 101.406
4 100 64   5090 12.529 20.551 200.554
4 100 128   6150 20.694 35.15 412.746
4 1000 8   1022 7.82 8.634 54.412
4 1000 16   1825 8.759 9.427 77.751
4 1000 32   3123 10.224 11.685 98.691
4 1000 64   4168 15.302 20.772 190.993
4 1000 128   5197 24.461 33.643 407.118

Detail for test set 4:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
4 181 10 8   1028 79.232 0 0 0 0 3463 0 0 3878912 183429048 00:00:00.924079
4 186 10 8   1046 64.771 0 0 0 0 601 0 0 5738496 166734888 00:00:00.779286
4 191 10 8   1031 78.313 0 0 0 0 884 0 0 7139328 179055272 00:00:00.94822
4 182 10 16   1087 193.201 0 0 0 0 511 0 0 9203712 177052328 00:00:00.768397
4 187 10 16   1082 172.794 0 0 0 0 508 0 0 3743744 151816224 00:00:00.78761
4 192 10 16   1082 170.535 0 0 0 0 880 0 0 6369280 186903872 00:00:00.968328
4 183 10 32   1216 384.817 0 0 0 0 568 0 0 9551872 185865272 00:00:00.87489
4 188 10 32   1197 459.283 0 0 0 0 915 0 0 8069120 183053104 00:00:00.879565
4 193 10 32   1213 458.967 0 0 0 0 941 0 0 4673536 195150904 00:00:01.134
4 184 10 64   1293 902.093 0 0 0 0 633 0 0 8253440 193584952 00:00:00.799526
4 189 10 64   1264 719.84 0 0 0 0 1034 0 0 7708672 196103032 00:00:00.970332
4 194 10 64   1287 1126.286 0 0 0 0 1092 0 0 3424256 198913016 00:00:01.042712
4 185 10 128   1290 1901.848 0 0 0 0 782 0 0 7380992 198666784 00:00:00.96922
4 190 10 128   1310 3212.441 0 0 0 0 1147 0 0 8065024 201840400 00:00:01.146428
4 195 10 128   1296 1894.025 0 0 0 0 859 0 0 3821568 200080976 00:00:00.94773
4 196 100 8   1029 61.612 0 0 0 0 26221 0 0 10113024 636311224 00:00:05.510534
4 201 100 8   1092 58.85 0 0 0 0 1031 0 0 4489216 558362768 00:00:03.7953
4 206 100 8   1049 61.417 0 0 0 0 500 0 0 6344704 538997256 00:00:03.668003
4 197 100 16   1887 84.694 0 0 0 0 4522 0 0 5910528 1116028224 00:00:05.795861
4 202 100 16   1909 79.258 0 0 0 0 1457 0 0 7667712 848623576 00:00:04.36835
4 207 100 16   1923 69.074 0 0 0 0 805 0 0 8638464 848854200 00:00:04.414788
4 198 100 32   3303 102.482 0 0 0 0 2412 0 0 5836800 1438174928 00:00:05.759381
4 203 100 32   3450 90.549 0 0 0 0 2113 0 0 5398528 1219650768 00:00:04.791329
4 208 100 32   3410 111.187 0 0 0 0 1332 0 0 4599808 1193811080 00:00:04.794698
4 199 100 64   4993 190.319 0 0 0 0 2896 0 0 6742016 1616689848 00:00:05.190577
4 204 100 64   5148 223.225 0 0 0 0 2692 0 0 9080832 1467684160 00:00:04.71495
4 209 100 64   5130 188.117 0 0 0 0 2008 0 0 6402048 1478092720 00:00:04.754659
4 200 100 128   6204 431.822 0 0 0 0 3392 0 0 7557120 1662012568 00:00:05.49015
4 205 100 128   6322 413.503 0 0 0 0 2771 0 0 8265728 1637841488 00:00:05.09542
4 210 100 128   5923 392.912 0 0 0 0 2354 0 0 7106560 1588262096 00:00:04.991039
4 211 1000 8   1019 55.765 0 0 0 0 117331 0 0 3289088 932581456 00:00:37.193358
4 216 1000 8   1017 50.686 0 0 37 0 41774 0 0 6639616 1247682680 00:00:42.486634
4 221 1000 8   1031 56.784 0 0 23 0 35064 0 0 10473472 1285242432 00:00:37.832466
4 212 1000 16   1813 108.171 0 0 0 0 174761 0 0 8245248 2026737584 00:00:39.290495
4 217 1000 16   1835 60.632 0 0 109 0 79869 0 0 7864320 2219557376 00:00:38.793936
4 222 1000 16   1826 64.45 0 0 56 0 63632 0 0 7991296 2211681848 00:00:34.954602
4 213 1000 32   3166 77.725 0 0 0 0 238230 0 0 8138752 3300297576 00:00:46.726202
4 218 1000 32   3095 145.124 0 0 8 0 144310 0 0 9142272 3589322040 00:01:01.818296
4 223 1000 32   3110 73.224 0 0 55 0 121096 0 0 8835072 3591895984 00:00:55.778571
4 214 1000 64   4157 162.226 0 0 0 0 232044 0 0 5668864 4417057752 00:01:00.040304
4 219 1000 64   4202 217.421 0 0 175 0 188317 0 0 5779456 4704702936 00:01:03.134753
4 224 1000 64   4146 193.333 0 0 227 0 171946 0 0 6135808 4652858920 00:01:05.18273
4 215 1000 128   5132 422.795 0 0 4345 0 217369 43 0 125702144 5308612592 00:01:21.760186
4 220 1000 128   5142 420.14 0 0 713 0 211262 0 0 9977856 5620639592 00:01:04.329275
4 225 1000 128   5316 378.419 0 0 2758 0 219666 15 0 13701120 5693253856 00:01:07.304406

Nutanix – dbi

Averages for test set 5 by scale:

set scale tps avg_latency 90%< max_latency
5 10 1180 39.556 88.218 687.456
5 100 3595 11.509 17.585 159.463
5 1000 3026 13.495 17.308 237.722

Averages for test set 5 by clients:
set clients tps avg_latency 90%< max_latency
5 8 1039 7.701 9.962 62.508
5 16 1622 10.563 15.751 98.106
5 32 2582 15.298 27.925 181.434
5 64 3572 25.792 50.641 403.439
5 128 4187 48.244 100.908 1062.247

Averages for test set 5 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
5 10 8   1035 7.728 13.236 74.035
5 10 16   1081 14.787 27.151 152.699
5 10 32   1216 26.279 56.683 349.963
5 10 64   1281 49.834 111.09 780.091
5 10 128   1285 99.151 232.933 2080.492
5 100 8   1071 7.462 7.851 62.966
5 100 16   1913 8.357 10.825 78.097
5 100 32   3431 9.308 14.918 103.797
5 100 64   5342 11.949 19.374 199.756
5 100 128   6216 20.467 34.957 352.698
5 1000 8   1010 7.914 8.8 50.522
5 1000 16   1871 8.545 9.276 63.523
5 1000 32   3099 10.308 12.173 90.542
5 1000 64   4092 15.594 21.459 230.47
5 1000 128   5060 25.114 34.834 753.552

Detail for test set 5:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
5 226 10 8   1009 78.105 0 0 0 0 3426 0 0 6483968 181371712 00:00:00.980108
5 231 10 8   1058 65.245 0 0 0 0 417 0 0 10354688 192060632 00:00:00.870046
5 236 10 8   1037 78.755 0 0 0 0 407 0 0 8126464 202758736 00:00:00.93794
5 227 10 16   1089 153.279 0 0 0 0 461 0 0 3371008 175797848 00:00:00.755001
5 232 10 16   1083 158.614 0 0 0 0 429 0 0 7905280 194268672 00:00:00.783676
5 237 10 16   1071 146.203 0 0 0 0 426 0 0 6938624 209602168 00:00:01.06011
5 228 10 32   1245 329.965 0 0 0 0 540 0 0 9777152 195567768 00:00:00.909809
5 233 10 32   1204 374.582 0 0 0 0 482 0 0 6393856 207959000 00:00:00.935597
5 238 10 32   1200 345.342 0 0 0 0 483 0 0 8318976 218615216 00:00:01.043394
5 229 10 64   1272 771.357 0 0 0 0 564 0 0 8577024 217958800 00:00:00.911229
5 234 10 64   1298 820.609 0 0 0 0 540 0 0 10797056 221213704 00:00:00.99838
5 239 10 64   1273 748.306 0 0 0 0 532 0 0 8417280 222210344 00:00:01.125778
5 230 10 128   1291 1625.371 0 0 0 0 751 0 0 5971968 223627496 00:00:00.946291
5 235 10 128   1294 2497.081 0 0 0 0 725 0 0 10723328 225117184 00:00:01.18691
5 240 10 128   1271 2119.024 0 0 0 0 716 0 0 7032832 225046760 00:00:01.089493
5 241 100 8   1046 75.15 0 0 0 0 25948 0 0 11362304 667928080 00:00:04.492989
5 246 100 8   1082 50.812 0 0 0 0 424 0 0 8478720 565695712 00:00:03.882777
5 251 100 8   1086 62.935 0 0 0 0 428 0 0 8966144 584298216 00:00:03.83447
5 242 100 16   1845 71.702 0 0 0 0 3886 0 0 11927552 1147731608 00:00:05.834672
5 247 100 16   1948 64.941 0 0 0 0 761 0 0 12013568 886041720 00:00:04.374209
5 252 100 16   1946 97.649 0 0 0 0 758 0 0 11546624 891069184 00:00:04.435589
5 243 100 32   3352 76.553 0 0 0 0 1919 0 0 10526720 1443019520 00:00:05.514423
5 248 100 32   3482 126.069 0 0 0 0 1360 0 0 12107776 1253916672 00:00:04.533398
5 253 100 32   3459 108.768 0 0 0 0 1351 0 0 11784192 1253522856 00:00:04.662608
5 244 100 64   5107 197.584 0 0 0 0 2410 0 0 6557696 1646231288 00:00:05.094045
5 249 100 64   5511 193.062 0 0 0 0 2157 0 0 12820480 1556700496 00:00:04.668066
5 254 100 64   5409 208.621 0 0 0 0 2111 0 0 12439552 1533689424 00:00:04.770895
5 245 100 128   6179 316.962 0 0 0 0 2763 0 0 11137024 1698765624 00:00:04.964204
5 250 100 128   6233 344.869 0 0 0 0 2467 0 0 9129984 1650312832 00:00:04.689394
5 255 100 128   6235 396.263 0 0 0 0 2474 0 0 12124160 1673031088 00:00:04.879265
5 256 1000 8   997 48.107 0 0 0 0 114974 0 0 4829184 926258936 00:00:25.887712
5 261 1000 8   1019 51.33 0 0 0 0 36510 0 0 10866688 1294744184 00:00:36.438794
5 266 1000 8   1014 52.13 0 0 40 0 33660 0 0 4014080 1273920592 00:00:34.849658
5 257 1000 16   1886 60.54 0 0 0 0 181730 0 0 12644352 2103069864 00:00:31.907607
5 262 1000 16   1857 69.269 0 0 11546 0 63078 114 0 50868224 2256516784 00:00:53.276331
5 267 1000 16   1869 60.76 0 0 77 0 62898 0 0 9760768 2290432272 00:00:34.87621
5 258 1000 32   3186 114.259 0 0 0 0 237857 0 0 13287424 3374860808 00:00:47.236097
5 263 1000 32   3096 91.081 0 0 618 0 165253 0 0 12660736 3611888056 00:00:57.31669
5 268 1000 32   3016 66.286 0 0 354 0 117914 0 0 10612736 3528717560 00:00:56.412759
5 259 1000 64   4019 191.455 0 0 0 0 223911 0 0 13594624 4375378408 00:00:57.461825
5 264 1000 64   4286 267.505 0 0 44 0 199225 0 0 10297344 4824715336 00:01:04.821371
5 269 1000 64   3970 232.449 0 0 295 0 158725 0 0 10301440 4509059528 00:01:07.059475
5 260 1000 128   5092 1492.372 0 0 0 0 216200 0 0 11792384 5272343712 00:00:56.719959
5 265 1000 128   5055 378.502 0 0 54 0 200819 0 0 9957376 5569926896 00:01:00.587166
5 270 1000 128   5035 389.783 0 0 981 0 214334 0 0 10903552 5552797416 00:01:09.615889

Nutanix baseline/hugepages

Averages for test set 6 by scale:

set scale tps avg_latency 90%< max_latency
6 10 1195 39.134 86.671 794.855
6 100 3749 11.089 16.441 118.702
6 1000 2904 11.864 15.112 85.109

Averages for test set 6 by clients:
set clients tps avg_latency 90%< max_latency
6 8 1052 7.604 10.071 253.804
6 16 1597 10.731 16.137 104.256
6 32 2530 15.778 28.708 171.63
6 64 3682 26.363 53.014 403.243
6 128 4204 53.757 117.036 1000.84

Averages for test set 6 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
6 10 8   1051 7.611 13.143 574.918
6 10 16   1099 14.553 26.666 159.316
6 10 32   1233 25.937 53.749 317.169
6 10 64   1298 49.255 109.662 856.805
6 10 128   1298 98.312 230.135 2066.068
6 100 8   1070 7.474 7.773 64.002
6 100 16   1918 8.339 10.16 77.578
6 100 32   3428 9.329 14.994 86.593
6 100 64   5499 11.622 18.225 148.744
6 100 128   6832 18.681 31.056 216.591
6 1000 8   1026 7.79 8.912 56.834
6 1000 16   1862 8.588 9.309 61.683
6 1000 32   3130 10.212 11.718 80.875
6 1000 64   4531 14.137 20.225 104.647
6 1000 128   5038 25.319 35.683 157.901

Detail for test set 6:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
6 271 10 8   1029 1573.706 0 0 0 0 3468 0 0 2211840 232997384 00:00:00.887628
6 276 10 8   1065 75.62 0 0 0 0 422 0 0 2396160 216784384 00:00:00.74217
6 281 10 8   1058 75.428 0 0 0 0 421 0 0 4431872 227656824 00:00:00.762266
6 272 10 16   1109 180.522 0 0 0 0 464 0 0 3399680 227030976 00:00:00.618077
6 277 10 16   1107 146.903 0 0 0 0 444 0 0 2777088 219168024 00:00:00.855723
6 282 10 16   1081 150.523 0 0 0 0 433 0 0 4132864 233593224 00:00:00.914606
6 273 10 32   1214 311.415 0 0 0 0 529 0 0 3895296 234612368 00:00:00.739927
6 278 10 32   1252 332.074 0 0 0 0 506 0 0 5005312 232455464 00:00:00.806284
6 283 10 32   1233 308.019 0 0 0 0 501 0 0 3919872 243950544 00:00:00.859141
6 274 10 64   1324 858.345 0 0 0 0 590 0 0 4489216 243969752 00:00:00.766067
6 279 10 64   1275 861.577 0 0 0 0 528 0 0 5083136 245423688 00:00:00.993042
6 284 10 64   1295 850.494 0 0 0 0 537 0 0 4698112 246946848 00:00:00.963112
6 275 10 128   1303 2147.887 0 0 0 0 693 0 0 3760128 248263536 00:00:00.84488
6 280 10 128   1291 1980.189 0 0 0 0 755 0 0 3440640 249595064 00:00:00.888391
6 285 10 128   1299 2070.129 0 0 0 0 699 0 0 3936256 218344968 00:00:01.02825
6 286 100 8   1054 73.717 0 0 0 0 26043 0 0 4313088 681225576 00:00:03.728298
6 291 100 8   1077 59.581 0 0 0 0 429 0 0 4591616 559665320 00:00:03.342027
6 296 100 8   1078 58.709 0 0 0 0 428 0 0 3850240 576777744 00:00:02.918296
6 287 100 16   1883 73.673 0 0 0 0 3837 0 0 5156864 1173371320 00:00:04.102839
6 292 100 16   1944 91.733 0 0 0 0 764 0 0 4751360 879237088 00:00:03.730581
6 297 100 16   1926 67.328 0 0 0 0 754 0 0 4452352 873302624 00:00:03.81608
6 288 100 32   3357 88.55 0 0 0 0 1921 0 0 4427776 1467371336 00:00:04.151036
6 293 100 32   3473 74.009 0 0 0 0 1355 0 0 4431872 1232738864 00:00:04.073555
6 298 100 32   3453 97.219 0 0 0 0 1350 0 0 4214784 1214532056 00:00:04.256876
6 289 100 64   5378 162.537 0 0 0 0 2547 0 0 4268032 1677823624 00:00:04.366011
6 294 100 64   5587 132.694 0 0 0 0 2177 0 0 5791744 1531265560 00:00:04.093575
6 299 100 64   5534 151 0 0 0 0 2160 0 0 4730880 1537065640 00:00:04.163585
6 290 100 128   6803 201.724 0 0 0 0 3014 0 0 7634944 1735442432 00:00:04.618786
6 295 100 128   6885 225.479 0 0 0 0 2707 0 0 5300224 1672685072 00:00:04.201567
6 300 100 128   6808 222.57 0 0 0 0 2684 0 0 4894720 1680710368 00:00:04.399026
6 301 1000 8   1029 59.765 0 0 0 0 118597 0 0 3731456 977717168 00:00:27.791992
6 306 1000 8   1023 53.903 0 0 0 0 35699 0 0 2957312 1311944656 00:00:27.914616
6 302 1000 16   1879 52.93 0 0 0 0 179977 0 0 4284416 2131312640 00:00:29.979038
6 307 1000 16   1845 70.436 0 0 13804 0 68696 134 0 18288640 2287957624 00:00:32.126318
6 303 1000 32   3153 75.294 0 0 0 0 235212 0 0 6029312 3367118600 00:00:35.226699
6 308 1000 32   3108 86.457 0 0 403 0 162450 0 0 4546560 3649147168 00:01:01.683903
6 304 1000 64   4758 107.416 0 0 0 0 260021 0 0 5025792 4809522616 00:00:52.768996
6 309 1000 64   4303 101.878 0 0 54 0 203657 0 0 3653632 4830725656 00:00:49.708473
6 305 1000 128   5038 157.901 0 0 0 0 204155 0 0 5623808 5477909168 00:00:55.205861

Nutanix – best practices/hugepages

Averages for test set 7 by scale:

set scale tps avg_latency 90%< max_latency
7 10 864 58.96 140.185 1449.489
7 100 3544 11.698 17.788 134.226
7 1000 3041 13.507 17.801 168.908

Averages for test set 7 by clients:
set clients tps avg_latency 90%< max_latency
7 8 756 23.753 51.795 704.811
7 16 1436 14.054 23.897 165.537
7 32 2319 22.687 46.28 374.571
7 64 3188 40.818 90.168 793.907
7 128 3630 53.91 119.55 1299.893

Averages for test set 7 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
7 10 8   393 47.361 116.378 1376.816
7 10 16   829 22.098 44.794 307.926
7 10 32   908 41.977 94.199 798.561
7 10 64   954 81.333 194.537 1812.006
7 10 128   1163 116.388 287.959 3453.018
7 100 8   1030 7.766 8.31 55.55
7 100 16   1870 8.552 10.298 77.103
7 100 32   3404 9.394 15.076 109.487
7 100 64   5155 12.395 19.971 140.257
7 100 128   6261 20.381 35.285 288.734
7 1000 8   968 8.262 9.169 458.066
7 1000 16   1811 8.831 9.634 64.117
7 1000 32   3116 10.26 13.592 74.337
7 1000 64   4199 15.221 21.207 90.091
7 1000 128   5111 24.963 35.405 157.928

Detail for test set 7:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
7 310 10 8   134 1835.645 1 3835 0 329 3971 0 0 2498560 249012128 00:00:04.354469
7 312 10 8   388 446.059 0 0 0 0 464 0 0 1339392 194379656 00:00:00.889081
7 321 10 8   79 3152.603 0 0 0 0 558 0 0 2940928 217501720 00:00:00.988867
7 326 10 8   970 72.957 0 0 0 0 385 0 0 2740224 198515384 00:00:00.771298
7 313 10 16   766 307.1 0 0 0 0 459 0 0 1646592 193471376 00:00:00.453246
7 315 10 16   428 593.912 0 0 0 0 491 0 0 2572288 185637640 00:00:00.453029
7 322 10 16   1063 158.021 0 0 0 0 426 0 0 1064960 195567984 00:00:00.760965
7 327 10 16   1060 172.673 0 0 0 0 426 0 0 2453504 206913312 00:00:00.797547
7 314 10 32   816 748.275 0 0 0 0 508 0 0 2617344 187636008 00:00:01.044989
7 316 10 32   423 1465.179 0 0 0 0 540 0 0 2748416 214869456 00:00:00.936743
7 323 10 32   1179 447.641 0 0 0 0 479 0 0 2973696 202565752 00:00:00.874704
7 328 10 32   1215 533.148 0 0 0 0 489 0 0 2072576 217104192 00:00:00.970723
7 317 10 64   845 1885.779 0 0 0 0 539 0 0 2822144 213879312 00:00:00.937535
7 318 10 64   426 3125.404 0 0 0 0 654 0 0 3026944 221975760 00:00:01.050127
7 324 10 64   1260 1033.375 0 0 0 0 526 0 0 2605056 214973144 00:00:00.916725
7 329 10 64   1284 1203.464 0 0 0 0 600 0 0 2199552 220323064 00:00:00.969576
7 311 10 128   1210   0 0 0 0 2022 0 0 2248704 193502232 00:00:00.659887
7 319 10 128   836 4569.153 0 0 0 0 652 0 0 2265088 219852280 00:00:00.945038
7 320 10 128   1178   0 0 0 0 559 0 0 2789376 219339120 00:00:00.950063
7 325 10 128   1307 3106.521 0 0 0 0 669 0 0 2916352 223596160 00:00:01.104451
7 330 10 128   1286 2683.379 0 0 0 0 713 0 0 2560000 222670848 00:00:01.003227
7 331 100 8   1032 56.894 0 0 0 0 25840 0 0 2871296 642641848 00:00:03.857754
7 336 100 8   1011 57.23 0 0 0 0 405 0 0 1650688 549765120 00:00:02.698866
7 341 100 8   1046 52.526 0 0 0 0 416 0 0 1789952 544840856 00:00:03.097596
7 332 100 16   1831 85.579 0 0 0 0 3968 0 0 2887680 1140224896 00:00:05.464319
7 337 100 16   1869 77.327 0 0 0 0 735 0 0 2666496 862913472 00:00:03.311832
7 342 100 16   1910 68.403 0 0 0 0 750 0 0 1675264 871859808 00:00:03.744191
7 333 100 32   3326 158.612 0 0 0 0 1938 0 0 4329472 1454170600 00:00:05.127418
7 338 100 32   3417 87.152 0 0 0 0 1330 0 0 3371008 1224333928 00:00:04.201848
7 343 100 32   3468 82.697 0 0 0 0 1356 0 0 3170304 1247157008 00:00:04.111192
7 334 100 64   5060 138.272 0 0 0 0 2384 0 0 4546560 1623365064 00:00:04.988647
7 339 100 64   5126 151.493 0 0 0 0 2003 0 0 4222976 1508054056 00:00:03.960109
7 344 100 64   5278 131.007 0 0 0 0 2065 0 0 3825664 1524149928 00:00:04.26117
7 335 100 128   6133 293.401 0 0 0 0 2755 0 0 5181440 1694596264 00:00:04.461912
7 340 100 128   6350 258.71 0 0 0 0 2504 0 0 4055040 1669683264 00:00:04.426487
7 345 100 128   6301 314.092 0 0 0 0 2485 0 0 4141056 1647306792 00:00:04.629933
7 346 1000 8   974 69.45 0 0 0 0 112610 0 0 3264512 919927000 00:00:33.457318
7 351 1000 8   956 1253.215 0 0 0 0 34013 0 0 2375680 1219402976 00:00:26.023542
7 356 1000 8   974 51.532 0 0 0 0 32933 0 0 2240512 1237963368 00:00:26.657654
7 347 1000 16   1834 67.689 0 0 0 0 177035 0 0 2244608 2028748136 00:00:35.243526
7 352 1000 16   1814 56.893 0 0 13580 0 61177 133 0 36573184 2204906928 00:00:52.684395
7 357 1000 16   1784 67.77 0 0 30 0 60208 0 0 3133440 2171799768 00:00:36.832522
7 348 1000 32   3170 85.321 0 0 0 0 238355 0 0 5783552 3348885120 00:00:39.593968
7 353 1000 32   3090 73.705 0 0 525 0 168056 0 0 3555328 3609379080 00:00:57.914705
7 358 1000 32   3087 63.984 0 0 174 0 120804 0 0 3981312 3597247424 00:00:52.642586
7 349 1000 64   4349 97.57 0 0 0 0 241996 0 0 3780608 4555358208 00:00:51.448927
7 354 1000 64   4128 91.07 0 0 26 0 191139 0 0 2867200 4647265104 00:01:08.918233
7 359 1000 64   4121 81.633 0 0 214 0 164819 0 0 3457024 4642251880 00:01:04.503952
7 350 1000 128   5160 173.859 0 0 0 0 214869 0 0 4681728 5394502136 00:00:55.061775
7 355 1000 128   5094 152.752 0 0 31 0 214376 0 0 2904064 5576875976 00:01:01.504165
7 360 1000 128   5077 147.174 0 0 1069 0 208016 0 0 3915776 5556539288 00:01:05.773985

Postgresql on Nutanix Test Results

Test sets comparison – latencies :

Nutanix – best practices

Averages for test set 4 by scale:

set scale tps avg_latency 90%< max_latency
4 10 1182 39.365 88.138 787.896
4 100 3518 11.721 17.968 170.601
4 1000 3067 13.313 16.832 165.793

Averages for test set 4 by clients:
set clients tps avg_latency 90%< max_latency
4 8 1038 7.702 9.926 63.048
4 16 1605 10.631 15.813 111.423
4 32 2573 15.364 27.823 211.484
4 64 3513 25.879 51.09 435.873
4 128 4215 47.755 100.246 1051.989

Averages for test set 4 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
4 10 8   1035 7.72 13.183 74.105
4 10 16   1084 14.749 26.905 178.843
4 10 32   1208 26.44 56.71 434.356
4 10 64   1281 49.805 111.948 916.073
4 10 128   1299 98.11 231.944 2336.105
4 100 8   1057 7.567 7.961 60.626
4 100 16   1906 8.385 11.107 77.675
4 100 32   3388 9.427 15.073 101.406
4 100 64   5090 12.529 20.551 200.554
4 100 128   6150 20.694 35.15 412.746
4 1000 8   1022 7.82 8.634 54.412
4 1000 16   1825 8.759 9.427 77.751
4 1000 32   3123 10.224 11.685 98.691
4 1000 64   4168 15.302 20.772 190.993
4 1000 128   5197 24.461 33.643 407.118

Detail for test set 4:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
4 181 10 8   1028 79.232 0 0 0 0 3463 0 0 3878912 183429048 00:00:00.924079
4 186 10 8   1046 64.771 0 0 0 0 601 0 0 5738496 166734888 00:00:00.779286
4 191 10 8   1031 78.313 0 0 0 0 884 0 0 7139328 179055272 00:00:00.94822
4 182 10 16   1087 193.201 0 0 0 0 511 0 0 9203712 177052328 00:00:00.768397
4 187 10 16   1082 172.794 0 0 0 0 508 0 0 3743744 151816224 00:00:00.78761
4 192 10 16   1082 170.535 0 0 0 0 880 0 0 6369280 186903872 00:00:00.968328
4 183 10 32   1216 384.817 0 0 0 0 568 0 0 9551872 185865272 00:00:00.87489
4 188 10 32   1197 459.283 0 0 0 0 915 0 0 8069120 183053104 00:00:00.879565
4 193 10 32   1213 458.967 0 0 0 0 941 0 0 4673536 195150904 00:00:01.134
4 184 10 64   1293 902.093 0 0 0 0 633 0 0 8253440 193584952 00:00:00.799526
4 189 10 64   1264 719.84 0 0 0 0 1034 0 0 7708672 196103032 00:00:00.970332
4 194 10 64   1287 1126.286 0 0 0 0 1092 0 0 3424256 198913016 00:00:01.042712
4 185 10 128   1290 1901.848 0 0 0 0 782 0 0 7380992 198666784 00:00:00.96922
4 190 10 128   1310 3212.441 0 0 0 0 1147 0 0 8065024 201840400 00:00:01.146428
4 195 10 128   1296 1894.025 0 0 0 0 859 0 0 3821568 200080976 00:00:00.94773
4 196 100 8   1029 61.612 0 0 0 0 26221 0 0 10113024 636311224 00:00:05.510534
4 201 100 8   1092 58.85 0 0 0 0 1031 0 0 4489216 558362768 00:00:03.7953
4 206 100 8   1049 61.417 0 0 0 0 500 0 0 6344704 538997256 00:00:03.668003
4 197 100 16   1887 84.694 0 0 0 0 4522 0 0 5910528 1116028224 00:00:05.795861
4 202 100 16   1909 79.258 0 0 0 0 1457 0 0 7667712 848623576 00:00:04.36835
4 207 100 16   1923 69.074 0 0 0 0 805 0 0 8638464 848854200 00:00:04.414788
4 198 100 32   3303 102.482 0 0 0 0 2412 0 0 5836800 1438174928 00:00:05.759381
4 203 100 32   3450 90.549 0 0 0 0 2113 0 0 5398528 1219650768 00:00:04.791329
4 208 100 32   3410 111.187 0 0 0 0 1332 0 0 4599808 1193811080 00:00:04.794698
4 199 100 64   4993 190.319 0 0 0 0 2896 0 0 6742016 1616689848 00:00:05.190577
4 204 100 64   5148 223.225 0 0 0 0 2692 0 0 9080832 1467684160 00:00:04.71495
4 209 100 64   5130 188.117 0 0 0 0 2008 0 0 6402048 1478092720 00:00:04.754659
4 200 100 128   6204 431.822 0 0 0 0 3392 0 0 7557120 1662012568 00:00:05.49015
4 205 100 128   6322 413.503 0 0 0 0 2771 0 0 8265728 1637841488 00:00:05.09542
4 210 100 128   5923 392.912 0 0 0 0 2354 0 0 7106560 1588262096 00:00:04.991039
4 211 1000 8   1019 55.765 0 0 0 0 117331 0 0 3289088 932581456 00:00:37.193358
4 216 1000 8   1017 50.686 0 0 37 0 41774 0 0 6639616 1247682680 00:00:42.486634
4 221 1000 8   1031 56.784 0 0 23 0 35064 0 0 10473472 1285242432 00:00:37.832466
4 212 1000 16   1813 108.171 0 0 0 0 174761 0 0 8245248 2026737584 00:00:39.290495
4 217 1000 16   1835 60.632 0 0 109 0 79869 0 0 7864320 2219557376 00:00:38.793936
4 222 1000 16   1826 64.45 0 0 56 0 63632 0 0 7991296 2211681848 00:00:34.954602
4 213 1000 32   3166 77.725 0 0 0 0 238230 0 0 8138752 3300297576 00:00:46.726202
4 218 1000 32   3095 145.124 0 0 8 0 144310 0 0 9142272 3589322040 00:01:01.818296
4 223 1000 32   3110 73.224 0 0 55 0 121096 0 0 8835072 3591895984 00:00:55.778571
4 214 1000 64   4157 162.226 0 0 0 0 232044 0 0 5668864 4417057752 00:01:00.040304
4 219 1000 64   4202 217.421 0 0 175 0 188317 0 0 5779456 4704702936 00:01:03.134753
4 224 1000 64   4146 193.333 0 0 227 0 171946 0 0 6135808 4652858920 00:01:05.18273
4 215 1000 128   5132 422.795 0 0 4345 0 217369 43 0 125702144 5308612592 00:01:21.760186
4 220 1000 128   5142 420.14 0 0 713 0 211262 0 0 9977856 5620639592 00:01:04.329275
4 225 1000 128   5316 378.419 0 0 2758 0 219666 15 0 13701120 5693253856 00:01:07.304406

Nutanix – dbi

Averages for test set 5 by scale:

set scale tps avg_latency 90%< max_latency
5 10 1180 39.556 88.218 687.456
5 100 3595 11.509 17.585 159.463
5 1000 3026 13.495 17.308 237.722

Averages for test set 5 by clients:
set clients tps avg_latency 90%< max_latency
5 8 1039 7.701 9.962 62.508
5 16 1622 10.563 15.751 98.106
5 32 2582 15.298 27.925 181.434
5 64 3572 25.792 50.641 403.439
5 128 4187 48.244 100.908 1062.247

Averages for test set 5 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
5 10 8   1035 7.728 13.236 74.035
5 10 16   1081 14.787 27.151 152.699
5 10 32   1216 26.279 56.683 349.963
5 10 64   1281 49.834 111.09 780.091
5 10 128   1285 99.151 232.933 2080.492
5 100 8   1071 7.462 7.851 62.966
5 100 16   1913 8.357 10.825 78.097
5 100 32   3431 9.308 14.918 103.797
5 100 64   5342 11.949 19.374 199.756
5 100 128   6216 20.467 34.957 352.698
5 1000 8   1010 7.914 8.8 50.522
5 1000 16   1871 8.545 9.276 63.523
5 1000 32   3099 10.308 12.173 90.542
5 1000 64   4092 15.594 21.459 230.47
5 1000 128   5060 25.114 34.834 753.552

Detail for test set 5:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
5 226 10 8   1009 78.105 0 0 0 0 3426 0 0 6483968 181371712 00:00:00.980108
5 231 10 8   1058 65.245 0 0 0 0 417 0 0 10354688 192060632 00:00:00.870046
5 236 10 8   1037 78.755 0 0 0 0 407 0 0 8126464 202758736 00:00:00.93794
5 227 10 16   1089 153.279 0 0 0 0 461 0 0 3371008 175797848 00:00:00.755001
5 232 10 16   1083 158.614 0 0 0 0 429 0 0 7905280 194268672 00:00:00.783676
5 237 10 16   1071 146.203 0 0 0 0 426 0 0 6938624 209602168 00:00:01.06011
5 228 10 32   1245 329.965 0 0 0 0 540 0 0 9777152 195567768 00:00:00.909809
5 233 10 32   1204 374.582 0 0 0 0 482 0 0 6393856 207959000 00:00:00.935597
5 238 10 32   1200 345.342 0 0 0 0 483 0 0 8318976 218615216 00:00:01.043394
5 229 10 64   1272 771.357 0 0 0 0 564 0 0 8577024 217958800 00:00:00.911229
5 234 10 64   1298 820.609 0 0 0 0 540 0 0 10797056 221213704 00:00:00.99838
5 239 10 64   1273 748.306 0 0 0 0 532 0 0 8417280 222210344 00:00:01.125778
5 230 10 128   1291 1625.371 0 0 0 0 751 0 0 5971968 223627496 00:00:00.946291
5 235 10 128   1294 2497.081 0 0 0 0 725 0 0 10723328 225117184 00:00:01.18691
5 240 10 128   1271 2119.024 0 0 0 0 716 0 0 7032832 225046760 00:00:01.089493
5 241 100 8   1046 75.15 0 0 0 0 25948 0 0 11362304 667928080 00:00:04.492989
5 246 100 8   1082 50.812 0 0 0 0 424 0 0 8478720 565695712 00:00:03.882777
5 251 100 8   1086 62.935 0 0 0 0 428 0 0 8966144 584298216 00:00:03.83447
5 242 100 16   1845 71.702 0 0 0 0 3886 0 0 11927552 1147731608 00:00:05.834672
5 247 100 16   1948 64.941 0 0 0 0 761 0 0 12013568 886041720 00:00:04.374209
5 252 100 16   1946 97.649 0 0 0 0 758 0 0 11546624 891069184 00:00:04.435589
5 243 100 32   3352 76.553 0 0 0 0 1919 0 0 10526720 1443019520 00:00:05.514423
5 248 100 32   3482 126.069 0 0 0 0 1360 0 0 12107776 1253916672 00:00:04.533398
5 253 100 32   3459 108.768 0 0 0 0 1351 0 0 11784192 1253522856 00:00:04.662608
5 244 100 64   5107 197.584 0 0 0 0 2410 0 0 6557696 1646231288 00:00:05.094045
5 249 100 64   5511 193.062 0 0 0 0 2157 0 0 12820480 1556700496 00:00:04.668066
5 254 100 64   5409 208.621 0 0 0 0 2111 0 0 12439552 1533689424 00:00:04.770895
5 245 100 128   6179 316.962 0 0 0 0 2763 0 0 11137024 1698765624 00:00:04.964204
5 250 100 128   6233 344.869 0 0 0 0 2467 0 0 9129984 1650312832 00:00:04.689394
5 255 100 128   6235 396.263 0 0 0 0 2474 0 0 12124160 1673031088 00:00:04.879265
5 256 1000 8   997 48.107 0 0 0 0 114974 0 0 4829184 926258936 00:00:25.887712
5 261 1000 8   1019 51.33 0 0 0 0 36510 0 0 10866688 1294744184 00:00:36.438794
5 266 1000 8   1014 52.13 0 0 40 0 33660 0 0 4014080 1273920592 00:00:34.849658
5 257 1000 16   1886 60.54 0 0 0 0 181730 0 0 12644352 2103069864 00:00:31.907607
5 262 1000 16   1857 69.269 0 0 11546 0 63078 114 0 50868224 2256516784 00:00:53.276331
5 267 1000 16   1869 60.76 0 0 77 0 62898 0 0 9760768 2290432272 00:00:34.87621
5 258 1000 32   3186 114.259 0 0 0 0 237857 0 0 13287424 3374860808 00:00:47.236097
5 263 1000 32   3096 91.081 0 0 618 0 165253 0 0 12660736 3611888056 00:00:57.31669
5 268 1000 32   3016 66.286 0 0 354 0 117914 0 0 10612736 3528717560 00:00:56.412759
5 259 1000 64   4019 191.455 0 0 0 0 223911 0 0 13594624 4375378408 00:00:57.461825
5 264 1000 64   4286 267.505 0 0 44 0 199225 0 0 10297344 4824715336 00:01:04.821371
5 269 1000 64   3970 232.449 0 0 295 0 158725 0 0 10301440 4509059528 00:01:07.059475
5 260 1000 128   5092 1492.372 0 0 0 0 216200 0 0 11792384 5272343712 00:00:56.719959
5 265 1000 128   5055 378.502 0 0 54 0 200819 0 0 9957376 5569926896 00:01:00.587166
5 270 1000 128   5035 389.783 0 0 981 0 214334 0 0 10903552 5552797416 00:01:09.615889

Nutanix – best practices/hugepages

Averages for test set 7 by scale:

set scale tps avg_latency 90%< max_latency
7 10 864 58.96 140.185 1449.489
7 100 3544 11.698 17.788 134.226
7 1000 3041 13.507 17.801 168.908

Averages for test set 7 by clients:
set clients tps avg_latency 90%< max_latency
7 8 756 23.753 51.795 704.811
7 16 1436 14.054 23.897 165.537
7 32 2319 22.687 46.28 374.571
7 64 3188 40.818 90.168 793.907
7 128 3630 53.91 119.55 1299.893

Averages for test set 7 by scale, client, and rate limit:
set scale clients rate_limit tps avg_latency 90%< max_latency
7 10 8   393 47.361 116.378 1376.816
7 10 16   829 22.098 44.794 307.926
7 10 32   908 41.977 94.199 798.561
7 10 64   954 81.333 194.537 1812.006
7 10 128   1163 116.388 287.959 3453.018
7 100 8   1030 7.766 8.31 55.55
7 100 16   1870 8.552 10.298 77.103
7 100 32   3404 9.394 15.076 109.487
7 100 64   5155 12.395 19.971 140.257
7 100 128   6261 20.381 35.285 288.734
7 1000 8   968 8.262 9.169 458.066
7 1000 16   1811 8.831 9.634 64.117
7 1000 32   3116 10.26 13.592 74.337
7 1000 64   4199 15.221 21.207 90.091
7 1000 128   5111 24.963 35.405 157.928

Detail for test set 7:
set test scale clients rate_limit tps max_latency chkpts buf_check buf_clean buf_backend buf_alloc max_clean backend_sync max_dirty wal_written cleanup
7 310 10 8   134 1835.645 1 3835 0 329 3971 0 0 2498560 249012128 00:00:04.354469
7 312 10 8   388 446.059 0 0 0 0 464 0 0 1339392 194379656 00:00:00.889081
7 321 10 8   79 3152.603 0 0 0 0 558 0 0 2940928 217501720 00:00:00.988867
7 326 10 8   970 72.957 0 0 0 0 385 0 0 2740224 198515384 00:00:00.771298
7 313 10 16   766 307.1 0 0 0 0 459 0 0 1646592 193471376 00:00:00.453246
7 315 10 16   428 593.912 0 0 0 0 491 0 0 2572288 185637640 00:00:00.453029
7 322 10 16   1063 158.021 0 0 0 0 426 0 0 1064960 195567984 00:00:00.760965
7 327 10 16   1060 172.673 0 0 0 0 426 0 0 2453504 206913312 00:00:00.797547
7 314 10 32   816 748.275 0 0 0 0 508 0 0 2617344 187636008 00:00:01.044989
7 316 10 32   423 1465.179 0 0 0 0 540 0 0 2748416 214869456 00:00:00.936743
7 323 10 32   1179 447.641 0 0 0 0 479 0 0 2973696 202565752 00:00:00.874704
7 328 10 32   1215 533.148 0 0 0 0 489 0 0 2072576 217104192 00:00:00.970723
7 317 10 64   845 1885.779 0 0 0 0 539 0 0 2822144 213879312 00:00:00.937535
7 318 10 64   426 3125.404 0 0 0 0 654 0 0 3026944 221975760 00:00:01.050127
7 324 10 64   1260 1033.375 0 0 0 0 526 0 0 2605056 214973144 00:00:00.916725
7 329 10 64   1284 1203.464 0 0 0 0 600 0 0 2199552 220323064 00:00:00.969576
7 311 10 128   1210   0 0 0 0 2022 0 0 2248704 193502232 00:00:00.659887
7 319 10 128   836 4569.153 0 0 0 0 652 0 0 2265088 219852280 00:00:00.945038
7 320 10 128   1178   0 0 0 0 559 0 0 2789376 219339120 00:00:00.950063
7 325 10 128   1307 3106.521 0 0 0 0 669 0 0 2916352 223596160 00:00:01.104451
7 330 10 128   1286 2683.379 0 0 0 0 713 0 0 2560000 222670848 00:00:01.003227
7 331 100 8   1032 56.894 0 0 0 0 25840 0 0 2871296 642641848 00:00:03.857754
7 336 100 8   1011 57.23 0 0 0 0 405 0 0 1650688 549765120 00:00:02.698866
7 341 100 8   1046 52.526 0 0 0 0 416 0 0 1789952 544840856 00:00:03.097596
7 332 100 16   1831 85.579 0 0 0 0 3968 0 0 2887680 1140224896 00:00:05.464319
7 337 100 16   1869 77.327 0 0 0 0 735 0 0 2666496 862913472 00:00:03.311832
7 342 100 16   1910 68.403 0 0 0 0 750 0 0 1675264 871859808 00:00:03.744191
7 333 100 32   3326 158.612 0 0 0 0 1938 0 0 4329472 1454170600 00:00:05.127418
7 338 100 32   3417 87.152 0 0 0 0 1330 0 0 3371008 1224333928 00:00:04.201848
7 343 100 32   3468 82.697 0 0 0 0 1356 0 0 3170304 1247157008 00:00:04.111192
7 334 100 64   5060 138.272 0 0 0 0 2384 0 0 4546560 1623365064 00:00:04.988647
7 339 100 64   5126 151.493 0 0 0 0 2003 0 0 4222976 1508054056 00:00:03.960109
7 344 100 64   5278 131.007 0 0 0 0 2065 0 0 3825664 1524149928 00:00:04.26117
7 335 100 128   6133 293.401 0 0 0 0 2755 0 0 5181440 1694596264 00:00:04.461912
7 340 100 128   6350 258.71 0 0 0 0 2504 0 0 4055040 1669683264 00:00:04.426487
7 345 100 128   6301 314.092 0 0 0 0 2485 0 0 4141056 1647306792 00:00:04.629933
7 346 1000 8   974 69.45 0 0 0 0 112610 0 0 3264512 919927000 00:00:33.457318
7 351 1000 8   956 1253.215 0 0 0 0 34013 0 0 2375680 1219402976 00:00:26.023542
7 356 1000 8   974 51.532 0 0 0 0 32933 0 0 2240512 1237963368 00:00:26.657654
7 347 1000 16   1834 67.689 0 0 0 0 177035 0 0 2244608 2028748136 00:00:35.243526
7 352 1000 16   1814 56.893 0 0 13580 0 61177 133 0 36573184 2204906928 00:00:52.684395
7 357 1000 16   1784 67.77 0 0 30 0 60208 0 0 3133440 2171799768 00:00:36.832522
7 348 1000 32   3170 85.321 0 0 0 0 238355 0 0 5783552 3348885120 00:00:39.593968
7 353 1000 32   3090 73.705 0 0 525 0 168056 0 0 3555328 3609379080 00:00:57.914705
7 358 1000 32   3087 63.984 0 0 174 0 120804 0 0 3981312 3597247424 00:00:52.642586
7 349 1000 64   4349 97.57 0 0 0 0 241996 0 0 3780608 4555358208 00:00:51.448927
7 354 1000 64   4128 91.07 0 0 26 0 191139 0 0 2867200 4647265104 00:01:08.918233
7 359 1000 64   4121 81.633 0 0 214 0 164819 0 0 3457024 4642251880 00:01:04.503952
7 350 1000 128   5160 173.859 0 0 0 0 214869 0 0 4681728 5394502136 00:00:55.061775
7 355 1000 128   5094 152.752 0 0 31 0 214376 0 0 2904064 5576875976 00:01:01.504165
7 360 1000 128   5077 147.174 0 0 1069 0 208016 0 0 3915776 5556539288 00:01:05.773985

Conclusions

The overall results we obtained with the Nutanix appliance by applying the Nutanix’s Best Practices for Postgresql were good. However, things could still be greatly improved depending on your workload.
For a purely OLTP workload mostly composed of reads, this will be fine. But mixed DSS/OLTP workloads might require additional attention. Unfortunately for us, we needed to test postgresql version 9.5
due to business constraints, but postgresql 11, which supports parallelism, partitioning and a number of DDL enhancements, is better suited for mixed workloads.

A proper appliance configuration for database use cases is essential before even starting any database performance tests. There are many moving parts in this kind of hybrid infrastructure and each layer requires special attention. The final result, however, was very good.

A final word of caution on “best practices”: they will be as good as your tests. Pgbench tests provide indications of how the database could behave under certain parameters of scale and concurrency, but the test model is very simple and may be very far from a real wo
rkload. Pgreplay could provide another insight on how a real workload could behave in your test environment.

In the next post I will publish the same comparative analysis on an Amazon AWS EC2 iXL NVMe instance.


Alexis

Alexis is the founder of Aleph Technologies, a data infrastructure consulting and professional services provider based in Brussels, Belgium.

More Posts - Website

Follow Me:
TwitterLinkedIn