Storage performance testing with vdbench

In my post on storage performance testing with IOmeter I showed a straightforward way to analyse storage performance. IOmeter’s interface makes a basic performance test quick and simple. It is not the ideal tool for every measurement, though, because the workload it generates does not account for what modern storage systems do. IOmeter offers a choice between repeating bytes, which deduplicate completely, and pseudo-random data, which neither deduplicates nor compresses. Neither suits testing a modern storage system.

vdbench is the better choice here. It is a CLI-based performance analysis and load generator that allows an enormous amount of configuration detail. vdbench generates storage load from one or several servers, so you control parameters such as read and write, random and sequential, the I/O size of the transfer and a great deal more.

Getting the maximum out of a storage system takes more than a performance test from a single host or virtual server. To reach the maximum you have to run a workload from several hosts onto several volumes. Most environments are virtualised, so this post focuses on a VMware vSphere platform, but the principles are the same elsewhere.

When running performance benchmarks, the number of hosts and HBAs (FC or iSCSI) decides how good the results get. Usually 4 to 8 hosts with dual HBAs, using all the storage host ports, is enough. There can be particular configurations and benchmarks that need more, a 100% sequential read benchmark in particular, because the host PCI bottleneck limits the throughput of each server.

Those 4 to 8 VMs should use several VMDKs, roughly 2 to 4 per VM, 8 to 16 in total, each at least 500 GB, on different datastores. Each VM should run on its own ESXi server. That way you use the maximum of the available ESXi host resources and of the datastores’ IO queues.

vdbench is a CLI tool, developed by StorageTek, taken over by Sun and currently maintained by Oracle. The idea is to help engineers generate disk and tape I/O workloads to check storage performance and data integrity. It runs on UNIX, Windows, Linux and OS X. Downloading vdbench needs a free Oracle account.

Because vdbench is written in Java, a Java runtime environment has to be installed on the servers. A freely available Java JRE is here.

Before a performance test starts, the storage array has to be filled with data. That task is called prefilling.

Prefilling the storage array

When benchmarking all-flash storage arrays, the array is usually filled with data first. The fill workload is a special case, used to fill an array with data before testing begins. The workload performs 100% sequential I/O with a block size of 256k, until every volume has been written once or until the elapsed time parameter is reached. If the goal is to write 100% of all volumes, the “elapsed time” parameter has to account for the environment’s write rate and the combined capacity of all the volumes. Choose too small a value and the test stops before the volumes are fully written. Choose too large a value and the volumes are fully written and the test ends before the elapsed time is reached. Choosing a very large elapsed time is recommended when using the fill workload.

The “fill” and “sample” workloads cannot be combined with other workloads, or the results will not be presented correctly in the report. If a series of system loads has to run without interruption, fill can be mixed with other loads, but the runtime option m (mixed workloads) has to be given. If you specify the fill workload together with other workloads in the same test, the right workloads run against the storage under test, but the resulting report may not represent the data accurately and the reporting step can even fail.

The reason the fill workload must not be mixed with others is the nature of the workload. It is meant to fill LUNs in preparation for a benchmark workload. It is not run at varying intensities to explore the workload characteristics, as is wanted with other workloads. Every other workload is run so that a saturation curve emerges. Mixing those workloads in the same report is not currently supported.

The following vdbench script (workload file) shows the syntax needed to prefill the array:

hd=default,shell=vdbench,user=administrator
hd=one
sd=sd_01,lun=\\.\PHYSICALDRIVE1,size=600G,host=one,threads=16
sd=sd_02,lun=\\.\PHYSICALDRIVE2,size=600G,host=one,threads=16
sd=sd_03,lun=\\.\PHYSICALDRIVE3,size=600G,host=one,threads=16
sd=sd_04,lun=\\.\PHYSICALDRIVE4,size=600G,host=one,threads=16
*This is our workload definition. Here we use seekpct=eof to tell vdbench to:
*write random data until it completely writes to the entire lun , rdcpct=0 sets the
*workload to 100% write and xfersize determines the block size. In this case we
*set the block size to 256k
wd=wd1,sd=*,hd=*,seekpct=eof,rdpct=0,xfersize=256k
*This is our run definition. It tells vdbench to run the workload definition
* defined in wd1 above at the maximum iorate it can sustain. The elapsed time *does not matter here. The script will stop when eof is reached as per the *workload definition.
rd=rd1,wd=wd1,elapsed=144000,interval=1,iorate=max,openflags=directio

Deduplication and compression

The results of some tests depend on the data pattern written to the storage. Data compaction such as deduplication and compression produces different results and different levels of performance as the pattern sent to the storage changes.

A storage array that supports deduplication, for example, has to write every block to the medium when every block is unique. Send such a unique pattern to an array that implements deduplication and the overhead of deduplication degrades performance (the cost of deduplication) without any of the benefit, because the data is all unique. For a valid test of an array’s deduplication function it matters that the pattern you send carries a certain amount of redundancy.

vdbench supports both deduplication and compression, through these parameters in the vdbench script:

dedupratio=1.5
compratio=3.0
dedupunit=16k

Different values can be used for dedupratio and compratio, but a dedupratio of 1.5 and a compratio of 3.0 produce benchmark values closest to those shown in HPE NinjaSTARS.

If the deduplication= or compression= parameters are absent from the configuration file, the data pattern generated for the test is all unique data that cannot be compressed. By default, vdbench writes an incompressible, non-deduplicable random data pattern.

How do I configure the vdbench config file?

Storage definitions

‘sd=’ identifies each storage definition uniquely. The SD name is used by the workload definition (WD) and run definition (RD) parameters to work out which SDs to use for their workload. Give “default” as the SD name and the values entered become the default for all SD parameters, as follows.

‘lun=’ describes the name of the raw disk or the file name of the file system to use. Take care not to name a disk holding data you do not want to lose.

‘host=name’ is only needed for a multi-host test where the LUN names differ per host. If a LUN is called \\.\PHYSICALDRIVE5 on hostA but \\.\PHYSICALDRIVE7 on hostB, for example, you have to configure vdbench accordingly.

‘size=’ describes the size of the raw disk or file, in bytes, kilobytes, megabytes, gigabytes or terabytes (k/m/g/t). Without it, the size is taken from the raw disk or the file. vdbench supports addresses larger than 2 GB.

‘threads=nn’ gives the maximum number of concurrent I/Os that can be outstanding for this SD. Note that depending on the storage system, some of those I/Os may be queued inside the operating system. Be somewhat conservative with this value: too many outstanding I/Os and the latency climbs again.

The table below shows the queue depth of each physical adapter.

Type	Default value	Value name
QLogic	64	qlfxmaxqdepth
Brocade	32	bfa_lun_queue_depth
Emulex	32	lpfc0_lun_queue_depth
Software iSCSI	128	iscsivmk_LunQDepth

To fill in the parameters for sd, “vdbench sds” lists the disks that are present.

Workload definitions

A workload definition is a single line beginning with ‘wd=’. You can define several workloads in the vdbench file, but usually only one is selected in the run definition (‘rd=’).

Some example workloads:

*prefill - 256k seq
wd=wd1,host=*,sd=*,rdpct=0,xf=256k,seekpct=eof
*8k 100Read 100Random - Small Random Reads
wd=wd2,host=*,sd=*,rdpct=100,xf=8k,seekpct=100
*8k 100Write 100Random - Small Random Writes
wd=wd3,host=*,sd=*,rdpct=0,xf=8k,seekpct=100
*16k 60Read 40Write 100Random - Small Mix-Read-Write Random
wd=wd4,host=*,sd=*,rdpct=60,xf=16k,seekpct=100
*256k 100Read 100Seq - Large Sequential Read
wd=wd5,host=*,sd=*,rdpct=100,xf=256k,seekpct=0
*256k 100Write 100Seq - Large Sequential Write
wd=wd6,host=*,sd=*,rdpct=0,xf=256k,seekpct=0
*32k 100Read 100Random - Avg Random Reads
wd=wd7,host=*,sd=*,rdpct=100,xf=32k,seekpct=100
*64K 70Read 30Write 100Random - Avg Mixed-Read-Write Random
wd=wd8,host=*,sd=*,rdpct=100,xf=80k,seekpct=100
*4k-50% 32k-50% 60Read 40Write 100Random - Mixed BlockSize Random
wd=wd9,host=*,sd=*,rdpct=60,xf=(4k,50,32k,50),seekpct=100

Run definition

The run definition says which of the previously defined workloads have to run, which I/O rates have to be generated and how long the workloads should run for. Depending on the parameters used, one run definition can produce several actual runs.

Example:
rd=run1,wd=(wd1,wd2),warmup=60,forthreads=(32), iorate=1000,elapsed=60,interval=5

‘rd=name’ defines a unique name for this run. Run names are used in the output reports to identify which run is being reported on. Enter ‘default’ as the RD name and the values entered become the default for all following SD parameters.

‘wd=’ identifies workloads to run. Specify a single workload as ‘wd=wd1’ or multiple workloads either by entering them individually ‘wd=(wd1,wd2,wd3)’, a range ‘wd=(wd1-wd3)’ or by using a wildcard character: ‘wd=wd*’.

warmup=60 means performance figures from the warm-up phase are not counted in the report.

forthreads=(32) overrides the thread parameter of the storage definition (optional).

‘iorate=’ specifies one or more I/O rates.

iorate=100 runs a workload of 100 IOPS

iorate=(100,200) runs a workload of 100 IOPS, then 200 IOPS

‘elapsed=nn’ gives the elapsed time in seconds for each run. It has to be at least twice the reporting interval below. Each requested workload runs for ‘elapsed=’ seconds while detailed performance interval statistics are reported every ‘interval=’ seconds.

‘interval=nn’ gives the duration in seconds of each reporting interval. At the end of each one, all the collected statistics are reported.

There are a good many more run definition parameters; the vdbench user guide has the detail.

An example workload parameter file

concatenate=no
dedupratio=1.5
compratio=3.0
dedupunit=16k
hd=default,vdbench=c:\vdbench,user=Administrator,shell=vdbench,jvms=4 hd=one,system=localhost
sd=default,openflags=directio,size=950g
sd=sd1,host=one,Lun=\\.\PHYSICALDRIVE1
sd=sd2,host=one,Lun=\\.\PHYSICALDRIVE2
sd=sd3,host=one,Lun=\\.\PHYSICALDRIVE3
sd=sd4,host=one,Lun=\\.\PHYSICALDRIVE4
sd=sd5,host=one,Lun=\\.\PHYSICALDRIVE5
sd=sd6,host=one,Lun=\\.\PHYSICALDRIVE6
sd=sd7,host=one,Lun=\\.\PHYSICALDRIVE7
sd=sd8,host=one,Lun=\\.\PHYSICALDRIVE8
*prefill - 256kseq
wd=wd1,host=*,sd=*,rdpct=0,xf=256k,seekpct=eof
*8k 100Read 100Random - Small Random Reads
wd=wd2,host=*,sd=*,rdpct=100,xf=8k,seekpct=100
*8k 100Write 100Random - Small Random Writes
wd=wd3,host=*,sd=*,rdpct=0,xf=8k,seekpct=100
*16k 60Read 40Write 100Random - Small Mix-Read-Write Random
wd=wd4,host=*,sd=*,rdpct=60,xf=16k,seekpct=100
*256k 100Read 100Seq - Large Sequential Read
wd=wd5,host=*,sd=*,rdpct=100,xf=256k,seekpct=0
*256k 100Write 100Seq - Large Sequential Write
wd=wd6,host=*,sd=*,rdpct=0,xf=256k,seekpct=0
*32k 100Read 100Random - Avg Random Reads
wd=wd7,host=*,sd=*,rdpct=100,xf=32k,seekpct=100
*64K 70Read 30Write 100Random - Avg Mixed-Read-Write Random
wd=wd8,host=*,sd=*,rdpct=100,xf=80k,seekpct=100
*8k 100Read 100Random
wd=wd9,host=*,sd=*,rdpct=100,xf=4k,seekpct=100
*8k 100Write 100Random
wd=wd10,host=*,sd=*,rdpct=0,xf=4k,seekpct=100
rd=rd1,wd=wd2,el=900,in=1,warmup=300,forthreads=(32),iorate=max

This example file defines 8 LUNs, one host (localhost), several workload definitions and one run definition. It runs workload ‘wd2’ (8k 100Read 100Random, small random reads) with a warm-up of 300 seconds, so the first 5 minutes of collected performance data are excluded, and runs at maximum IO rate for 900 seconds with 32 threads per LUN.

An example mixed workload parameter file

concatenate=no
dedupratio=1.5
compratio=3.0
dedupunit=16k
hd=default,vdbench=c:\vdbench50407,user=Administrator,shell=vdbench,jvms=4
hd=one,system=localhost
sd=default,openflags=directio,size=950g
sd=sd1,host= one,Lun=\\.\PHYSICALDRIVE1
sd=sd2,host= one,Lun=\\.\PHYSICALDRIVE2
sd=sd3,host= one,Lun=\\.\PHYSICALDRIVE3
sd=sd4,host= one,Lun=\\.\PHYSICALDRIVE4
sd=sd5,host= one,Lun=\\.\PHYSICALDRIVE5
sd=sd6,host= one,Lun=\\.\PHYSICALDRIVE6
sd=sd7,host= one,Lun=\\.\PHYSICALDRIVE7
sd=sd8,host= one,Lun=\\.\PHYSICALDRIVE8
*prefill - 256kseq
wd=wd1,host=*,sd=*,rdpct=0,xf=256k,seekpct=eof
*8k-50% 32k-50% 60Read 40Write 100Random - Mixed BlockSize Random
wd=wd9,host=*,sd=*,rdpct=60,xf=(8k,50,32k,50),seekpct=100
*8k 60Read 40Write 100Random 128K 50Read 50Write 100Seq **wd10 and wd11 will complete this
wd=wd10,host=*,sd=*,rdpct=60,xf=8k,seekpct=100
wd=wd11,host=*,sd=*,rdpct=50,xf=128k,seekpct=0
*When running wd10 & wd11 loop as *wd=(wd10,wd11) at bottow in rd= section
*4k-20% 8k-20% 16k-20% 64k-20% 128k-10% 256k-10% 65Read 35Write 70Random
wd=wd16,host=*,sd=*,rdpct=65,xf=(4k,20,8k,20,16k,20,64k,20,128k,10,256k,10),seekpct=70
rd=rd1,wd=wd9,el=900,in=1,warmup=300,forthreads=(32),iorate=max

This runs workload ‘wd9’ (8k-50% 32k-50% 60Read 40Write 100Random, mixed block size random) with a warm-up of 300 seconds, so the first 5 minutes of collected performance data are excluded, and runs at maximum IO rate for 900 seconds with 32 threads per LUN.

Starting vdbench with the parameter file

This command starts vdbench with the parameter file you created.

vdbench.bat -f parameter.file

vdbench from several servers (multi-host)

To run vdbench simultaneously but under central control from a master host, extend the parameter file with the following and adjust the storage definitions accordingly.

hd=host1,system=localhost
hd=host2,system=192.168.1.2
hd=host3,system=192.168.1.3
hd=host4,system=192.168.1.4
sd=sd1,host=host1,Lun=\\.\PHYSICALDRIVE1
sd=sd2,host=host2,Lun=\\.\PHYSICALDRIVE1
sd=sd3,host=host3,Lun=\\.\PHYSICALDRIVE1
sd=sd4,host=host4,Lun=\\.\PHYSICALDRIVE1

On the remote hosts, start the vdbench remote daemon with this command so the master host can connect:

vdbench rsh

Prefilling the array with multi-host

To prefill in a multi-host test environment, build the parameter file like this:

hd=default,shell=vdbench,user=administrator
hd=host1,system=localhost
hd=host2,system=192.168.1.2
hd=host3,system=192.168.1.3
hd=host4,system=192.168.1.4
sd=sd_11,host=host1,lun=\\.\PHYSICALDRIVE1,size=600G,threads=16
sd=sd_12,host=host1,lun=\\.\PHYSICALDRIVE2,size=600G,threads=16
sd=sd_13,host=host1,lun=\\.\PHYSICALDRIVE3,size=600G,threads=16
sd=sd_14,host=host1,lun=\\.\PHYSICALDRIVE4,size=600G,threads=16
sd=sd_21,host=host2,lun=\\.\PHYSICALDRIVE1,size=600G,threads=16
sd=sd_22,host=host2,lun=\\.\PHYSICALDRIVE2,size=600G,threads=16
sd=sd_23,host=host2,lun=\\.\PHYSICALDRIVE3,size=600G,threads=16
sd=sd_24,host=host2,lun=\\.\PHYSICALDRIVE4,size=600G,threads=16
sd=sd_31,host=host3,lun=\\.\PHYSICALDRIVE1,size=600G,threads=16
sd=sd_32,host=host3,lun=\\.\PHYSICALDRIVE2,size=600G,threads=16
sd=sd_33,host=host3,lun=\\.\PHYSICALDRIVE3,size=600G,threads=16
sd=sd_34,host=host3,lun=\\.\PHYSICALDRIVE4,size=600G,threads=16
sd=sd_41,host=host4,lun=\\.\PHYSICALDRIVE1,size=600G,threads=16
sd=sd_42,host=host4,lun=\\.\PHYSICALDRIVE2,size=600G,threads=16
sd=sd_43,host=host4,lun=\\.\PHYSICALDRIVE3,size=600G,threads=16
sd=sd_44,host=host4,lun=\\.\PHYSICALDRIVE4,size=600G,threads=16
*This is our workload definition. Here we use seekpct=eof to tell vdbench to:
*write random data until it completely writes to the entire lun , rdcpct=0 sets the
*workload to 100% write and xfersize determines the block size. In this case we
*set the block size to 256k
wd=wd1,host=*,sd=sd_*,seekpct=eof,rdpct=0,xfersize=256k
*This is our run definition. It tells vdbench to run the workload definition
* defined in wd1 above at the maximum iorate it can sustain. The elapsed time *does not matter here. The script will stop when eof is reached as per the *workload definition.
rd=rd1,wd=wd*,elapsed=144000,interval=1,iorate=max,openflags=directio

Next step

Would you like to know more about measuring and analysing storage systems? We have years of experience from countless storage projects and support cases and can help you too.