As per my daily work at eNovance, I do like benchmarking but prior to any good one, it’s important to check the sanity of the system.
This article will be about understanding how your local storage device behave and the associated tools.
Understanding your environment
Today’s Server density is pretty high and could make setups like 24 internal disks like on a 2U Dell R720xd. You can even add more disks device by attaching external storage to it. I’ve met such configuration with a partner (CloudWatt) with a total of 36 disks attached to a single physical server (24 x 2.5″ internal disks + 12 x 3.5″ external disks).
But how to run a benchmark on such big beast while not understanding :
- how every single disks runs ?
- how does the controller is playing while doing IOs on all the disks at the same time ?
- is there any disks not performing as expected ?
To answer all those questions, I’m using fio.
Using Fio to benchmark storage devices
fio written by Jens Axboe is clearly my favourite storage benchmark tool.
It got really made for being versatile by featuring :
- many IO engines (sync, mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio, and more)
- complex IO pattern (controlling read/write and sequential/random distribution)
- writing job control files
- both file or block device storage targets support
- threaded or forked multiple jobs
- {latency|iops|bandwidth} performance traces to be used later for complementary analysis.
fio runs on many operating systems and is of course a free software under GPL.
Introducing genfio
Even if fio is a very versatile tool, it’s pretty boring to setup a configuration file for testing 36 disks. At least, no benchmark tool I know give helpers to ease the configuration of such large test.
genfio is a 3rd party tool included in fio since release 2.1.2. A very simple command line generates big jobs.
In this 36 disks case, genfio allow you to create a single job file that will test :
- one disk at a time
- all disks at the same time
- sequential read, sequential write, random read, random write patterns
- define a list of block size you want to test
- a time based runtime for each test
- separate log file for each test for IOPS and Bandwidth
- with or without caching
As an example, you can have the following command line :
[root@host] genfio -d /dev/sda,/dev/sdb,/dev/sdc -m write,randwrite,read,randread -b 4k,128k,1m -D 32 -B 'doing_something_before_the_job' -x <outputdir>/ -r 300 -a Generating outputdir/localhost-4k,128k,1m-all-write,randwrite,read,randread-sda,sdb,sdc.fio 48 jobs generated Estimated Time = 14400 seconds : 4 hour 0 minutes
As shown in this example, the generated fio file will be made of 48 individual jobs of 300 seconds to complete the benchmark which last 4 hours. A single execution of this fio job will runs all the tests without any human input.
Pros:
- no need of boring copy/paste to get numerous job
- no possible human typo/mistake
- no need of human inputs to run very long jobs
- automated job naming
- automated log file naming
- automated execution of numerous jobs
Cons:
- not all fio features are available yet
Plotting the result
After running a complex fio job, a great amount of data are generated. Understanding how the benchmark run is pretty complex as :
-
numerical values (average, min, max, standard deviation) doesn’t give a complete overview
- If the storage device stops providing IOs for a short, the metrics doesn’t reports it in a obvious way)
- graphical plotting huge data requires a great time to setup the plotting properly
-
graphical plotting doesn’t always provides the good view
- plotting 36 traces on the same graph isn’t easy to read
- estimating the average bandwidth of a graph isn’t easy neither
To help a plotting massive fio outputs, fio2gnuplot tool got developed and contributed to fio and included since release 2.1.2
fio2gnplot selects fio’s output file by using regular expressions, parse them and generate a series of graphics by using gnuplot.
To understand how every since part of the bench run, a series of graphics will be generate for each trace file :
- a raw graph to show the real values reported during the test
- a smooth graph using CSplines plotting to ease the performance reading
- a Bezier’s based graph plotting to see the trends (performance peaks are typically hidden)
- a green line is printed on each kind of graph to report the average value during the benchmark
- an automatic naming of axis and title is done accordingly to the type of benchmark (IO vs Bandwidth, Block size reporting)
To understand how the global bench was running, a series of graphics will be generated to compare the various trace file :
- a raw graph to show the real values reported during the test
- a smooth graph using CSplines plotting to ease the performance reading
- a Bezier’s based graph plotting to see the trends (performance peaks are typically hidden)
- a green line is printed on each kind of graph to report the average value during the benchmark
- an automatic naming of axis and title is done accordingly to the type of benchmark (IO vs Bandwidth, Block size reporting)
- a 3D graph reporting a surface
- the flatter the surface is, the closer performances are
- a graph bar to compare the following computation based on each trace
- Average value
- Min value
- Max value
- Standard Deviation
Bar graphs are very useful to make a quick comparison between many traces while not having the readability issues of plotting many lines on a single graph.
Sample plotting are available at the end of this article.
On one hand, fio2gnuplot generates many traces file and many could be useless regarding the kind of information you are interested on. On the other hand, fio2gnuplot runs the same series of graphs automatically in a very easy way letting users being concentrated at reading the most interesting ones.
Using fio2gnuplot, graphing fio’s output is become easier and quicker. Every fio user benefits of graphical rendering while not having the need of understanding how to plot them using gnuplot.
[…] By eNovance Engineering Teams: Adding new 3rd party tools in Fio […]
[…] eNovance:在FIO添加新的第三方工具 […]
Hi Erwan,
I have started using fio for some of my IO benchmarking projects. So far I am using the inbuilt fio_generate_plots tool to plot basic bandwidth and latency graphs. However, the graphs I get seem pretty basic compared to the examples you have posted.
How can I install fio2gnuplot? I did some some searching and did not find any resources.
Dear Harsh,
fio_generates_plots is the original work integrated to fio. fio2gnuplot is a new tool that have been part of fio since release 2.1.2. So please update fio or take my tools directly from the git : http://git.kernel.dk/?p=fio.git;a=tree;f=tools/plot;h=d95cec812fe0d95fdffe5d5d550a23a48f25c7dd;hb=HEAD
fio2gnuplot is not tied to any special version of fio, so feel free to use it on any output of fio you may have.
If you need more assistance on how to use it, feel free to join #fio on oftc.
Thanks for the explanation Erwan. I have started using fio2gnuplot and for single thread (numjobs=1) fio configs, the results are accurate.
Have you tried running fio2gnuplot for multiple threads i.e numjobs=4? I am getting garbled/unclear plots. The commands I am using is:
fio2gnuplot.py -g -i
fio2gnuplot.py -g -b
Hi Erwan,
regarding the results it is very sad you did not make some deeper analysis.
If you can explain me what motivate you to use data destructive cspline and bezier smoothing instead of standard filtering tools I would be happy to read you !
May be you ran an fft on your results and results were not relevant ?
I encourage you to make a second post focused on true analysis.
Thanks again for sharing your data.
Yuhan.
ps: sharing raw data would be awesome
Let me clarify this blog post. It was made to present a tool and how it works with a couple of samples output to help people understand its features.
This blog post is not about analyzing a particular benchmark, it’s really about presenting a tool.
Regarding your concern about the “true analysis”, you surely missed that the tool is able to provide a _raw_ graphing _or_ a cspline _or_ a bezier rendering.
It’s really up to the user to decide which output he wants regarding its needs.
Really, this tool is about turning a text output of the wonderful fio tool into something easier to read through a graphical output while having several outputs.
I hope this clarify the blog post and answer your concerns.
Erwan,
[…] You also can look at the nice eNovance blog post on it. […]
I love this article so much! Thanks! But I have a question. I’m testing 72 hard drives at the same time. So genfio will generate a very long filename jobfile and it will report file name too long. Is there any way to specify a short name instead? Thank you so much!!!
I modified the code and the problem solved!! Thanks!
Erwan,
I’m running into an issue with fio2gnuplot where log files generated by fio are not graphed. I tried using different versions of fio but continue to get the same problem with a “$3.png” is generated and errors:
-rw-r–r– 1 root root 306 Jul 6 15:37 test.fio
-rw-r–r– 1 root root 5225 Jul 6 15:38 4k.results_iops.log
-rw-r–r– 1 root root 0 Jul 6 15:39 $3.png
-rw-r–r– 1 root root 83 Jul 6 15:42 result.stddev
-rw-r–r– 1 root root 81 Jul 6 15:42 result.min
-rw-r–r– 1 root root 85 Jul 6 15:42 result.max
-rw-r–r– 1 root root 80 Jul 6 15:42 result.global
-rw-r–r– 1 root root 85 Jul 6 15:42 result.average
-rw-r–r– 1 root root 3398 Jul 6 15:42 result
-rw-r–r– 1 root root 656 Jul 6 15:42 mymath
-rw-r–r– 1 root root 248 Jul 6 15:42 mygraph
-rw-r–r– 1 root root 3353 Jul 6 15:42 gnuplot_temp_file.0
Running gnuplot Rendering
“/usr/share/fio//math.gpm”, line 25: no datablock named $1
“/usr/share/fio//graph2D.gpm”, line 25: no datablock named $1
I am facing the same problem.