Re: fi massive testing

On Wed, Mar 05 2008, Joshua Root wrote:
> Zhang, Yanmin wrote:
> >Do you have any recommendations about the job file which could simulate
> >application
> >models?
>
> If you're interested in simulating the workload as seen at the block
> layer, I have a script that will generate an fio job that approximates
> the workload seen in a blktrace. There's a download link here:
> <http://www.gelato.unsw.edu.au/IA64wiki/JoshuaRoot/MarkovChains>
>
> As that page indicates, the primary purpose of the package is a little
> different, and I just added the fio job creation recently. You first
> need to run prep.sh on your blktrace files as described in the first
> paragraph of the README. But then, instead of buildChain.py, you want to
> run traceStats.py like so (which requires SciPy to be installed):
>
> ./traceStats.py -i OPFILE -f > output.job
>
> You'll probably want to change a few of the values in the global section.
>
> Most of the aspects of the workload are reproduced pretty well. Some
> parts are still a little rough though; like the queue depth and
> thinktime (they end up really huge sometimes, so I artificially cap
> them). Sync vs async also doesn't work (everything ends up as async).
> Same with priorities.

That's pretty cool! I've long played with the idea of enhancing fio a
bit, so that you could more closely model anything. Right now the io_u
tells you where to do IO and and what offset. It was also extended to
pass information on doing an fsync. My plan would be to extend it even
more, so you could describe things like file open etc with the io_u.
Once that is done, you could inject a pending list of things to do into
fio, or even generate it on-the-fly by parsing a workload description
language.

Of course, you can also replay blktrace traces directly with fio if you
want, that works ok right now.

-- 
Jens Axboe