Re: 'offset' doesn't work in fio-1.16.9

From: ljzhang,Yaxin Hu,Jianchao Tang <nonggia_at_sjtu.edu.cn>
Date: Fri, 27 Jul 2007 19:33:18 +0800

On Fri, 2007-07-27 at 13:27 +0200, Jens Axboe wrote:
> On Fri, Jul 27 2007, ljzhang,Yaxin Hu,Jianchao Tang wrote:
> > 1. The job file we prepared:
> > ----------offset-----------------
> > [global]
> > filename=./temp/HOWTO
> > nrfiles=1
> > rw=read
> > size=8K
> > offset=16k
> > bs=1k
> > thread
> > loops=10
> > [offset]
> > description="Option 'offset' doesn't work."
> > ---------------------------------
> > Besides, we copied the file HOWTO to the directory ./temp/.
> > So we thought fio would read 8K text sequentially from the point 16K off
> > the beginning of HOWTO.
> >
> > we run fio with gdb to see what was read out after the first io, and we
> > got this:
> > ---------------------------------
> > ... ...
> > (gdb) bt
> > #0 0x0804cb4d in do_io (td=0xb7ca8000) at fio.c:449
> > #1 0x0804e1b1 in thread_main (data=0xb7ca8000) at fio.c:873
> > #2 0xb7fbc504 in start_thread ()
> > from /lib/tls/i686/cmov/libpthread.so.0
> > #3 0xb7f1a51e in clone () from /lib/tls/i686/cmov/libc.so.6
> > (gdb) x/4w io_u->xfer_buf
> > 0x8073f08: 0x6c626154 0x666f2065 0x6e6f6320
> > 0x746e6574
> > (gdb)
> > ---------------------------------
> > The above was performed after the first io finished by td_io_queue.
> > The contents read out as we can see was the ASCII codes for 'Table of
> > content', which was right from the beginning of file HOWTO.
> > That means fio read from the beginning but not the point 16K off the
> > beginning.Is that right?
> >
> > 2. Reason for the problem:
> > It seems the offset is saved in td->o.start_offset. But it isn't passed
> > to f->file_offset.And the setting up of f->io_size doesn't refer to
> > f->file_offset.
> >
> > 3. Fix suggestion:
> > I tried this:
> > ---------------------------------------------------------------
> > diff -Nraup fio-7.25/filesetup.c fio-7.25-offset/filesetup.c
> > --- fio-7.25/filesetup.c 2007-07-25 14:25:05.000000000 +0800
> > +++ fio-7.25-offset/filesetup.c 2007-07-27 19:10:29.000000000 +0800
> > @@ -378,6 +378,8 @@ int setup_files(struct thread_data *td)
> > extend_size = total_size = 0;
> > need_extend = 0;
> > for_each_file(td, f, i) {
> > + f->file_offset = td->o.start_offset;
> > +
> > if (!td->o.file_size_low) {
> > /*
> > * no file size range given, file size is equal to
> > @@ -385,20 +387,27 @@ int setup_files(struct thread_data *td)
> > * zero, set it to the real file size.
> > */
> > f->io_size = td->o.size / td->o.nr_files;
> > - if (!f->io_size)
> > - f->io_size = f->real_file_size;
> > + if (!f->io_size) {
> > + if (f->file_offset > f->real_file_size)
> > + goto err_offset;
> > + f->io_size = f->real_file_size - f->file_offset;
> > + }
> > } else if (f->real_file_size < td->o.file_size_low ||
> > f->real_file_size > td->o.file_size_high) {
> > + if (f->file_offset > td->o.file_size_low)
> > + goto err_offset;
> > /*
> > * file size given. if it's fixed, use that. if it's a
> > * range, generate a random size in-between.
> > */
> > if (td->o.file_size_low == td->o.file_size_high)
> > - f->io_size = td->o.file_size_low;
> > + f->io_size = td->o.file_size_low - f->file_offset;
> > else
> > - f->io_size = get_rand_file_size(td);
> > - } else
> > - f->io_size = f->real_file_size;
> > + f->io_size = get_rand_file_size(td) - f->file_offset;
> > + } else if (f->file_offset > f->real_file_size)
> > + goto err_offset;
> > + else
> > + f->io_size = f->real_file_size - f->file_offset;
> >
> > if (f->io_size == -1ULL)
> > total_size = -1ULL;
> > @@ -406,12 +415,12 @@ int setup_files(struct thread_data *td)
> > total_size += f->io_size;
> >
> > if (f->filetype == FIO_TYPE_FILE &&
> > - f->io_size > f->real_file_size &&
> > + (f->io_size + f->file_offset) > f->real_file_size &&
> > !(td->io_ops->flags & FIO_DISKLESSIO)) {
> > need_extend++;
> > - extend_size += f->io_size;
> > + extend_size += (f->io_size + f->file_offset);
> > f->flags |= FIO_FILE_EXTEND;
> > - }
> > + }
> > }
> >
> > if (!td->o.size)
> > @@ -431,7 +440,7 @@ int setup_files(struct thread_data *td)
> >
> > assert(f->filetype == FIO_TYPE_FILE);
> > f->flags &= ~FIO_FILE_EXTEND;
> > - f->real_file_size = f->io_size;
> > + f->real_file_size = (f->io_size + f->file_offset);
> > err = extend_file(td, f);
> > if (err)
> > break;
> > @@ -447,6 +456,9 @@ int setup_files(struct thread_data *td)
> >
> > td->total_io_size = td->o.size * td->o.loops;
> > return 0;
> > +err_offset:
> > + log_err("%s: you need to specify valid offset=\n", td->o.name);
> > + return 1;
> > }
> >
> > int init_random_map(struct thread_data *td)
> > diff -Nraup fio-7.25/ioengines.c fio-7.25-offset/ioengines.c
> > --- fio-7.25/ioengines.c 2007-07-25 14:25:05.000000000 +0800
> > +++ fio-7.25-offset/ioengines.c 2007-07-27 17:14:05.000000000 +0800
> > @@ -279,7 +279,7 @@ int td_io_open_file(struct thread_data *
> >
> > f->last_free_lookup = 0;
> > f->last_completed_pos = 0;
> > - f->last_pos = 0;
> > + f->last_pos = f->file_offset;
> > f->flags |= FIO_FILE_OPEN;
> > f->flags &= ~FIO_FILE_CLOSING;
> >
> > diff -Nraup fio-7.25/io_u.c fio-7.25-offset/io_u.c
> > --- fio-7.25/io_u.c 2007-07-25 14:25:05.000000000 +0800
> > +++ fio-7.25-offset/io_u.c 2007-07-27 16:56:35.000000000 +0800
> > @@ -165,7 +165,7 @@ static int get_next_offset(struct thread
> > if (f->last_pos >= f->real_file_size)
> > return 1;
> >
> > - b = f->last_pos / td->o.min_bs[ddir];
> > + b = (f->last_pos - f->file_offset) / td->o.min_bs[ddir];
> > }
> >
> > io_u->offset = (b * td->o.min_bs[ddir]) + f->file_offset;
> >
> > ---------------------------------------------------------
> >
> > And now we can check it with gdb:
> > --------------------------------------------------------
> > ... ...
> > (gdb) bt
> > #0 0x0804cb4d in do_io (td=0xb7c50000) at fio.c:449
> > #1 0x0804e1b1 in thread_main (data=0xb7c50000) at fio.c:873
> > #2 0xb7f64504 in start_thread ()
> > from /lib/tls/i686/cmov/libpthread.so.0
> > #3 0xb7ec251e in clone () from /lib/tls/i686/cmov/libc.so.6
> > (gdb) x/4w io_u->xfer_buf
> > 0x8073f08: 0x206b636f 0x7420666f 0x66206568
> > 0x20656c69
> > (gdb)
> > ---------------------------------------------------------
> > After the first io, I got '0x206b636f 0x7420666f 0x66206568 0x20656c69'
> > read out.And they represent the string 'ock of the file '.Then i search
> > the string in HOWTO and got only one line containing the string:
> > 'norandommap Normally fio will cover every block of the file when doing'
> > I removed the text following that line in HOWTO, and then I found that
> > the size of the rest text in HOWTO was 16K, which equaled to the offset
> > we set in job file.
> > So I think the option 'offset' does work after the changes.
> >
> > In fact, I am not sure of the fix, but i do hope it can help.
>
> It's tricky. I think the fix looks correct, but I'd be a lot more
> comfortable if you could reassure me that it didn't break any other
> tests you have? I'll merge it, but please do run through your normal io
> tests and verify that they seem sane and work as expected. Thanks!
>
OK! I will do more tests! And if any problem of the fix found, I will
report to you as soon as possible :)
Received on Fri Jul 27 2007 - 13:33:18 CEST

This archive was generated by hypermail 2.2.0 : Fri Jul 27 2007 - 14:00:02 CEST