O_DIRECT for ARM

[Tobias]

Hi,

i want to use O_DIRECT on my custom ep9315 board. Actually i am usign kernel
2.6.15 with a special linux patch of cirrus. But when i try to use this on my
hdd (i open it with - open("/dev/hda",O_RDWR | O_DIRECT); -); the buffer is
not written/read correctly to the device although they are aligned to 512
bytes border. The read and write command does not fail but i can see that it
is not working correctly when i look with a diskeditor on my harddisk.

I have to use O_DIRECT to increase performance during disk writes or is there
any other possebility to avoid copying data from user space to kernel space
and then transfer it to the device ? I have tested rawctl also with the same
result.

Regards

Tobias

-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php

[Erik Mouw]

Post by Tobias
i want to use O_DIRECT on my custom ep9315 board. Actually i am usign kernel
2.6.15 with a special linux patch of cirrus. But when i try to use this on my
hdd (i open it with - open("/dev/hda",O_RDWR | O_DIRECT); -); the buffer is
not written/read correctly to the device although they are aligned to 512
bytes border. The read and write command does not fail but i can see that it
is not working correctly when i look with a diskeditor on my harddisk.

O_DIRECT has some prerequisites:

1/ userspace buffers have to be page aligned
2/ reads and writes have to be in multiples of the sector size (usually
512 bytes) of the underlying device in 2.6, and in multiples of the
soft block size in 2.4
3/ if on a filesystem, the filesystem needs to support O_DIRECT on
files

Post by Tobias
I have to use O_DIRECT to increase performance during disk writes or is there
any other possebility to avoid copying data from user space to kernel space
and then transfer it to the device ?

O_DIRECT usually *decreases* performance. You have to be absolutely
sure that you can be smarter than the kernel io scheduler.

Post by Tobias
I have tested rawctl also with the same result.

raw devices are obsolete, use O_DIRECT. See
Documentation/feature-removal-schedule.txt in your kernel tree.


Erik

[Tobias]

Post by Erik Mouw
1/ userspace buffers have to be page aligned

i open my device and allocate my buffers with this:

nIDE2 = open("/dev/hda"O_RDWR | O_DIRECT);
pWriteSector = memalign(getpagesize(),512);

Post by Erik Mouw
2/ reads and writes have to be in multiples of the sector size (usually
512 bytes) of the underlying device in 2.6, and in multiples of the
soft block size in 2.4

They are always multiples of 512.

Post by Erik Mouw
3/ if on a filesystem, the filesystem needs to support O_DIRECT on
files

No file system is used.

I am not sure but why is in /include/asm-arm/fcntl.h this behind the
definition of O_DIRECT:

#define O_DIRECT 0200000 /* direct disk access hint - currently ignored */

Regards

Tobias


-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php

[Erik Mouw]

Post by Tobias

Post by Erik Mouw
1/ userspace buffers have to be page aligned

nIDE2 = open("/dev/hda"O_RDWR | O_DIRECT);

, over here
Hope that's a typo...

Post by Tobias
pWriteSector = memalign(getpagesize(),512);

Post by Erik Mouw
2/ reads and writes have to be in multiples of the sector size (usually
512 bytes) of the underlying device in 2.6, and in multiples of the
soft block size in 2.4

They are always multiples of 512.

Oh, I forgot to tell that you also have to read/write from/to a
multiple of the sector size. So first seek() to the correct position.

Post by Tobias
I am not sure but why is in /include/asm-arm/fcntl.h this behind the
#define O_DIRECT 0200000 /* direct disk access hint - currently ignored */

Don't know if that's still true. It works just fine on x86, I use it
almost every day. If you want to be sure, test your program on x86
first.


Erik

[Tobias]

Post by Erik Mouw

Post by Tobias
nIDE2 = open("/dev/hda"O_RDWR | O_DIRECT);

, over here
Hope that's a typo...

Yes it was...

Post by Erik Mouw
Oh, I forgot to tell that you also have to read/write from/to a
multiple of the sector size. So first seek() to the correct position.

Of course.

Post by Erik Mouw
Don't know if that's still true. It works just fine on x86, I use it
almost every day. If you want to be sure, test your program on x86
first.

I have tested it with success, it works.


Russel: Where do i have to start debugging it ? Can you give me a hint ?

Nicolas: You wrote: and it probably has cache coherency issues to start with.

i thought O_DIRECT does not use the cache.


Regards

Tobias


-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php

[Jean-Philippe Francois]

Post by Tobias
Russel: Where do i have to start debugging it ? Can you give me a hint ?
Nicolas: You wrote: and it probably has cache coherency issues to start with.
i thought O_DIRECT does not use the cache.

Tobias, what driver are you using ? If O_DIRECT needs debugging, may be
it should be debugged on a reviewed ide and dma implementation.

Post by Tobias
Regards
Tobias
-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php

-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php

[Russell King - ARM Linux]

Post by Tobias
i want to use O_DIRECT on my custom ep9315 board. Actually i am usign kernel
2.6.15 with a special linux patch of cirrus. But when i try to use this on my
hdd (i open it with - open("/dev/hda",O_RDWR | O_DIRECT); -); the buffer is
not written/read correctly to the device although they are aligned to 512
bytes border. The read and write command does not fail but i can see that it
is not working correctly when i look with a diskeditor on my harddisk.

You're into new territory - you're probably the first to ever try
O_DIRECT on ARM. If you're seeing problems it probably needs
reviewing and debugging.

-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php

[Nicolas Pitre]

Post by Russell King - ARM Linux

Post by Tobias
i want to use O_DIRECT on my custom ep9315 board. Actually i am usign kernel
2.6.15 with a special linux patch of cirrus. But when i try to use this on my
hdd (i open it with - open("/dev/hda",O_RDWR | O_DIRECT); -); the buffer is
not written/read correctly to the device although they are aligned to 512
bytes border. The read and write command does not fail but i can see that it
is not working correctly when i look with a diskeditor on my harddisk.

You're into new territory - you're probably the first to ever try
O_DIRECT on ARM. If you're seeing problems it probably needs
reviewing and debugging.

... and it probably has cache coherency issues to start with.


Nicolas

-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php

[Tobias]

Post by Nicolas Pitre
... and it probably has cache coherency issues to start with.
Nicolas

With D-cache enabled it does not work.

I have tried my test program with disabled D-cache and it works.

But when i try to use "Force write through D-cache (CPU_DCACHE_WRITETHROUGH)"
it doesn't.

Tobias

-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php