Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

I have a scientific application which is gathering data in real time
and the way things work out, I need to write 513k files to the hard
drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
second). What is the most robust, reliable way to assure this kind of
speed?

We already have a small RAID array (2 HDs, RAID0), but it maxes out at
60MB/sec. That's just under what I need. But more to the point, it
can't consistently give me 60MB/sec. Should I just add more HDs to my
RAID array? If so, what kind should I get? What kind of performance
can I expect then? Thanks in advance. james

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

In article <1108487674.382990.49350@o13g2000cwo.googlegroups.com>,
J. Franchino <jfranchino0000@yahoo.com> wrote:
>I have a scientific application which is gathering data in real time
>and the way things work out, I need to write 513k files to the hard
>drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
>second). What is the most robust, reliable way to assure this kind of
>speed?
>
>We already have a small RAID array (2 HDs, RAID0), but it maxes out at
>60MB/sec. That's just under what I need. But more to the point, it
>can't consistently give me 60MB/sec. Should I just add more HDs to my
>RAID array? If so, what kind should I get? What kind of performance
>can I expect then? Thanks in advance. james
>


Do you need to do this continuously ? For how long ? How big is your
budget ?

A couple Gig of memory as a RAMdisk could buffer about an hour of data
that could be dumped to disk as time allows.

It may be that a couple digital videotape recorders may be your best
bet if you need more that that.




--

a d y k e s @ p a n i x . c o m

Don't blame me. I voted for Gore.

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

J. Franchino wrote:

> I have a scientific application which is gathering data in real time
> and the way things work out, I need to write 513k files to the hard
> drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
> second). What is the most robust, reliable way to assure this kind of
> speed?
>
> We already have a small RAID array (2 HDs, RAID0), but it maxes out at
> 60MB/sec. That's just under what I need. But more to the point, it
> can't consistently give me 60MB/sec. Should I just add more HDs to my
> RAID array? If so, what kind should I get? What kind of performance
> can I expect then? Thanks in advance. james

Do you for some reason need to have the data committed to the physical
surface of the disk at the end of the acquisition cycle? If so, and if I'm
reading what you posted correctly, then you need to rethink your strategy
because a commercial disk can buffer up to 4 seconds of your data in its
internal RAM, which would defeat such a purpose.

If that's not the case, then you should take a look at a buffering
strategy--store the data in RAM as it is captured and write it to the disk
in the interval before the next acquisition cycle starts. Any current
production disk should be fast enough to support this if I've read the
numbers you provided above correctly.

--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net)

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

"J. Franchino" <jfranchino0000@yahoo.com> wrote in message
news:1108487674.382990.49350@o13g2000cwo.googlegroups.com...
> I have a scientific application which is gathering data in real time
> and the way things work out, I need to write 513k files to the hard
> drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
> second). What is the most robust, reliable way to assure this kind of
> speed?

Ok there are two parts to this problem. The average data rate and the peak
data rate....

1) Average data rate

Say we aim for 10 cycles per second. That works out at 513 x 1024 x 10 =
5.2MBytes/sec which doesn't sound too hard.

2) Peak data rate

513 x 1024 /8^-3 = 65MBytes/S

So it's clear the problem is the peak data rate not the average data rate.
Why 8mS?

If you can't give the CPU more than 8mS to do this what you need is a buffer
to reduce the peak data rate. Something like a RAM disk.

Perhaps even something as simple as writing the data to a shared drive on
another PC over a fast LAN might work. Far from the most efficient route
though.

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

I need to do it in 8ms or less because that's all the clock time I
have. I'm grabbing at 30 fps, then doing some online data reduction to
have a final output of 513k. But since I'm doing a lot of online
computations, I only have about 8ms left in my cycle to have it written
to disk. The rest of the time, the CPU is busy.

I'd like to have the process be able to run indefinately at least up to
30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
seconds/minute = 6.9 GB. Far to big to fit in RAM.

I'm already running a multithreaded process, and I'll try some more
code tricks, but the simpler solution (though more expensive) is to
reduce the amount of time it takes to write to disk. Hence my
question. Thanks. I appreciate all of your responses so far.

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

If you are creating 7.5 files/sec you are doing almost 100 IO/sec on NTFS. You
can probably do that on SCSI 10K or a Raptor. FAT32 should be faster.

You can write 1/2MB in 8ms, but not create a file. You could pre-create all the
files ahead of time to eliminate file-create overhead.

"J. Franchino" <jfranchino0000@yahoo.com> wrote in message
news:1108509956.788735.225830@c13g2000cwb.googlegroups.com...
>
> I'd like to have the process be able to run indefinately at least up to
> 30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
> seconds/minute = 6.9 GB. Far to big to fit in RAM.
>

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

> I need to do it in 8ms or less because that's all the clock time I
> have. I'm grabbing at 30 fps, then doing some online data reduction to
> have a final output of 513k. But since I'm doing a lot of online
> computations, I only have about 8ms left in my cycle to have it written
> to disk. The rest of the time, the CPU is busy.
>
> I'd like to have the process be able to run indefinately at least up to
> 30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
> seconds/minute = 6.9 GB. Far to big to fit in RAM.
>
> I'm already running a multithreaded process, and I'll try some more
> code tricks, but the simpler solution (though more expensive) is to
> reduce the amount of time it takes to write to disk. Hence my
> question. Thanks. I appreciate all of your responses so far.
>
First find your bottleneck. I don't think it is a disk, more likely it is a
CPU.
Do you have AntiVirus software turned on?
Turn it off and try this test on a quiet system:
@echo off
for /L %%a IN (1,1,4000) DO copy source target%%a >nul

My two systems (AV off) produced the following results:

PIII-933 WD200BB (20GB)
4000 files 566KB each (26 files per second)
152.8s, ~270IOps, 16.0MB/s, CPU 36%

2xPIII-1.3G and 2xMaxtor 9 (80GB) - RAID0
5000 files 566KB each (43 files per second)
115.6s, ~800IOps, 24.5MB/s, CPU0 20%, CPU1 27%

With AV turned on, CPU was much higher, IOps less than a half.
So it depends what your CPU is doing at the time of file aceess...
You may try at least with the hyperthreading turned on and a decent CPU.

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

"J. Franchino" wrote:
>
> I need to do it in 8ms or less because that's all the clock time I
> have. I'm grabbing at 30 fps, then doing some online data reduction to
> have a final output of 513k. But since I'm doing a lot of online
> computations, I only have about 8ms left in my cycle to have it written
> to disk. The rest of the time, the CPU is busy.
>
> I'd like to have the process be able to run indefinately at least up to
> 30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
> seconds/minute = 6.9 GB. Far to big to fit in RAM.
>
> I'm already running a multithreaded process, and I'll try some more
> code tricks, but the simpler solution (though more expensive) is to
> reduce the amount of time it takes to write to disk. Hence my
> question. Thanks. I appreciate all of your responses so far.


Run it from a server machine with a RAM drive of 8GB?

I don't know offhand what the largest RAM drive you can have will be - I
used to use something called Ramdisk Pro and I had the option of
formatting the ram disk as NTFS or FAT32 - so I assume you will be able
to use 8GB formatted as NTFSs.


Odie
--

RetroData
Data Recovery Experts
www.retrodata.co.uk

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

Previously J. Franchino <jfranchino0000@yahoo.com> wrote:
> I have a scientific application which is gathering data in real time
> and the way things work out, I need to write 513k files to the hard
> drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
> second). What is the most robust, reliable way to assure this kind of
> speed?

> We already have a small RAID array (2 HDs, RAID0), but it maxes out at
> 60MB/sec. That's just under what I need. But more to the point, it
> can't consistently give me 60MB/sec. Should I just add more HDs to my
> RAID array? If so, what kind should I get? What kind of performance
> can I expect then? Thanks in advance. james

The only choice to get sustained, untinterrupted high bandwidth
access is RAM. Whether this is main memory or a solid state disk
depends on your specific needs and how much money you have.

I suspect memeoy is cheapest, solid state disk is most reliable.

Arno

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

J. Franchino wrote:

> I need to do it in 8ms or less because that's all the clock time I
> have. I'm grabbing at 30 fps, then doing some online data reduction to
> have a final output of 513k. But since I'm doing a lot of online
> computations, I only have about 8ms left in my cycle to have it written
> to disk. The rest of the time, the CPU is busy.
>
> I'd like to have the process be able to run indefinately at least up to
> 30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
> seconds/minute = 6.9 GB. Far to big to fit in RAM.
>
> I'm already running a multithreaded process, and I'll try some more
> code tricks, but the simpler solution (though more expensive) is to
> reduce the amount of time it takes to write to disk. Hence my
> question. Thanks. I appreciate all of your responses so far.

You may be hitting the real world limit of the PCI bus here. Typically in
network servers the PCI bus becomes a bottleneck somewhere between 250 and
400 Mb/sec--400 works out to 50 MB/sec with no overhead. That means that
no matter how fast your disks are if you're connecting the host adapter via
the PCI bus you're not likely to get above 50 MB/sec. In your application
in principle you might be able to but it's not the way to bet.

So to start out, I'd try staying with my existing disks and going to a
faster bus and host adapter. While you're about it it you might want to
consider going to an AMD-64 system--some of them can handle 32 gig or more
of RAM--with a PCI-X bus and a PCI-X host adapter.

Beyond that, you should be able to burst 513K to any disk on the market in
8ms if the bus will handle it--they all have 2 meg buffers or larger, some
up to 16, and interfaces that can handle 100+ MB/sec into that buffer. One
strategy, once you've got a fast enough bus in place, would be to alternate
your writes to several different disks--write to one, then the next cycle
write to the next, then the next cycle write to the next and so on in turn
until the first has had enough time to complete its write.



--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net)

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

"J. Franchino" <jfranchino0000@yahoo.com> wrote in message
news:1108509956.788735.225830@c13g2000cwb.googlegroups.com...
> I need to do it in 8ms or less because that's all the clock time I
> have. I'm grabbing at 30 fps, then doing some online data reduction to
> have a final output of 513k. But since I'm doing a lot of online
> computations, I only have about 8ms left in my cycle to have it written
> to disk. The rest of the time, the CPU is busy.

30fps = 33mS...

You need to set up some buffers. Could you set up a double buffer? Compute
on one frame while writing the results of the previous frame to disc. That
gives 33mS for writing the data bringing the bandwidth required down to
about 15MBytes/Sec.

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

"J. Clarke" <jclarke@nospam.invalid> wrote in message
news:cuvn7o01uh8@news1.newsguy.com...
> You may be hitting the real world limit of the PCI bus here. Typically
in
> network servers the PCI bus becomes a bottleneck somewhere between 250 and
> 400 Mb/sec--400 works out to 50 MB/sec with no overhead.

He only hits >50MB/sec because he's trying to write all the data in 8mS. A
simple double buffer would allow upto 33mS (30fps) and reduce the rate to
about 15MBytes/Sec.

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

CWatters wrote:

>
> "J. Clarke" <jclarke@nospam.invalid> wrote in message
> news:cuvn7o01uh8@news1.newsguy.com...
> > You may be hitting the real world limit of the PCI bus here. Typically
> in
>> network servers the PCI bus becomes a bottleneck somewhere between 250
>> and 400 Mb/sec--400 works out to 50 MB/sec with no overhead.
>
> He only hits >50MB/sec because he's trying to write all the data in 8mS.

If you read my earlier post you would be aware that I had already addressed
this point.

> A simple double buffer would allow upto 33mS (30fps) and reduce the rate
> to about 15MBytes/Sec.

If his circumstances are such that that is a feasible solution. He seems
adamant about his 8 ms write window.

--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net)

Archived from groups: comp.sys.ibm.pc.hardware.storage (More info?)

Previously J. Clarke <jclarke@nospam.invalid> wrote:
> CWatters wrote:

>>
>> "J. Clarke" <jclarke@nospam.invalid> wrote in message
>> news:cuvn7o01uh8@news1.newsguy.com...
>> > You may be hitting the real world limit of the PCI bus here. Typically
>> in
>>> network servers the PCI bus becomes a bottleneck somewhere between 250
>>> and 400 Mb/sec--400 works out to 50 MB/sec with no overhead.
>>
>> He only hits >50MB/sec because he's trying to write all the data in 8mS.

> If you read my earlier post you would be aware that I had already addressed
> this point.

>> A simple double buffer would allow upto 33mS (30fps) and reduce the rate
>> to about 15MBytes/Sec.

> If his circumstances are such that that is a feasible solution. He seems
> adamant about his 8 ms write window.

As I (and others) said, RAM and solid-state disks are the only solution
with a filesystem, since all other will have larger access times,
at least occasionally.

Maybe a single 15000RPM SCSI disk used in sequential mode (i.e. no
filesystem, sequential sector writes only) with no other devices
on the SCSI bus will work also.

In both cases 66MHz and or 64 bit PCI, alternatively PCI-E,
would likely need to be used as bus.

Arno