Can a RAID 5 NAS that you make yourself outperform ready-made alternatives? Bill Meade finds out.
 
Speak out in the Tom's Networking reader survey!

Nice review    
 
I have now since about a year something comparable:
- athlon xp 2200+
- MSI Athlon MP server board
- 1GB registered ddr ram
- 3Ware 7506 raid controller
- 4x WD hard drives 200GB
- Intel 1000GT network card
 
I use Fedora Core 4 on it combination with Samba, Apache and MySQL.
 
The average performance mentioned in the review is about the same in my case. Both reading and writing peaks at 40MB/s
 
I saw in the Samba configuration also a home directory share. There's something more usefull with Samba. It's possible to set all home directories shared automatically (no matter when a user is added/deleted/..).
 
You need the following configuration into the share section:
 
[homes]
 read only = No
 browseable = No
 
I'm not sure if this will work in share level security (I use user level). But anyway, with the right chmod settings per home directory, your able to limit every user to it's own home directory (in case you log on on the system itself and in case of accessing the shares through the network). The logged-on user in the network will also only see his/her share in the Network Neighbourhood (Windows).

Edward said:

John said:

I built my NAS with the 6/06 release of Solaris 10 utilizing ZFS.
Intel D820, Asus P5LD2 mobo, 1GB ram, onboard sata controllers, 250GB sata drives(4) in raidz, netgear 1Gbit nic.
 
With it, I get average speeds of ~38MB/s read, ~29MB/s write over nfs.
I haven't run iometer tests on it yet, these are just calcs with dd from a remote linux box.

A good article on building a cheap RAID 5 box, but I have to disagree with your selection of RAID card.  
 
If you build everything from the ground up and want to save money, then by all means, go with the hardware mentioned in the review.
 
Many folks, though, will not be starting from scratch. Most people who will take the time to read this article and tackle the project will most likely have a fair amount of equipment already on hand. If all you need to buy is the card and the drives (assuming you have everything else), then you definitely should go with a SATA version. I use, and recommend, the Areca PCI-E SATA II cards. Although they are a little pricey, they definitely give you a long term usage strategy. PCI-E and SATA are here to stay for quite a while, so buying one of these cards now will stave off obsolescence for the foreseeable future. You should be able to get a good Areca 8-port card and four 300GB drives for the same bottom line price as all the equipment listed in the article. This leaves you 4 free SATA ports to add storage to your array when you need it.

 
Edward,
 
I'll be able to tell you more in a couple days.  I'm re-running the tests on a 1.6 Ghz Duron right now.  I don't have comparable numbers (I'm adding Windows XP to the operating systems, but even XP is surprisingly fast without RAID).  My experience with Pentium II and Pentium III is that I am too pessimistic.  I've set up ClarkConnect home servers for friends expecting them to want to upgrade.  They never do!  
 
So, try it, you will probably not care.  The Linux big file problem is the bottleneck in these results, not the CPU I suspect.  
 
bill

 
krisia2006
 
The iozone benchmark is pretty easy to set up.  Here is an example command line.  When you install iozone it wants by default, to live in /Program Files/benchmarks/Iozone 3.326.  So you run a dos window, switch to that directory, and then you feed iozone a command like this:
 
iozone -Rab 014XPD16.wks -i 0 -i 1 -+u -f z:\test\013.tmp -q 64k -n 32M -g 1G -z
 
The above is a command I just ran testing Windows XP on a Duron 16.  I like to code what the hardware is into the output .wks file name.  Iozone runs the tests and lists the data on screen and then dumps everything into the .wks file.  So, it is really pretty easy to replicate what Tim Higgins has set up.  If I can do it, someone who can touch solaris and live can do it easily!  :-)
 
Tim's how-we-test article is available at http://www.tomsnetworking.com/2004/03/02/how_we_test/
 
If you want any help I'd be happy to step you through it.  
 
bill

Very good points!
 
It's essential to understand that the system reviewed
has a bottleneck in the old PCI bus i.e. 32 bits @ 33 MHz
<=  133 MB/s aka ATA-133 MAXIMUM (that's where the
"133" in ATA-133 comes from
 
The Areca PCI-E SATA RAID controller is definitely superior,
mainly because it harnesses the native speed of the PCI-Express
bus, up to x8 lanes, instead of the old PCI bus.
 
And, certain Areca versions can be upgraded with as much
as 1 GB of on-board cache, for real I/O acceleration
(might be a special order from Areca, or so our systems
consultant tells us).
 
The Promise model EX8350 is very similar in overall
design and performance, and a bit less expensive.
This is an x4 lane device.
 
Both the Areca and Promise cards now support RAID 6.
 
FYI:
We just installed 4 x WD2500YD SATA-II HDDs to the
4 main SATA-II ports on an Intel ICH7R I/O controller
hub.  They are wired via Addonics Multilane bridge
and cable connectors (4 SATA sets in one cable)
to a recycled AT-style case with Antec 300W AT-style
power supply.  This allows us to "spin-up" this RAID 0
array PRIOR TO booting the host PCI-E motherboard,
and it splits the power load between 2 x PSUs.
 
We have measured raw READs up to 160 MB/s
with this RAID 0 configuration -- definitely greater
than the MAXIMUM possible from the old PCI bus.
 
The ONLY saving grace of the PCI subsystem reviewed
is that the PCI bus oscillates at nearly the same speed
at a 1 GbE LAN.  Thus, upgrading to a PCI-Express  
subsystem will simply shift the bottleneck to the
Gigabit LAN switch.  (32 bits @ 33 MHz  =  1,056 Mb/s).
 
10 Gigabit Ethernet anyone?  
 
 
Sincerely yours,
/s/ Paul Andrew Mitchell
Webmaster, Supreme Law Library
http://www.supremelaw.org/

 
This is really great information thanks for posting it.  It is one thing to suspect that some component is better than another.  But it is another to have expeience with it.  I had a conversation with the ClarkConnect president and his chief hardware guru about ethernet adapters and their preference was 3com.  Of course, the 3com adapters are PCI-X (64 bit) and all the motherboards I've been playing with are 32bit.  Then intel you mentioned is 32bit so it is definitely worth a try.  Have you had any bad luck with open box products from NewEgg?  I've gotten close to buying them in the past but NewEgg's warnings are so draconian I always chicken out!  
 
Thanks!
 
bill

 
All I can say is Doh!  
 
One of the bad things about having a RAID habit, is that you don't often run into people you can talk about RAID with.  Then, when you do run across someone, as you ran across my article, they are interested in some different aspect of RAID than you are.  
 
My bad.  In my article, the whole idea was how much performance can I get with hardware that is the minimum configuration to get the job done.  "How little can I get away with?" is my big question.   I didn't consider your RAID card because I have been living 100% in the used RAID card market!  
 
Your thing is clearly performance.  I can dig that too.  Just not while working for Tim Higgins :-).  Reading about RAID the disk reads should be faster.  The writes should be slower.  The relationships between all the behaviors are described theoretically.  We need to test these because a lot of the time, the theories are wrong.  
 
Do your SATA drives have command queueing?  How does command queueing change RAID behavior?  It would not surprise me if command queueing tripps up the striping that the RAID adapters are trying to do.   And, what about defraggin RAID arrays?  Is that a good thing, neutral, or a bad thing?  
 
Thanks for the recommendation!  One thought I've had is to take a great RAID card, like yours, and re-do the analysis with another slow CPU.  I was hoping to use the 4 WD ATA drives, but I may have to spring for new SATA drives as well.  No .... maybe I can us one of those ATA to SATA adapter kits!    
 
Just dinging you!   Thanks again!!!
 
bill meade

A small point i wanted to make - living in the UK, or computer equipment is a bit more expensive than the US.
 
I tried to configure a BYO system with similar kit - slightly higher end, e.g. a case with decent cooling, a good psu, but definitely sticking to budget stuff (about the 2nd cheapest model I could find).  
 
I found the cost close to ready made solutions like the Intel SS4000-E.  
Given that it's ready made, with more advanced features and needs little configuration, I'm inclined to think that it only makes sense to build your own if you (a) have most of the kit lying around (perfectly sensible) and (b) have the time to do it.  
 
 
I do NOT mean to suggest in any way that the article wasn't excellent, by the way. I hope there will be many more like it!
 
 
One weakness of the ready made solutions though is that they all seem to have annoying bugs, e.g. Buffalo Terrastation doesn't support very long filenames, the Intel SS4000-E, has a lot of bugs, doesn't support NTFS, and while its the fastest of the existing equipment its still way slower than it should be.
 
One area that it would be interesting to see, for home/SoHo users is to build a RAID inside a workstation, using on of the new motherboards that support up to 10 SATA devices. You could easily allocate 4 of them to your RAID setup for ackups etc, keeping the rest for everyday use. Apparrently the performance of the built in RAID controllers is quite good. And of course you could still use the Gigabit interface to connect to other clients.
 
Any thoughts?

 
That was an ATA100 controller...    besides find a single drive that can sustain 133MBs, let alone burst a read at the speed.  
 
Explained:
 
Congestion on the host bus to which the ATA controller is attached may also limit the maximum burst transfer rate. For example, the maximum data transfer rate for conventional PCI bus is 133 MB/s, and this is shared among all active devices on the bus.
 
In addition, as of October 2005 no ATA (this includes SATA) hard drives exist capable of measured sustained transfer rates of above 80 MB/s. Furthermore, sustained transfer rate tests do not give realistic throughput expectations for most workloads: They use I/O loads specifically designed to encounter almost no delays from seek time or rotational latency. Hard drive performance under most workloads is limited first and second by those two factors; the transfer rate on the bus is a distant third in importance. Therefore, transfer speed limits above 66 MB/s only really affect performance when the hard drive can satisfy all I/O requests by reading from its internal cache.
 
Most would soil themselves if they had an array that could sustain that speed in a DIY “cheap” box.
 

 
This is about "cheap" NAS.  
 

 
$400-$1000 cards aren't what I would call cheap. If they are pick me up a few at the corner mart and a 20oz drink as well. While you’re at it I’ll take an 2105 ESS made by IBM as well.
 
I would say four drives stays within the bounds of cheap. An $80 card still stays in the cheap zone. I’m setting $500 and below as the standard and quintessential precedence for “cheap.”
 
Bill maybe you could run those numbers on raids, 0, 1, 0+1, 10, and 100 for those who like me don't like Raid 5, but I do agree 5 is really easy and safe.

My thoughts:
 
First, "Amen" to getting the best stuff for the least.  For every job there is a perfect tool.  If you can get a cheap perfect tool.  Buy it!   I don't mean to advocate that people build PCs from scratch for this project.  I did not do that.  I took an overly expensive power supply I ALREADY HAD, a cheap mobo I Already Had, memory, you get the idea.  I even had the RAID controller because I had bought 5 of them at one time to get the price down.  Possession takes the *sting* out of acquisition!    
 
Second, the ugly tradeoffs of RAID are not a product of commercial vs. home-built systems.  They are a function of RAID's immaturity in the personal computer ecology.  File naming is a problem.  I'm a Mac user and this is always tripping up clean backups.  What to do .. step-by-step to recover from a bad hard drive is a problem.  Which RAID to use is a problem.  RAID is the "media PC entering-the-we-are-not-proud-zone" of the storage world.  All potential, weld splatter, and rough edge.    THE BIGGEST UGLY THING is the large file slow down of Linux-based RAID systems.  When you get a file larger than 600 megabytes, Linux grinds to a halt.  2.4 kernel, 2.6 kernel, makes no difference.  LINUX SUCKS ON LARGE FILES.  I've never heard anyone say "Oh, that does not matter, we don't need to fix it." but it sure seems like key people in Linux land are thinking that to themselves.  
 
Third "Amen" to a RAID rack whether it is inside or outside a workstation.  I love the RocketRaid external enclosure, the X4 which is $450 http://www.newegg.com/Product/Prod [...] 6817124012 and connects to the internal RocketRaid card via Infiniband.  
 
"Infiniband" what a cool name, makes me want to flush money down the toilet NOT!.  But it is a cool name.  This product if it housed 10 drives would be VERY COOL because I don't know how many 160 gig hard drives you have, but I have like 10 of them.  They'd make a nice 1.6 terabyte array given the right hardware.  
 
My other thought on this is that I know way too many people who have discovered that firewire is unreliable.  Wiebetech is living off firewire 800 direct attached storage systems that are way fast.  But I'm skeptical of putting my life on firewire.  It would be nice to have some other more robust standard like a dumbed down fibrechannel.  Doh, probably what Infiniband is.  
 
Look at the sustained transfer rates they are getting from RAID0 sata drives.  Link: http://www.wiebetech.com/products/G5Jamplus.php  I've been asking Jeff Hedlesky (Wiebe Sales Guru) for (a couple) weeks to loan me one of these setups so I can see if it is real or if it is Kool-aid.  Yo Jeff, got your ears on????  
 
Dimitrik you raise a number of great points.  What I think they all point to is that we are just beginning to figure out what RAID can do for us in the way of reliability and performance.  And ... what RAID can do for us in the area of re-using or repurposing computer resources like drives and NASes.  
 
Thank you for posting.  If I missed "ask again" and I'll try again.  
 
bill

I'm re-running the numbers as I write this, on non-RAID NAS configurations.  I can't seem to trick FreeNAS into working on hardware without the LSI Logic i4 card in it (did you notice we snuck FreeNAS into the graphs?).  But I'm testing Win XP, ClarkConnect Home 3.2, Ubuntu Desktop 6.06 on dog-slow (Duron 1.6 ghz) hardware.  I'm surprised by how the good the results are.  
 
Makes me want to find a more current RAID card and then try RAID 0, 1.  I ran RAID 10 for a year and couldn't stand the wasted drive space.  What is 0+1?  I haven't heard of 100.  Can you please reply and educate?  
 
Thanks!  
 
bill meade

I did this same thing 2 and a half years ago.  My approach was to compare software raid with hardware raid using SATA technology.  I picked up a sata version LSI controller and 5 hitachi 250GB disks, etc. There are some nice hot-swap cages that improve airflow over the drives from enlight and others that simplify this too, for 2, 3, 4, and 5 drive bays. Using the hardware based raid I was actually depressed by the performance.  Somewhat lower than your numbers, around ~5000kB/s sustained, but I was learning.  Moving to software raid on linux I was able to sustain 7-12kB/s per drive.  IOStat is a great tool that isn't normally installed in most distributions.
 
You are wise to establish how to repair the raid when it breaks before starting.  Assuming you are using MD for your software raid, its actually quite simple to repair using mdadm commands.  Insert the new drive, add it to the raid group and let it rebuild.  Its a good idea to remove the faulty drive from the configuration first.  I avoid lvm on linux as it incurs roughly a 33-50% preformance hit.  It has nice features but the cpu spends too much time at idle or IO wait states.
 
to fix md raid 5 is something like this (assuming you've already swapped disks)
mdadm --manage /dev/md0 --remove /dev/sda  (something like that)
mdadm --manage /dev/md0 --add /dev/sda      
(*I don't know if thats accurate syntax - thats why there are manpages)
Watch /proc/mdstat for progress (cat /proc/mdstat)
 
I've recently upgraded my nas to the following - (don't cry)
AMD64 x2 3800, 1GB DDR2(800), 1GB nic, linuxfromscratch.org OS.  Still running MD for raid5 - I've only done rough performance numbers of 90MB/s across the GiGe port sustained for 4 minutes.  
 
I feel there are two benefits to software raid - memory or cache is important - linux will use available memory as cache for the filesystem and that can increase perceived performance.  The inexpensive hardware raid adapters don't have enough IMO.  Secondly, accessing the disks.  All raid5 configurations write the data in a proprietary format - accessing this data after a crash can be difficult, especially with hardware raid, but software raid too.  I feel more comfortable with a linux/software raid knowing that I can recover my data manually.  However I have a few advantages - I built it, I've worked with linux on and off for 13 years, and I work in the storage industry, mostly with fibre channel.
 
Some tips I picked up---
 
Someone asked what impact the processor will have, really it all depends.  I started on a Via 500mhz itx board with 512MB memory.  The cpu's main task is to calculate the parity in a raid5.  Hardware raid should handle this so the cpu becomes less important.  The more memory you can add to the raid solution helps also.  Most enterprise storage arrays have gigabytes of cache memory.  A 64bit OS can increase performance also, so the newer AMD64 and intel 64 chips will help, but you don't need the fastest one out there.
 
Power - Disk drives consume roughly 2 amps of power to boot or spin up and can use the full 2a under heavy conditions.  Verify your powersupply can handle the total number of amps on the +12v line.  So 3 disks at 2a each would be 6a total.  Don't forget the internal one plus the CD/DVD?  Each vendor lists their drive's consumption on the web.
 
Samba - You could remove the force user and force group and go with only guest only = yes or allow guest = yes.  Remember to chmod 777 your shares also! )  User level security requires authentication, share level is dependant on the share itself, there is also Wins, domain, and ads options, good for keeping the kids or parents out of your stuff!
 
Great article, nice to see others doing the same geeky stuff I'm doing.