Terabyte array on the Cheap

Scratching at my brain

Fast forward 1.5 years.

After hopping around several deal sites and seeing the extraordinarily cheap prices of hard drives, I was once again inspired to revisit the terabyte array. I did a more detailed analysis of the pricing and here's what I came up with (I had to do some upgrades because I couldn't find some parts exactly). And yes I've ignored a video card, but the original spec did too.

Wow, what a difference. I could do that for sure. But then I thought to myself, "Self, you know technology has gotten better why not a few upgrades to make this thing really scream, but be reasonable about price ". I mean we have Serial ATA drives now for a negligible price difference.

"ChaChing!" That's a pretty good system. All under $2k. The logistical problem with building the terabyte array is really the hard drives. It's extremely price sensitive. Normally when you go out and buy a drive, you think, "oh 10 bucks difference no biggie", obviously it adds up when you buy 8 of them. After giving a really good hard look at this, I was a little wishy washy on if I wanted a terabyte drive so bad that I would shell out this amount of money. Also, I wanted to have money left over so I could (like every good geek) do the lights and special whiz bangs. So I went and spec'ed out a "budget" version. (There were a few problems but I'll get to that)

This was an *extremely* reasonable figure for what I wanted. Now I had to overcome some technical hurdles. I understood putting the parts together, but I had never implemented a raid array before. I found out quickly that you need dedicated channels for the drives to make effective use of the array. So, the Fastrack s150 cards were out because there was only 1 Ultra133 channel on the card (but 2 serial). Ok back to the original Promise 100TX2. Also, The case and power supply on the budget version provided very little venting and not enough power (350W). Grr. ok that bumps things back up a bit. Now I had to consider how the rubber meets the road. How do you get a bunch of raid 0 cards all on different channels to be what was described in the article as a raid 5 array implementation. These were the convoluted theories I had about what I had to do:

1) Ok, I raid 0 each set of 2 drives, which gives me 4 logical drives for the OS. Well, I bet he meant for me to raid 5 the 4 drives so I have redundancy from there. But that stinks because if one of the drives die on one of the controllers I'll need to reformat the other one on the controller put in a new drive and let the raid 5 sync it all back up again.

2) Wait that doesn't sound right. I mean raid5 gives me striping already. Why aren't I using just plain old ultra ata133 controller cards and raid5 via software?

Alas I was stuck. I needed help. So I posted back at the kcgeek forums looking for answers. Everyone was extremely helpful. Heck Peloquin responded with what he intended with the article. All really good stuff. Of course I learned a few things.
"I never actually built that box, it was purely theory"
"Promise raid controllers were never designed to be used five or six at a time on one PCI motherboard, so warranty stuff here is basically out the window." -- Peloquin

Doh, and double Doh.

What does this mean? What options did I have? I mean I already started my commitment by buying a few of the parts already:

1) Pay for (e.g. hop on eBay and wait) a 8 channel raid card from 3ware or someone else

2) Pay for 2 4-channel raid cards that support raid 5 and then have essentially 2 volumes to work with that total up to 1 terabyte.

3) See what kind of controllers are showing up with these hard drives I've mail ordered, use those. Then go with the raid 5 and disable any raid on the controllers themselves (if any)

Well, after some voting in the forum, general consensus said go with a better raid controller card. This upped the price of the controller (8 port 3ware 7500-8) to about 260 (via eBay). I needed a way to bring the price back down a little bit more. Remember when I mentioned the worst part of the terabyte array was the drives. Ok well I decided to do it the hard way, I set myself to go online to various deal sites and take advantage of the "rebate" price drives, the really bad part is I could really only take advantage of a rebate from a given store 1 time from 1 drive. This would really slow down my progress but would bring the average drive price down to $82-$85 (and have more reliable drives to boot). This also meant that I had to take what I could get out of the deal. It had come down to 2 brands of drives that were on rebate: Maxtor 160 ATA133 8mb cache and Western Digital 160 ATA100 8mb cache.

I was concerned about the ATA133/ATA100 thing and asked about it in the forums but everyone seemed ok with it. Then I remembered option 2; buy 2 3ware 7500-4 channel drives and have 2 volumes that are raid 5. <lightbulb> hey, that means if I can load 1 card with the ATA133's and the other with ATA100's I can satisfy the "ideally you should have the same drives in the array" and the "worry about ata100/133 problem".

The addition of the true raid5 cards also freed up money from buying a dual proc system with a lot of memory. The original article had the intention of doing software raid, which required lots of processor as well as memory. So here are the new Specs (I splurged on memory and proc because it was neglible price difference)

So as you can see the core of the system (the drives) cost me 678. With an average cost of drives being $84.75. This resulted in the rock bottom cost of the system.

** side note: I spent extra money to go ahead and buy some fancy case modding stuff like fans, uv lights, and cables etc. I'm not including in the master total. But I think I spent like 80 dollars total on the modding stuff (so when you see the finished pictures don't be surprised by all the lights)

** side note #2: I did notice that the upgraded fans I bought seemed to move air significantly better than the fans that came with the case. It might be worthwhile to plunk down 20 bucks and buy a couple of the better fans (as heat will undoubtedly be a problem for this machine)

Now to the installing.....

Hardware:

Pretty standard all told nothing really special to the 3ware cards, there was no need to jumper them or anything. Just plug in all your cables and away you go.

Notables that I ran into:

1) As I said above in "side note #2" heat is a problem. The upgraded cables were good to help with air movement. I moved one of the stock fans to the front of one of the "drive cubes" (cage that holds 3 drives) to help with more ventilation. I'm hoping all the additional fan/lightage will still keep my power wattage in line with the power supply.

2) Cabling is an issue, get some zip ties and neaten that stuff up. At minimum it allows for better ventilation and since I bought that nice acrylic windowed case it'll look nice.

Software:
After entering the bios as I said, I elected to make 2 raid 5 drives. For those of you that don't know much about how much space various raid conditions cost you:
Raid0 = Number of Drives * the capacity of the smallest drive
Raid1 = the capacity of the smallest drive
Raid5 = (Number of Drives - 1) * the capacity of the smallest drive
Raid10 = (number of drives/2) * the capacity of the smallest drive

I know what you're thinking. Well (4-1)*160 only gives me 480gb * 2 = 960. Well Ok but I still have a 60 gb hdd hanging off the mainboard. So it adds up to 1.02 terabytes.

I'm still experimenting with linking the two controllers to yield (8-1)*160 = 1.12 Terabytes (plus 60gb = 1.18 terabytes)

I like the idea of the 1.02 terabytes option in that the Maxtor drives will all be on 1 controller (with ata133) and the Western Digital drives will be on the other controller (with ata100). I'm certain it wouldn't make much difference, but the homogenous drive setup puts a bit more peace of mind for me (even if I lose 160gb)

I set up the controller just as the manual dictated and there was no deviations.

partition/format done (which took about 3.5 hours, it's a terabyte whadya expect?). Voila, my very own Terabyte array. (see pictures at the bottom)

oh and I named it: yoda

Future improvments/Conclusions:

All in all it was really easy to do (ok the drives were a pain to get). And for the price it's a no-brainer. Granted this isn't some sort of production system for a business, but it's fault tolerant and fine for the average user/hobbiest. I suppose if I really want to get into it, I could use the antec case Peloquin suggested (which would allow hot swapping drives, backup power etc).

A big thanks to everyone on the kcgeek forums for input and advice. And of course thanks to Peloquin for the original inspiration. Hopefully my realworld implementation of "the Array" lives up to everyone's expectations. And, I hope you enjoyed/found useful this little article.

Pictures:

The full box	Better shot with lights on
The drives	The board and cards lit like a xmas tree