New HBA Storage Controller > I need some advice: Card / Cables

[danfer]

I have reached the limit of installable drives: I am pretty sure you know this situation 

My motherboard can handle 6 drives and I installed one SSD and five WD Reds. Due to the fact that my data appetite is still vast, I have to buy me my first storage controller 

I already read a lot of threads and reviews and decided to invest my money into a controller with a LSI chipset. By the way, I just need a "simple" storage controller, no RAID or other fancy functions are needed. I am fine in investing some extra money in the newest chipset generation, because I want to use the controller for a lot of years 

So far so good, I am completely new to SAS, HBA, JBOD etc., therefore I need an expert advice. I already figured out two potential storage controllers:

[1] Intel RAID Controller RS3WC080 | Specs > http://ark.intel.com/products/77345/ [sorry cannot link them]

[2] Intel RAID Controller RS3UC080 | Specs > http://ark.intel.com/products/76066/ [sorry cannot link them]

 

As far as I understood I have to look for a storage controller with JBOD mode, because I do not need the RAID functions and simply want to attach new drives.

Am I right that [2] is the right decision for my setup? Are there any issues/problems I might face in installing the storage controller? Do I you have to take something into account when I add the new drive into my (lovely) pool?

 

Thanks so much for your help and patience!

[Christopher (Drashna)]

Danfer,  

 

I definitely know the "no more space for drives" issue. And it's not a fun one.  Well, it can be fun, if you're into that sort of stuff.

 

As for the controllers, LSI is a fantastic company, and their chipsets are some of the best.  Intel are very nice, as well.

 

As for the terminology here, let me help clarify them:

HBA:  "Host Bus Adapter". This is basically what all consumer grade controller cards are.  This basically means that the controller card doesn't do RAID. It just passes the disks through to the OS.  This is generally what you want (there are exceptions, though).

SAS: "Serial Attached SCSI".  It's basically an industrial grade "SATA" protocol. It's ... a lot better, and a lot more stable.  However, "SAS" connectors are usually compatible with SATA drives. It depends on the hardware in question.  But you'll need "breakout cables" in your case. Breakout cables allow you to using the SAS connector to connect normal SAS or SATA drives.

JBOD: Just a Bunch Of Disks:  This isn't necessarily what you want. In some controllers, it creates an array with all the disks listed, rather than passing the disks through.  

 

 

These controllers look like the may work for what you want, but I'm not sure.

 

However, I actually have a couple of IBM ServeRAID M1015 cards. These are SAS cards that use the LSI chipset. They've been flashed to allow normal RAID functionality AND to pass through disks to the OS. They work great with StableBit Scanner and StableBit DrivePool.  If you're interested in them, PM or email me (christopher@covecube.com), and I'll see what I can do.

And these cards are a lot cheaper than what you'd pay for the Intel cards you've listed, or a new LSI card. (as in ~$100, instead of $300+).

 

 

As for the breakout cables:

http://www.amazon.com/Logic-LSI00411-SFF8643-miniSAS-Cable/dp/B00H5NPLHW/ref=pd_sim_sbs_147_1?ie=UTF8&refRID=1ZXN740AT0YT0JJJQJ58

or 

http://www.amazon.com/CBL-SFF8087OCF-06M-Mini-SAS-SFF-8087-Multi-lane-Internal/dp/B000HVXCXO/ref=pd_sim_sbs_147_3?ie=UTF8&refRID=1R86XTV7V56WXH0XAEBJ

Depending on what controller card you end up with.

 

 

Also, depending the card, you may need to use the "Unsafe" DirectIo option to get SMART data (this is rather normal on the higher end cards)

[p3x-749]

+1 for the HBAs based on M1015...these are great.

And you can grow later by adding an expander, instead more HBAs.

 

But...what drive sizes do you employ at the moment?

With drive capacity rising fast the money per TB ratio is falling.

There is as well the solution to just replace drives with larger ones and even sell the used ones afterwards.

Saving on the controller and the energy, too.

[danfer]

Hi Christopher,

 

Thanks for your feedback and the explanation 

The mentioned Intel controllers are RAID controllers, as you described they are not what I am looking for. I need a "simple" HBA.

I already read a lot about the LSI 2008 chipset. There are a lot of OEMs out there ~> link | Here is also a list of the other OEMs ~> link

But I am worried that the chipset is too "old". I want to stay with the card for at least 5 years, therefore I was looking for cards with the newer LSI 3008 chipset.

For the moment the LSI 9300-8i (link) is on top of my wishlist. It can handle 8 disks and has the new chipset generation.

By the way the HP220 (link) card is a good alternative to the M1015. It has a LSI 2308 chipset (according to the above link) and you can get it for ~150€.

 

@p3x-749:

I am currently at 15TB with 5 WD Reds (3TB version).

The cost per TB is almost the same: A 3 TB WD Red costs ~110€ (37€/TB) a 4 TB WD Red costs 160€ (40€/TB).

I would get ~90€ per drive if I would sell them. This gap is too big for me and I am pretty sure in six months I am facing the controller problem again 

 

But perhaps I am too anxious, due you have "speed problems" with your LSI 2008 cards?

[otispresley]

Hi, I used to use two LSI 2008 cards and they were great, one was RAID and one was not.  There are no speed issues with them as they are faster than your network can currently handle.  The maximum throughput of a Gigabit Ethernet link is 125MB/s.  Due to overhead, I usually got about 113MB/s across the network downloading from a mechanical HDD connected to the LSI 2008.  Drive to Drive on the same controller, it was as high as 170+MB/s.  I built a new server and am now using a HighPoint RocketRaid 2760A, only because it can take 24 drives; otherwise I would still be using the LSI 2008.  The speeds on the HighPoint card are much the same as they were on the LSI 2008.

[danfer]

Hi otispresley,

 

Thanks for your feedback!

Ok, that sounds good. What drives do you use?

 

Does anybody know the advantage/disadvantage of SFF-8643 vs. SFF-8087?

[p3x-749]

But perhaps I am too anxious, due you have "speed problems" with your LSI 2008 cards?

 

...the LSI208 are available since a long time.....they usually offer 6GB/s SAS/S-ATA links....you will never max-out these with a spinning disk,

I am using a single M1015 (flashed to HBA) along with an intel res2cv240 (LSI2008 based) 24port expander in a norco4224-like case..not with DrivePool but as my ZFS based NAS.

Doing a scrub, which means checking all data and checksums in the array, I still get 600+MB/sec accross (the array is fully encrypted btw, employing a mix of seagate and hitachi 3TB/7.2k disks).

As long as your server is limited to 1 or up to 4x single GBit NICs, I would not worry about the speed of this "old" LSI 2008 cards....a single one, driving even 24 disks will max that out.

The PCIe 2.0x8 cards might impose an electrical problem, but only if the next gen PCIe type slots are not downward compatible....but there are PCIe3.0 versions out there.

[Christopher (Drashna)]

Hi Christopher,

 

Thanks for your feedback and the explanation 

The mentioned Intel controllers are RAID controllers, as you described they are not what I am looking for. I need a "simple" HBA.

I already read a lot about the LSI 2008 chipset. There are a lot of OEMs out there ~> link | Here is also a list of the other OEMs ~> link

But I am worried that the chipset is too "old". I want to stay with the card for at least 5 years, therefore I was looking for cards with the newer LSI 3008 chipset.

For the moment the LSI 9300-8i (link) is on top of my wishlist. It can handle 8 disks and has the new chipset generation.

By the way the HP220 (link) card is a good alternative to the M1015. It has a LSI 2308 chipset (according to the above link) and you can get it for ~150€.

 

@p3x-749:

I am currently at 15TB with 5 WD Reds (3TB version).

The cost per TB is almost the same: A 3 TB WD Red costs ~110€ (37€/TB) a 4 TB WD Red costs 160€ (40€/TB).

I would get ~90€ per drive if I would sell them. This gap is too big for me and I am pretty sure in six months I am facing the controller problem again 

 

But perhaps I am too anxious, due you have "speed problems" with your LSI 2008 cards?

Yes, the cards I've linked are RAID cards.  However, they also support passing through the drives, as well.  

 

The M1015 has two firmware options for it: IR (RAID) and IT (HBA).  However, I've found that the IR firmware actually works better.  .... and lists the drives in a rational order (mostly).

 

As for the LSI 2008 chipset, it supports 6Gbps speeds, That means SATA III.   But to make a point here, the drives are never going to hit that speed, which would be 768MB/s.  An SSD may.  But your spinning hard drives are definitely not.

 

Couple that with the PCI Express 2.0 8x interface, you would need to run 35 drives at 120MB/s to start bottlenecking the connection. 

 

 

So, I recommend these cards because they may be a bit older, but they're absolutely, completely relevant still.  

 

 

And if you're interested in one of the cards, I could see about flashing it to IT (HBA) mode. But as I said, it's not necessary as the IR mode works as both RAID and HBA. 

 

 

 I built a new server and am now using a HighPoint RocketRaid 2760A, only because it can take 24 drives; otherwise I would still be using the LSI 2008.  The speeds on the HighPoint card are much the same as they were on the LSI 2008.

Since ... well you were agreeing with most of what I was saying, I've edited the quote.

 

As for the card... an SAS Expander would have solved the issue as well. I've linked an Intel one. It gets 20 additional drives, by expanding the one SAS connection to 5 additional ones.  And you could connect two of these expanders and get a total of 40 drives.  If that's too many, they make internal to external adapters for these SAS connections.

 

 

 

Doing a scrub, which means checking all data and checksums in the array, I still get 600+MB/sec accross (the array is fully encrypted btw, employing a mix of seagate and hitachi 3TB/7.2k disks).

 

The PCIe 2.0x8 cards might impose an electrical problem, but only if the next gen PCIe type slots are not downward compatible....but there are PCIe3.0 versions out there.

 

https://en.wikipedia.org/wiki/PCI_Express

A PCI Express 2.0 8x (8 lane) card has a max throughput of 4GB/s (that's gigaBYTE, not gigaBIT). So seeing 600MB/s or higher is still not even close to saturating the bus, in theory. 

 

 

As for backwards compatibility, that's ... more or less mandatory for the specification, IIRC. However, the Intel Expander cards have a Molex power option, so ... it shouldn't really matter.

And Wiki indicates that PCI Express 4.0 is maintaining backwards compatibility, so should be a moot point.

[danfer]

Thanks for your feedback and the evaluation.

I will follow some ebay auctions and buy my lovely HBA  

[Christopher (Drashna)]

Well, let us know what you end up going with, and how well it works out for you!

[danfer]

yes of course 

[danfer]

I still have two questions 

 

Just to be on the safe side about the M1015:

[1] It does not matter if the IT or IR firmware is installed, in both modes I can use the controller as HBA. Correct?

 

[2] Do you know if "all" LSI controllers handle the HBA mode this way?

[Christopher (Drashna)]

I still have two questions 

 

Just to be on the safe side about the M1015:

[1] It does not matter if the IT or IR firmware is installed, in both modes I can use the controller as HBA. Correct?

 

[2] Do you know if "all" LSI controllers handle the HBA mode this way?

1. Correct.  I'm using them in IR mode, but passing through the disks (as per HBA). 
2. It depends on the firmware.  I believe the actual LSI branded cards do work this way. But I'm not 100% sure about that.

[danfer]

@Christopher: Thanks

 

And again I am one step closer to my controller   And sorry, again new questions / remarks:

 

[1] I just found out, that I definitely need right angled SATA cables, due to the fact that my chassis is too small. I did some research and the only cable I found is one from Adaptec. Does somebody else is aware of a SAS cable with four right angled SATA ports? Or has a different method to solve that "problem"?

 

[2] The Adaptec cable is a SFF-8643 cable. I did not find a converter from SFF-8087 > SFF-8643. Therefore I have to buy a SFF-8643 controller.

 

[3] Back to the Intel controller. I just found out that the RS3UC080 has a IT/IR firmware (link to Intel downloads). I think this should work for me. @Christopher: I think this is what you meant, correct?

[Christopher (Drashna)]

No need to apologize. I defintiely don't mind the questions, at all!

 

1. Well, I'm sorry to hear about the tight space, but that's the problem with small cases.
   SIIG has the SFF-8087 to right angled SATA cables:
   http://www.siig.com/it-products/cables/sas/mini-sas-sff-8087-to-4x-right-angle-sata-fanout-cable-1m.html
   http://www.amazon.co.uk/SIIG-Mini-SAS-SFF-8087-Fanout-CB-S20B11-S1/dp/B00B3329OS
    
2. See above, you shouldn't need that card, but it would definitely work fine.
    
3. Yeah, that should work.  If the IR (RAID, stock) firmware doesn't work right, then you should be able to easily flash it to IT (HBA).  
   So yeah, that's what i mean.

[p3x-749]

I still have two questions 

 

Just to be on the safe side about the M1015:

[1] It does not matter if the IT or IR firmware is installed, in both modes I can use the controller as HBA. Correct?

 

[2] Do you know if "all" LSI controllers handle the HBA mode this way?

 

Just be aware that the stock M1015 does *not* come with IR or IT Firmware, it comes with a "true" RAID card firmware.

Flashing it with the IT/IR firmware is a "cross flash"....there is a risk that you can produce a brick during that process.

 

The M1015 is a true, low-end IBM Raid Controller, but technically identical to the 9210 HBA, only sold to OEMs,

There is another HBA model, the 9211 and luckily its firmare is compatible with the 9210.

You can force the 9211 Firmware to be flashed to the M1015, creating a 9210 (or 9211 respectively), hence.

 

If you buy a M1015 from a dealer "at the bay", some do offer it cross flashed already.

 

I own a couple, crossed flashed myself.

Since this "trick" was made public, prices for the M1015 have risen a lot.

If you want to play save, buy a "real" LSI HBA or a pre-cross flashed M1015.

Note: once cross flashed, you can swap IR/IT firmware anytime, just like with a normal 9211 HBA, using the normal flash method advertised by LSI.

[danfer]

@ Christopher:

Thanks so much for the cable-hint! That saves my life   Now I have to check where to buy 

 

@ p3x-749:

Thanks for the explanation. As you mentioned the price for the M1015 has risen, also the other 2008 OEM cards aren't a bargain (anymore).

For the moment the H220 from HP is back on my shopping list. It has the "newer" 2308 chip and is already flashed as a HBA. I can grab it for ~150€, I think that is the best deal I can make.

 

By the way, I also checked some LSI firmware readmes:

9300-8i [3008 chip / HBA]

9207-8i [2308 chip / HBA]

9211-8i [2008 chip / HBA]

> all firmware packages contain an IR and an IT *.bin [as you mentioned].

 

Then I checked the Intel firmware package [i mentioned]: 

> It is named IT/IR firmware, but it only contains one bin. So I think this one should not work as HBA with Intel firmware. But it might be working with a LSI firmware. But I won't risk that 

[p3x-749]

Yes, if the controller is advertised as a HBA, you should find the IT/IR firmwares...a Raid Controller fom LSI should come with MegaRaid firmware.

Actually the M1015 is a 9240-8i with a "crippled" MegaRAID firmware (no Raid levels 5, 50, a software key for these is sold separately by IBM)...but electrically it is the same.

 

Also, if you buy, do so from a reputable source...there are a lot of fakes already out there.

I remember the good old days, when the M1015 was shipped with every X-Server and then often just got pulled as enterprise customers wanted better RAIDs with BBU and such.

....these M1015 ended in the bay...remembering that I got 2x M1015 for 70USD total ;-)

 

There is a good thread over in the servethehome site/forum, but I can't access it ATM...

Edit: here it is: http://www.servethehome.com/ibm-serveraid-m1015-part-4/

[danfer]

two for 70$!? that sounds like paradise 

today a M1015 costs ~140€/150€ (used or new - makes no difference).

 

ths siig cables seems hard to get in europe, therefore i did some research and found an alternative: the supermicro CBL-0388L-01 (ebay.com auction).

it has four sata ports and a sideband!? and here is my next question: 

i am not sure what the purpose of the sideband is. can i attach four drives with the cable and remain the sideband unused?

[Christopher (Drashna)]

Supermicro has a list of cables with the right angle connector, actually.

 

however, their site is atrocious (seriously, it's from the 90's ...)

 

However, I believe the additional connector is a LED disk activity header, so you shouldn't need it.

 

 

And as for the M1015's, I got mine for about $80USD a piece.

[danfer]

you are right, the supermicro site is not the best.

do you know the cable matrix page? if you select SAS cables, you get a better overview of what they have.

 

by the way: they have awg 30 and awg 26 cables. i have no idea what the difference is!? what is "better"?

i would go for the awg 26 cables, because they look "cooler" 

[Christopher (Drashna)]

Yup, I saw that page.  But it doesn't like to the products, in part because they don't have any pages for them... but there is a list, at least!

 

As for AWG, I am pretty sure it's "American Wire Gauge". So, IIRC, lower means larger wires.  Which is better if you're bending them a lot. 

[danfer]

you are right, it is very hard to find the (supermicro) product you are looking for.

thanks for explaning, the difference between awg 26 and awg 30 does not really sound big.

personal question, what cables do you use?

[Christopher (Drashna)]

The size does make a difference, in the quality of the signal and how much interference it may pick up, IIRC.

 

And I don't use breakout cables.  The Norco RPC 4220 case I have uses SAS connections for the backplanes for the drives.  So SAS to SAS cables. Though the cables I have are Norco and IBM/LSI cables.

[danfer]

I finally bought a SAS-Controller   A LSI SAS 9207-8i HBA [link]

An IT colleague sold it for under 100€. I bought a Supermicro CBL-0278L cable and now everything is installed.

Thanks for your help, your patience and advices!

 

By the way, the new Scanner 2.5.2.3101 BETA has a bug.

It crashes when you start the UI [Win 8 and WHS 2011].

 

[Christopher (Drashna)]

I finally bought a SAS-Controller   A LSI SAS 9207-8i HBA [link]

An IT colleague sold it for under 100€. I bought a Supermicro CBL-0278L cable and now everything is installed.

Thanks for your help, your patience and advices!

 

By the way, the new Scanner 2.5.2.3101 BETA has a bug.

It crashes when you start the UI [Win 8 and WHS 2011].

 

 

Could you try the 3100 build and see if that works?

 

 

And congrats on the purchase. Very nice price, and I hope it treats you well!