Jump to content
  • 0

The Largest Stablebit Drivepool In The World!!


RFOneWatt

Question

Someone has to have it, right?

 

How about we start with the largest pool of the members that participate here?

 

I'm taking an uneducated guess that theoretically Drivepool should scale indefinitely (or to some insane limitation imposed by the O/S, hardware or something else) however we all know the real world is where it's at, yes?

 

I'm sure I'm nowhere near the largest but I've maxed out my Norco 4220, and then some.

 

It's NOW time to start building the successor! 

 

dp.6-2015.JPG

 

 

 

Would love to see what everybody else has going on!

 

~RF

 

 

Link to comment
Share on other sites

Recommended Posts

  • 0

Definitely.   :)

 

It's clearly not me though:

My Pool.png

 

However, my entire pool is duplicated. I've lost drives before and it caused me to lose a bunch of files. Even though most would be easy to grab again, I'd rather just avoid it.

 

I'm also using a Norco RPC-4220. Though, I suspect you have the later revision (do you have 2x 2.5" drive mounts? Because I don't :'( )

Also, I still do have 6 bays empty on my server (something I need to correct :P)

I've already detailed my hardware here:

http://community.covecube.com/index.php?/topic/5-my-rackmount-server/&do=findComment&comment=13

 

 

Also, what controller card are you using?

Link to comment
Share on other sites

  • 0

Well, hopefully we'll find that person. :)  

 

Lookin' good Drashna!

 

I was a lurker here for a good long time before I built this system (AP1, for reference) so I have read your thread and was certainly influenced by it. 

 

No, I don't have two internal drive bays unless you are counting the space on top in the front that is now occupied by four more slots in the 4224?  In that space I have four 600GB White Label Velociraptors acting as my two feeder drives. (hardware striped) 

 

The Norco case came from E-Bay about eight months ago from some strange fella who had a single unit. I think I paid $317.00 delivered for it and I do believe it's the older chassis? The only thing I did was change the fan wall out from the three stock 80MM fans to the "upgraded" wall for three 120MM units.

 

In back, tucked on the side of the power supply (a RaidMax RX-1000AE) is a single Samsung 840 Pro w/ the O/S and the transcoding cache for my Plex server. (this is a bad idea and was meant to be temporary as this drive is on an internal SATA2 motherboard port. No SATA3 on this older Supermicro X8DTi-F) 

 

This server has two controllers, a Highpoint 2740 and a 2720.

 

Highpoint 2720

 

 highpoint.2720-1.JPG

 

Highpoint 2740

 

highpoint.2740-1.JPG

 

Just about all the components have arrived for my new Norco 4224 build, tentatively named WOPR (because we all know how important server names are :D

 

I've been slowly purchasing everything for this build (as I can afford it) basically since the above build was finalized so I'm pretty excited to get moving on that. This will allow me to separate my media server (Plex) from my (primary) storage server.

 

This will also allow me to move forward slightly slower with this configuration and do things a bit differently based on the knowledge gained from building and running the above server for the last several months.  

 

I'm not much of a hardware guy but I'm learning.

 

~RF

Link to comment
Share on other sites

  • 0

 

No, specifically there are two revisions of the Norco case.

One that has 2x 2.5" internal mounts where the "last 4" of the 24 drive bays would go.  That's the newer model (and is theoretically the only one that supports swapping out the middle fan mount for the 120mm mount. 

And one that has only 1x 2.5" bay (and what ... I think is a slim FDD mount). 

 

 

As for my controller:

MSM Output.png

 

 

 

 

RFOneWatt,

 

I just have to know. What are you using all that space for?? Video?

I'm curious too.

 

For me, yes, mostly videos (Movies, TV Shows, Anime, etc). But a lot of software too (ISOs and installers), and WSE Client Backups.

Link to comment
Share on other sites

  • 0

Ahhhh!!! Very nice, Drashna. So your pool is the second largest I've ever seen!!  100% duplicated too. Nice,nice.

 

I know you've been bit before but don't you feel the least bit cheated?   ;)

 

I've used Adaptec controllers almost exclusively since I was a kid but I got a chance to play with an LSI a few months ago and was very impressed.

 

If I remember correctly I was a bit annoyed about a specific feature (can't remember what it is at the moment) not being available without shelling out more $$.  I remember thinking "nickle and diming the consumer" for some reason.  I can't remember exactly why at the moment.. Either way, the price point was good and I bought one for my 4224 build. I liked most everything about the LSI and could tell from the get-go that it would be a bit easier than the HighPoints when it came to configuration and/or compatibility.  My HighPoint controllers which are all FAST, stable and work wonderfully were finicky and a PIA to get right. Several hoops to jump through and even a bit of B.S...  BUT -- They are rock solid now. I have zero complaints (at this point) I even have a RMA'd 2740 that I think is going to end up in my new 4224 build.

 

Ok, so that explains the different 4220 cases. My 4220 is the one with the slot for the slimline CD drive.  I have a small hole drilled in the plastic cutout where the CD drive would be that holds a POT for the fan speed control. (I am using Tornado brand fans.) and to the left of that sits the four WL 600's.... I think I have a picture here somewhere, actually..

 

As to what I am using my space for...... 

 

It's ALL porn.

 

~RF

Link to comment
Share on other sites

  • 0

I've got a few different setups (NASes, Storage Pools, convential JBOD server and a couple DrivePools).

 

My largest DrivePool is currently configured at 88.2TB usable and I've got a couple 6TB drives not in that figure used for parity (SnapRaid).

This pool is strictly for media mostly used by Plex.

 

I've got 4 Windows 2012 R2 servers and two are currently dedicated for multimedia. 2 are more generic servers and hold a lot of VHDX images using windows De-Duplication. Then I've got a few smaller NAS boxes used to hold typical family stuff.

 

But getting back to DrivePool.  I'll be increasing the storage space or the 88.2TB pool in about a month (guessing) when I add the next 8 to 15 bay enclosure for more movies and especially TV Shows.

 

Currently stored on that pool:

160 - 3D Movies
6,200 - Movies
1,150 - Educational Video
18,700 - Music Videos
850 - NFL Games
10,100 - TV Episodes (132 Shows, 613 Seasons)
Music: 4,400 Artists, 12,900 Albums, 105,000 Tracks

 

Carlo

Link to comment
Share on other sites

  • 0

Ahhhh!!! Very nice, Drashna. So your pool is the second largest I've ever seen!!  100% duplicated too. Nice,nice.

 

I know you've been bit before but don't you feel the least bit cheated?   ;)

 

I've used Adaptec controllers almost exclusively since I was a kid but I got a chance to play with an LSI a few months ago and was very impressed.

 

If I remember correctly I was a bit annoyed about a specific feature (can't remember what it is at the moment) not being available without shelling out more $$.  I remember thinking "nickle and diming the consumer" for some reason.  I can't remember exactly why at the moment.. Either way, the price point was good and I bought one for my 4224 build. I liked most everything about the LSI and could tell from the get-go that it would be a bit easier than the HighPoints when it came to configuration and/or compatibility.  My HighPoint controllers which are all FAST, stable and work wonderfully were finicky and a PIA to get right. Several hoops to jump through and even a bit of B.S...  BUT -- They are rock solid now. I have zero complaints (at this point) I even have a RMA'd 2740 that I think is going to end up in my new 4224 build.

 

Ok, so that explains the different 4220 cases. My 4220 is the one with the slot for the slimline CD drive.  I have a small hole drilled in the plastic cutout where the CD drive would be that holds a POT for the fan speed control. (I am using Tornado brand fans.) and to the left of that sits the four WL 600's.... I think I have a picture here somewhere, actually..

 

As to what I am using my space for...... 

 

It's ALL porn.

 

~RF

 

Well, Reddit's /r/DataHoarder/ section has people with 100+TB arrays. So, I feel inadequate compared to some of them. :)

 

As for cheated? How so? If you mean in terms of space?

Not really. I've had drives die on me before. So I lost a lot of the videos I had.  It took me months to recover most of it.  And that was tedious because in some cases, it was a single file missing... and impossible to find JUST that file. Or having to re-rip stuff (which takes forever). 

In fact, the last time that happened, it wiped out 90% of my porn.... so I just deleted it rather than trying to fix it (too much of a headache) 

 

Having everything duplicated means that if a drive fails, nothing is at risk, and don't have to worry about rebuilding all of that.

And that's worth the price I have to pay. And it worries me (literally, and every time) I see somebody post something about "oh, I can easily rebuild it if a drive fails".  Yes, but it's a royal pain doing so.

 

As for the LSI, I've not seen the special feature stuff, other than in some of the rebranded cards, like the untouched IBM ServeRAID M1015. Crossflashing that fixes it (and this card can support ~35 drives at 120MBs before bottlenecking, and that's not as likely to happen with DrivePool). 

 

As for the HighPoint cards, I've had issues with them dropping drives, but in retrospect... that was due to the pile of ST3000DM001's I was using (they're literally all bad now :'( ). Bad drives were probably the cause of the issue.

However, I like the LSI's as they're much more flexible. I'm using one SAS Expander card to host ALL of those drives, as is show. And I plan on getting an SFF-8087 to SFF-8088 adapter and a SAS JBOD expander eventually. :)

Hosting all my storage on the one card. :)

 

 

 

I've got a few different setups (NASes, Storage Pools, convential JBOD server and a couple DrivePools).

 

My largest DrivePool is currently configured at 88.2TB usable and I've got a couple 6TB drives not in that figure used for parity (SnapRaid).

This pool is strictly for media mostly used by Plex.

 

I've got 4 Windows 2012 R2 servers and two are currently dedicated for multimedia. 2 are more generic servers and hold a lot of VHDX images using windows De-Duplication. Then I've got a few smaller NAS boxes used to hold typical family stuff.

 

But getting back to DrivePool.  I'll be increasing the storage space or the 88.2TB pool in about a month (guessing) when I add the next 8 to 15 bay enclosure for more movies and especially TV Shows.

 

Currently stored on that pool:

160 - 3D Movies

6,200 - Movies

1,150 - Educational Video

18,700 - Music Videos

850 - NFL Games

10,100 - TV Episodes (132 Shows, 613 Seasons)

Music: 4,400 Artists, 12,900 Albums, 105,000 Tracks

 

Carlo

 

Very nice!

And yeah, definitely larger than my pool. :)

 

And as for the Data Deduplication feature, that's a really nice one for hosting VHD(X) files! Depending the exact contents ... 60-70% savings! (I'm using it for HyperV, specifically)

 

And which 8-15 bay enclosure?

Link to comment
Share on other sites

  • 0

Depends for me on which way I want to go.

For 8 bay http://www.newegg.com/Product/Product.aspx?Item=N82E16817707367

For 15 bay http://www.istarusa.com/raidage/products.php?model=DAGE415U20-PM#.VYI2zflVhBd

 

The StarTech eSATA 8-Bay Hot-Swap SATA III Hard Drive Enclosure with UASP works quite well and pretty cheap at around $300 plus shipping. Connect it up via eSata, USB3 or UASP and it works quite well with DrivePool.  I picked one up a couple of weeks ago and put 8 4TB drives in it and haven't had a single issue with it.  I've first tested eSata, then UASP and then normal USB3.  I left it connected via USB3 and haven't had a single issue.

 

It's not quite as fast as the 15 bay could be since you have 8 drives on a channel vs 5, but for normal (non benchmark use) and Plex use it works just fine.  I stream to anywhere from 6 to 10 people in the evening and it handles that load without breaking a sweat.   What I like is that you can take this box and plug it into just about any other computer/notebook or router and be able to access your data.  One of these boxes and a higher end home router like an ASUS gives you a nice NAS box.

 

The reason I'm leaning toward it is that it's very quite and keeps the drives cool.  It's not a rackmount unit but I don't really care about that for a home unit since it's versitile and works well via UASP/USB3 so I don't need to worry about sata/esata ports.  I still have a few sata ports available (8 I think) but I'm reserving them for internal drives.

 

I could go SAS also but really just don't honestly see the need for a media server when a simple box like this can easily add 8 drives at a shot. Throw 6TB WD Reds in it for a reasonable large amount of data at a decent price.

 

Carlo



And as for the Data Deduplication feature, that's a really nice one for hosting VHD(X) files! Depending the exact contents ... 60-70% savings! (I'm using it for HyperV, specifically)

 

And which 8-15 bay enclosure?

 

I've got a 24 core (48 with hyperthreading) SuperMicro server with a few 1 TB drives in it running over 75 virtual machines.  They are all 2012 R2 installs.  Machine flys.

 

Not only does the de-duplication help with storage but also helps the machines run faster as it helps with caching/memory and other important items (not trying to be technical). :)

 

Carlo

Link to comment
Share on other sites

  • 0

Hi Carlo, welcome Philmatic.

 

Looks like Carlo is king but as he said 33TB isn't anything to sneeze at. Especially if it's running, working and reliable.

 

I can certainly see your point on duplicating everything Drashna however that's basically the reason I run DrivePool. I have my important stuff duplicated and the replaceable stuff unduplicated. It's nice to pick and choose with folder duplication. I haven't really done the math but I *think* if I wanted to take that big of a hit on drivespace I'd run a conventional RAID. Maybe. (Not that there aren't plenty of other advantages to Drivepool..and I do need to rethink some things)

 

I was kidding about my Drive Pool being ALL PORN however the majority of my pool is video. (my personal favorite being older TV) however I know I fit the definition of data hoarder. I've got every file and picture that's ever been on any of my machines since around 1997..heck.. even from before that. I've got all of my C-64/128/Amiga disc images, all of my old BBS files, clients, business' etc. all online.

 

I also bought the first consumer digital camera that ever came out and went through several girlfriends with that one so needless to say "stuff adds up!' 

 

This new 4224 build is going to be Server 2012 Standard and I think I'll have some fun with deduplication. I can see it has it's place.

 

I really want the disk subsystem to fly on the this new 4224 build.

 

On hand I have a HighPoint 2740, a LSI 9240-8I and a Dell PERC H710P I scavenged in addition to the ports available on the motherboard, a Supermicro X-10DRI-O however I am not opposed to spending some "real money" on a new controller...

 

Heck, here's a picture of most the hardware that has arrived so far, all sitting on top of my running 4220 on the mini "bench." I'm pretty sure the only thing missing in this picture are two additional EVO 850's that just showed up. (for a total of four)

 

I do also have six 600GB 15K Cheetah SAS drives but I don't know if they will make it into this build or not. They are used with about two years of power on time but barely any usage.

 

I'm pretty sure I'm going to populate most of the bays in the Norco with 7200RPM HGST NAS drives or WD Reds.. I haven't made up my mind yet.  

 

The 4224 case just arrived and is sitting next to this 4220 awaiting insertion of parts.  B)

 

Decisions, decisions.

 

~RF

 

2015-06-14.4224.build.jpg

 

 

 

 

Link to comment
Share on other sites

  • 0

Well, the Dell card is a rebranded LSI controller (LSISA2208), and the LSI 9240-8i is a slightly older chipset. :)

 

Either of those cards will serve you quite well. (all three of these cards, I believe are PCI Express 2.0 8x cards, which can handle running 34 "spinning" drives at 120MB/s before saturating the bus) 

 

 

Deduplication is very nice, and Microsoft has a list of what contents get what sort of savings. The best savings is with Virtual Machines.  Which shouldn't be surprising, as if you're using the same OS for a bunch of VMs (or even similar), there is a lot of redundant data.

 

 

Also, if you do plan on doing virtualization, I would recommend using a separate box for that, or at least separate from your media server system.  Fewer things competing for resources is better. 

Link to comment
Share on other sites

  • 0

Now I am really jealous. Do you actually run them through an emulator?

 

Yup, sure do.

 

For the C-64 I've been using C-64 Forever by Cloanto for years (It comes with about 100 free games).  :D  For my old modded X-Box I use Vice64.   For the Amiga, it's Amiga Forever.

 

If you're a fan, looking for disk images or the least bit interested in the C-64 head on over to C-64.com.  Lots to explore... certainly the best C-64 site on the net IMO.

 

 

Well, the Dell card is a rebranded LSI controller (LSISA2208), and the LSI 9240-8i is a slightly older chipset.  :)

 

Either of those cards will serve you quite well. (all three of these cards, I believe are PCI Express 2.0 8x cards, which can handle running 34 "spinning" drives at 120MB/s before saturating the bus) 

 

 

Deduplication is very nice, and Microsoft has a list of what contents get what sort of savings. The best savings is with Virtual Machines.  Which shouldn't be surprising, as if you're using the same OS for a bunch of VMs (or even similar), there is a lot of redundant data.

 

 

Also, if you do plan on doing virtualization, I would recommend using a separate box for that, or at least separate from your media server system.  Fewer things competing for resources is better. 

 

Not sure how this should influence but the Highpoint 2740 is a 16x PCI Express 2.0 card and the Adaptec has 512MB of onboard cache.

 

This showed up today so I should have the motherboard fired up in the next couple of days and I'll be able to play with the controller on that.  

 

corsairRM1000.jpg

 

Then to decide on the disk configuration.

 

I really liked the LSI for the few minutes I played with it and the few days I ran it.  (While I waited for my first Highpoint 2740 to be RMA'd) so I'm still up in the air on how to configure things.

 

I of course agree with ya on the VM's..Currently my virtualization is handled by ESX 5.5 on a freshly maxed out / refurbished Dell Poweredge 2900 box running in the neighbor's basement. (the price they pay for a wired gigabit connection to my network :D ) 

 

I also get free eggs from the other neighbor for DLNA access to Serviio... haha it all works out.   :)

 

This 2900 is the box I back up all of my important stuff to, secondary domain controller,  W7 script box, etc.

 

esx.JPG

 

 

My virtualization preferences might be changing as I start to play with Server 2012 a bit though.. 

 

-RF

Link to comment
Share on other sites

  • 0

Well, it sounds like you ahve a nice setup. :)

 

One major difference between ESX and HyperV, ESX lets you change hardware on the fly. That may not be a huge thing for you, but it may.

However, HyperV works really, really well in a domain environment. :) Really well.

 

 

Thanks Drashna.

 

I'm an old man with many hobbies and in the beginning of the year I sold a prized piece of HAM radio gear to fund these two Norco builds.

 

That is good to know re: hardware modification on the fly.

 

My first experience with Hyper-V was several years ago when I was working with a client that needed USB passthrough. At that time I wasn't able to make it happen with Hyper-V however ESX supported it... so that let me on the road to ESX, which I've deployed extensively since then.

 

When I was working on the configuration of my 4220 box a few months ago I installed Hyper-V and played with it for several days and am pretty much sold on it.  

 

Although I didn't test USB passthrough on Hyper-V -- I'm guessing it works fine these days?

 

-RF

Link to comment
Share on other sites

  • 0

Yup, sure do.

 

For the C-64 I've been using C-64 Forever by Cloanto for years (It comes with about 100 free games).  :D  For my old modded X-Box I use Vice64.   For the Amiga, it's Amiga Forever.

 

If you're a fan, looking for disk images or the least bit interested in the C-64 head on over to C-64.com.  Lots to explore... certainly the best C-64 site on the net IMO.

-RF

 

OK, OT, sorry, but do you use a digital joystick with those? I would not know where to get one and how to connect it. And it is part of the experience IMHO.

Link to comment
Share on other sites

  • 0

Well, it sounds like you ahve a nice setup. :)

 

One major difference between ESX and HyperV, ESX lets you change hardware on the fly. That may not be a huge thing for you, but it may.

However, HyperV works really, really well in a domain environment. :) Really well.

You can adjust many things in Hyper-V also if you use versino 2 machines images.

Link to comment
Share on other sites

  • 0

HyperV still doesn't support USB passthrough, sadly.

And even Gen2 VMs don't really let you change the hardware either (it doesn't let you change the RAM or CPU cores, which is what I was talking about, but in both, you can always add and remove SCSI drives without powering off the system).

 

There are ways to do the USB passthrough, but they're commercial software.

 

Though, USB passthrough is really an edge case.  Depending on what you need it for... it may not even be necessary.

Link to comment
Share on other sites

  • 0

OK, OT, sorry, but do you use a digital joystick with those? I would not know where to get one and how to connect it. And it is part of the experience IMHO.

 

No, I never have.

 

Sounds like you need to poke around over at http://c64.org  -- You will certainly find the answer to your question. 

 

I never actually play too many video games.. heck, even when I was importing them into the US when I was a kid I didn't play many of them.

 

I was more into the demo scene, importing and collecting.

 

One of these days I need to hook all of my Commodore stuff back up.  It's been sitting in a storage closet at my parents house since the FBI gave it back. (15 years after they took it!)

 

~RF

Link to comment
Share on other sites

  • 0

I feel kinda weak here. I investigated a NORCO setup years ago when I started, I went instead with a lower cost modular strategy and am using that same strategy still, years down the road.

I started with WHS using built in pooling, went with DP shortly after it came out, abandoned WHS after trying the "upgrade" without happiness, tried the Server "Essentials" line, again unhappy, tried Server 2012, again unhappy, I ended up moving to Windows 7 x64, and splitting my pool into two parts, one on a dual Xeon workstation, with two drivepools, and another drivepool attached to my everyday workstation.

So, the everyday workstation has a drive pool size of 40.5 TB, including two 256GB SSDs as 'landing zones'. The dual cpu workstation has two drivepools, one that is 46.5 TB, and the second is 16 TB. The larger pool has two SSDs as a landing zone.

All three pools are fully duplicated. The dual CPU WS holds material that is not needed on a daily basis, so I can turn it off to save electricity.

Bottom line, I have 103 TB of drivepool storage, all duplicated.

 

Interesting reading the C=64 stuff, I was really into it back in the day, late 1980's early 90's, in the end I had several, my main machine had a CMD SCSI HDD, it was a laugh to see the light blink on disk access, you had to watch quick. I also had 8 1MB SIMMS in an expansion card device. About that time I hit the limits of the format, and moved to the Amiga in 1991 or so.

 

I would like to add also that I have various HDDs from various manufacturers, here is my experience:

 

Largest number of failures: WD green drives. My 5th failed drive is sitting in front of me right now.

Second: Nobody "stands out". Rare failure due to age. Lots of drives have Stablebits warnings but still keep on ticking.

I have heard Seagate is bad, my personal experience does not, at this time, reflect that.

I have Seagate, Hitachi, WD, Toshiba, Samsung drives. A lot of my more recent drives are external models removed from the case, generally Seagate. No issues at this time.

 

At this time I have four 5TB drives, the first two were Seagate, the most recent are Toshiba. So far they 'work'.

Link to comment
Share on other sites

  • 0

Gringott, don't fee "weak" at all.  A modular strategy definitely isn't a bad one, at all. It just depends on your needs.

 

I also got the rackmount case for really cheap. :)

Largest number of failures: WD green drives. My 5th failed drive is sitting in front of me right now.

Second: Nobody "stands out". Rare failure due to age. Lots of drives have Stablebits warnings but still keep on ticking.

I have heard Seagate is bad, my personal experience does not, at this time, reflect that.

I have Seagate, Hitachi, WD, Toshiba, Samsung drives. A lot of my more recent drives are external models removed from the case, generally Seagate. No issues at this time.

 

At this time I have four 5TB drives, the first two were Seagate, the most recent are Toshiba. So far they 'work'.

And as for the experience, I'll second that.

I've not really had any one brand do significantly different from the others.  And I  think that every manufacturer has had large batches of "duds", as well. 

Link to comment
Share on other sites

  • 0

I myself use a variety of different manufacture drives for my personal stuff.  I work in IT and work with large data (I mean LARGE data) spread out over numious SANs with more than 10K drives.

 

In my professional life I've seen a trend of higher failure rates among Seagate drives than any other manufacture (percentage wise), bot not in the last 2 years (about even).   These are mostly all Enterprise drives and not the common home user or NAS drives that most of us would use routinely.  So I'm not sure how much this matters.

 

I think a lot of failures over time by some companies/users is choosing the wrong drive for the application at hand.  Drives are built differently.  For example putting 40 drives in one case/cabinet will be different then running one or two drives in a normal case.  Lots more vibration and movement.  This can also be even more of a factor for people who try and stuff lots of "home" drives in large cases then use suspend/power down features.  When the drives spin down/spin up they cause extra vibration that other drives in the running in the case won't like, especially when you have 5 to 10 drives spinning up at one time, etc...  Modern drive are much better with this type of thing, but can take it's toll over time.

 

I'm not sure what value this post has except to say that if any one particular vendor had a much higher failure rate than other vendors they probably wouldn't still be in business. The HDD market place is extremly competitive and any manufacture putting out "junk" would surely be noticed in this day and age.

 

Just pick the right disk for the job at hand based on the features it has you need and don't worry to much about who made it. Of course follow common sense rules such as use the same drives if building a traditional RAID type thing, etc

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...