Project: Rackmount Overkill

thideras said:

You could probably do something like that, but if I'm routing and filtering on a network level, I'd rather give the pfSense box its own dedicated NIC. Doing anything else is just asking to have your network exploited.

Click to expand...

^^. I always keep at least one $10.00 PCI-X Intel PRO/1000 MT for things like this.

Since I retired the R410 from router duty, I had two Samsung 840 Evo drives laying around. I could have sold them, but they don't go for a ton. Instead, I ordered some very nice 2.5" to 3.5" backplane compatible (this is important) converters. The drive bolts into the connector, and the converter bolts right into the stock sled, which goes into the SAS expander. They work slick. Compared to other products out there, these for 12.50 shipped is a great deal. They are Newertech AdaptaDrive converters. Solid aluminum and built well.

The remaining solid state drives arrived today!

Everything has been working great, and I haven't changed much on the servers, so there isn't much to talk about. The biggest change I made was with the SAS expander: I swapped out the weird stock unit to work with a standard ATX power supply. I'm not sure how long the stock power supply has been running, since I bought the unit used, but I don't trust non-standard non-easily-replaceable power supplies. I have a bunch of pictures, so I'll likely post them here soon, if I remember.

A couple of days ago, the UPS units got a great workout. The utility company cut the power to my apartment on accident. It was out so long that one UPS unit ended up powering down, which ran one PSU of both servers and the entire network stack, and the remaining one had about 2-3 minutes before it was going to power down.

ZFS is running like poop (30-50mb/sec), but I think the issue is multi-faceted. I had millions of small files on the server, accumulated over many backups and lack of cleaning things up. Took about a day to go through the whole array, and there is a fraction of the files it once had, about the same space used, though. Fairly sure that was causing a lot of fragmentation. Additionally, I'm only running one vdev, which is probably not helping.

Since I got my Rackable Systems SAS expander, I've been worried about the power supply going out and leaving me without an array. The stock PSU has connectors built directly on the unit, and while they look and are standard ATX specification connectors, they are the wrong type of connector. Finding a replacement power supply is interesting, and I didn't want to be dropping big money for such a simple replacement. I've done quite a bit of rewiring computers in the past, so this one should be easy.

I picked up a Silverstone 300W ST30SF SFX form factor power supply. Using an extra unit I have laying around, I determined a full size ATX power supply won't fit without modification, and I didn't want to bother with that. However, a SFX unit barely fits without modification.

I started by disassembling the unit, removing drives, and taking all the cables out. Then, I located a spot for the PSU, found a spot to bolt it down, and made custom cables.


SAS unit with the drives removed and cleaned up a little bit.



Inside of the SAS unit before any cables were removed. The stock power supply isn't bolted down because I had removed it to check a few things.



Silverstone PSU that will be installed. Completely stock.



All cables removed and pushed out of the way. The tie down power got removed later.



Another view with no cables.



I stole one of the tie down brackets from the stock power supply, and bolted it to the stock PSU with stock screws. I was surprised the stock screws were long enough for this to work!



Power supply installed in its final location. It moves around fairly easy, but this shouldn't be an issue for a unit that never moves.



Now to start on making and converting cables. Both these cables went into a ATX connector, but I need the "male" version of this connector instead of the "female" one, so I can plug it directly into the Silverstone.



Pins installed on the cables power the SAS logic board. These pins are pretty small; had to use a macro lens to get a decent picture.



Final version of the flipped ATX 24 pin connector, which powers the SAS logic board and the power control board.



Final version of the cables which power the hard drives. The 4 pin "Molex" connectors in the bottom right used to be 4 pin Minifit Jr connectors, like the ones on the left.



Power supply and all cables installed in the SAS unit. A little more messy, but a lot more reliable than the stock unit.





I've been using it for two months without any issue. The fan in the stock PSU unit was very loud, and could get louder when it was warm. This SAS unit with voltage drops on the back fans is very quiet while keeping the drives at a reasonable temperature.

Nice work man, looks great!

Wow, it has been nearly a year since any activity in this thread! Nothing has really changed and the systems have been working really well, except for vCenter filling up its main virtual disk. I've been considering upgrading the drives in the server since 3tb drives are getting dated, but I'll wait for tax returns and work stuff.

I ordered a pair of wireless cards for the router, since my wireless router has been flaky, and I'm simply fed up with it. Was a pain to figure out what works with pfsense, but I seem to have found one which works well.

Once it starts warming up, I'll need to find a way to clean the dust out of the servers or haul them outside for a really good cleaning. They are getting pretty bad.

So all your servers or VM's rather, host your website, and serve files?
Trying to wrap my head around all that server pr0n and what you use it for

I'm suddenly starting to want a rackmount server even though I said I don't like 'em.

Silver_Pharaoh said:

So all your servers or VM's rather, host your website, and serve files?
Trying to wrap my head around all that server pr0n and what you use it for

I'm suddenly starting to want a rackmount server even though I said I don't like 'em.

Click to expand...

All the servers host internal stuff; file server for storage (main usage), backup server, a couple Windows domains, stuff for work, etc. I have a VPS for my website.

thideras said:

All the servers host internal stuff; file server for storage (main usage), backup server, a couple Windows domains, stuff for work, etc. I have a VPS for my website.

Click to expand...

Ohh Okay so combined, it's basically one large learning tool that also server a purpose like file storage and such.
Play 'n learn?

Silver_Pharaoh said:

Ohh Okay so combined, it's basically one large learning tool that also server a purpose like file storage and such.
Play 'n learn?

Click to expand...

Yeah, I try things out, and if they are useful, I'll keep them. If you go back through the thread, you can see it was just a file server. Usage has expanded beyond that.

I got the wireless cards today and remembered my router only has one PCIe slot; crap. To test them in a live setup, I used my backup pfsense box and set the wireless up over a couple hours. I finally got a working configuration with multiple wireless SSIDs running from one wireless card. It was a pain to setup, as there is very little information online.

To get wireless fully working, I'll need to upgrade the motherboard (and case) in the router. Tempted to just go full ATX in case I want to add more crap in the future.

Good to see your still playing with the rack Thiddy, I just got my rack out of storage recently and its curled up in the corner of the garage waiting for some love.

New router parts are here. I can't fully assemble the system until I can take down the current one; I need its RAM.



EDIT: I just realized I may not have a heatsink for this system. Crap. I may steal the one from the HTPC, since I'm fairly sure it is 1150.

Whoops, posted the above in the wrong Project thread!

Looking good

Well, I finally got the router up and routing. Effectively, anything which could have gone wrong, did. I got the case, power supply, and processor on Saturday from Austin, and it was well packaged. When I started putting the system together, I quickly realized I didn't have a heatsink for the new system. Luckily it is the same socket as the HTPC, so I decided to cannibalize it, stealing its memory and heatsink. OK, crisis averted.

I assembled the system, making the wiring all neat and hidden. It looked good and took me a bit of time. Fired up the system, and there is a loud noise which sounds like something hitting a fan. I look the system over and find out the noise is from the power supply. Crap, ok, time to rip the whole thing apart and figure out what is wrong. The box it was shipped in was dropped so hard it bent the bottom of the case slightly and bent a component far enough it was touching the fan. I located the part in the power supply clipping the fan and was able to push it back slightly. The part was on the mains side of power supply, and I made sure it wasn't touching anything around it. Firing up the power supply outside the case showed my fix worked, yay.

Then, I put everything back in the case, and made the wiring all fancy. I grabbed my spare optical drive, and started up the install for pfSense. While it was loading, I heard a "ting ting", which I initially thought was the optical drive until I realized it was the exact same noise the power supply made before. I lightly tapped the top of the power supply and the noise came back entirely. Not wanting a fire hazard, I decided to remove the power supply.

There were a couple options at this point: wait until a new power supply comes in, pull the one from the HTPC (pita), or pull the one from the current router (risky). I worked early-ish this morning, and my internet must be up to do my job, so I decided to take a risk and take down the current router. I got everything moved over to the new system fairly easily, got all the wires hidden and fancy, fired up the system, and everything just works. Yay.

Except, pfSense keeps getting stuck at 38% while installing. Searching online, I find very little information which was helpful. People suggested to disable virtualization, turn on virtualization, disable speedstep (and the like), and I think someone mentioned sacrificing animals at one point. Anyway, nothing helped. I tried a slightly older disk I used on the now-torn-apart router, and it got stuck in the same place. This wouldn't be a problem if the pfSense disk didn't take 5 minutes to load to the point where it got stuck, with a 9 second window in the middle to hit "install" (who designed, this, seriously?). I hate BSD. I've always had issues with it, and the errors it throws are useless.

Around 3am this morning, I got everything running. Have some poorly taken photos way after my bed time:


Picture of the guts, without the support brackets installed, which hides nearly everything.




I've yet to create the wireless networks, which is the entire reason I got this thing in the first place: my current wireless router is concentrated distilled butt.

You got the router setup and you got some sleep
Well done Corey!

So what was the deal with pfsense anyway?

Silver_Pharaoh said:

You got the router setup and you got some sleep

Click to expand...

"sleep"

Silver_Pharaoh said:

So what was the deal with pfsense anyway?

Click to expand...

Not sure what you mean.

thideras said:

"sleep"

Not sure what you mean.

Click to expand...

Stuck at 38% during install.

Or was it just that window you had to wait 5 minutes for.