Project Baby Rack

[cullam3n]

I figured I needed to document some of this stuff, and also because rackmount overkill has been such a inspiration.

I'll update this thread with things I'm doing, both hardware and software.





EQUIPMENT

Rack:

Great Lakes 13U

Server #1
Hardware:
3U Chenbro RM31300 with two SK33502T2 HDD cages
2U Chenbro 500w (PS-P2H-5500V)
Supermicro X8DTE
2x Xeon L5520s (Socket 1366)
48GB (8x6GB) DDR3 FB-ECC RAM
200GB Seagate 7200.8 (boot)
2x3TB Seagate 7200.14 (backup of important stuff)
IBM M1015 with passthrough IT firmware
---4x3TB Seagate NAS drives (RAID 5, storage)
---4x1TB WD Blues (RAID 10, VM hosts)
Intel PRO/1000 dual NIC card

Operating system(s)
Proxmox
---VZ Container - Debian 6 (file server)
---VZ Container - CentOS 6 (MySQL)
---VZ Container - CentOS 6 (Web Server)
---QEMU - Windows 2008R2 (DHCP, general server duties)

Server #2
Hardware:
1U Supermicro 812L-280UB
1U 280W Athena
Supermicro X7DBU
2x Xeon L5320s (Socket 771)
8GB (4x2GB) DDR2 FB-ECC RAM
160GB Maxtor

Operating system(s)
Windows 2012 R2

Server #3
Hardware:
2U Norco RPC-230
Seasonic 300w ATX
ASRock E350M1/USB3 (AMD E-350 CPU)
2GB DDR3 1066
4x400GB (2x7200.8, 2x7200.10s)
Intel PRO/1000 MT single NIC

Operating system(s)
FreeNAS 6

Network device(s)
Dell Powerconnect 5224

So Server #1 is really my main server that has everything. Server #2 I'm just messing around with and trying stuff out. Server #3, I had a 4x400GB NAS back when I didn't know anything around 2008 and I thought I lost all my data (poor practices, ignored warning signs, etc). I recently put it in this setup because I'm trying to recover some of the data that's on it.

 

[wagex]

gimme my baby rack baby rack baby rack bacy rack... lol

nice rack

 

[cullam3n]

Bought this so that I can migrate my setup out of the 1U. Theoretically it should fit in the Norco 2U unit. I should have it mid-late next week.

 

[cullam3n]

So a local guy was selling registered ECC DDR2 (2x2GB), I figured I could add it to the 4x2GB I already have.



Turns out that my current 771 motherboard uses FB-DIMMs (fully buffered ECC) which are keyed differently and not inter-operable with registered ECC RAM.

I was about to see if I can give them back, when I realized that the motherboard I ordered uses regular registered ECC. Might as well keep them now.

 

[cullam3n]

Server #1: Made my 3TB on-site backup a RAID 1. Previously it was in my old setup it was by itself. I made a RAID 1 with the empty 3TB drive in a degraded state, copied everything over, and then re-partitioned the old drive and added it to the array.

root@Proxmox:/mnt/backup2# parted /dev/sdj
GNU Parted 2.3
Using /dev/sdj
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sdj: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  3001GB  3001GB  ext3         primary  lvm

(parted) mklabel
New disk label type? gpt
Warning: The existing disk label on /dev/sdj will be destroyed and all data on this disk will be lost. Do you
want to continue?
Yes/No? yes
(parted) mkpart primary 1 -1
(parted) set 1 raid on
(parted) exit
root@Proxmox:/mnt/backup2# mdadm /dev/md1 -a /dev/sdj1

root@Proxmox:/mnt/backup2# mdadm -D /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Sun Oct 20 18:12:52 2013
     Raid Level : raid1
     Array Size : 2930133824 (2794.39 GiB 3000.46 GB)
  Used Dev Size : 2930133824 (2794.39 GiB 3000.46 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Tue Oct 22 01:37:02 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : Proxmox:1  (local to host Proxmox)
           UUID : 5c12a6ac:576e3fc3:a778d826:5c04965e
         Events : 8227

    Number   Major   Minor   RaidDevice State
       2       8      145        0      active sync   /dev/sdj1
       1       8      129        1      active sync   /dev/sdi1

root@Proxmox:/mnt/backup2# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid10] [raid1]
md1 : active raid1 sdj1[2] sdi1[1]
      2930133824 blocks super 1.2 [2/2] [UU]

md10 : active raid10 sde1[0] sdh1[3] sdg1[2] sdf1[1]
      1953262592 blocks super 1.2 64K chunks 2 near-copies [4/4] [UUUU]

md127 : active raid5 sdb1[0] sdc1[4] sda1[2] sdd1[1]
      8790795264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

 

[TiZakit]

love me some mdadm

 

[cullam3n]

Got my LGA 771 motherboard today. More ECC RAM on the way, should be in tomorrow.

 

[cullam3n]

Had a little snag... couldn't get it to POST at first, had one long continuous beep. I was using my ATX Seasonic 300W to power it, but I found the cause of the issue is that it only has a 4pin 12V instead of a 8pin. I plugged in my 1U 280W from the original setup and it booted up fine. I checked the amps on the rails and they are about the same, the only difference being the Seasonic has dual rails by the 1U PSU does not.

In the Norco case I can't use a bottom/top mounted fan because there is no ventilation there, it has to be a pass through. Going to get a 8pin adapter and run with it, since the 1U and the Seasonic put out the same power (theoretically).

Also, even though this board is labeled as an MATX, it doesn't quite fit the standard standoff spacing specification. It's still secure enough with the ones that did line up. I also don't have another mATX mobo lying around, but I think that it's slightly bigger by half an inch in both directions.

 

[cullam3n]

Forgot I took this, but here are all the drives in Server #1.

2x3TB Seagate 7200.14 (backup of important stuff)
IBM M1015 with passthrough IT firmware
---4x3TB Seagate NAS drives (RAID 5, storage)
---4x1TB WD Blues (RAID 10, VM hosts)

 

[Adragontattoo]

Suggestion, if possible look at changing the Center 5.25 bay covers on the 3u Chenbro to solid covers to force more airflow through the Drive caddies. I know on a few different systems I have built that this would help keep the drives cooler.

 

[cullam3n]

Appreciate the tip but both those NAS and WD Blue hard drives run very cool. The Blues are unbelievably light, and those NAS drives never seem to run warm. The Barracuda drives can run a little warm when they are being used, but they don't do much except mirror data weekly. In the bottom of the 3.5in drives is my boot drive which is an older drive that runs warm so I don't want to block that.

 

[cullam3n]

Got another 2x4GB ECC RAM pair today.

 

[Adragontattoo]

Keep an eye on Server Supply dot com and also check Unix Surplus dot com (Unix also has sellers on Ebay with a TON of parts and some decent deals occasionally).

I have bought from both without issue previously and have no complaints about either site.

Unless you mandate sticking with Supermicro, you might want to look at the Intel S5000PSL MB. You can get them shipped with 2x heatsinks for ~50USD. Depending on a few things, it is likely to be part of a planned upgrade to my rack HW.

 

[cullam3n]

Thanks for the tip Adragontattoo.

So swapping in the motherboard, I had two issues:

1. Even though my ATX PSU that I have in there is comparable to the 1U, it lacks an 8pin EPS connector. Without all 8 pins connected, the system would have a continuous beep and not boot. I managed to put in an adapter to use all 8 pins and that now that works.

2. Windows 2012R2 would not boot at all. It would get freeze at the splash screen with the rotating circles every time. After some research it looked like a BIOS update would fix it... but their utility .exe only accepts floppies. They had a zip file with the flashware on it but I didn't have a chance to do it myself. I threw 2008R2 on a thumbdrive, and it installed successfully. I figure I'm going to run with that on here, and I may migrate my other 2008R2 to a 2012R2 in the future. Or actually upgrade the BIOS to see if that helps. Not sure yet.

 

[cullam3n]

On the above server, I ended up putting a 92mm fan lying on top of the CPU heatsinks, and a 80mm fan blowing out the expansion card bracket. The chipset was just getting too hot and there was no airflow. All is fine now.

I finally got around to creating the MySQL database on the CentOS container, and now have synced my XBMC HTPC fully. I did this because I want to have another XBMC box and TV in the bedroom for those lazy days where I don't feel like getting up to the couch.

What my Proxmox looks like at the moment:

 

[cullam3n]

Had to power down server #1 the other day due to the storm taking the power out. Took the opportunity to see if there was any dust; it was pretty clean.

 

[cullam3n]

Some updates, in no particular order:

Relocated the switch to the back, and cleaned up some of the wiring a little bit. Excuse the mess


I moved the M1015 up a few slots in server #1 compared to the last pic I posted. I was looking through the manual and saw the bottom slot is only Gen1 X4, the slot above that is Gen2 X4, and then the rest are Gen2 X8s. I also swapped out the HD4350 that was in there for a X1300XT (i think). The cooler isn't as big as the passively cooled HD4350. The onboard video doesn't work on this board, and I'm not quite sure if it will boot without a video card detected (it does a video beep code, but I haven't checked if it actually boots after that).

Speaking of stuff not working, it seems like that the HDD controller on the mATX Supermicro died. Can't get it to detect any hard drives at all. I took the L5320s out and put them back in the 1U.

 

[Adragontattoo]

I know on a couple of my boards that they will NOT boot if a NON GPU is in the first PCIe x8/16 slot. Some of them give an option to choose what the display order is but even then they require a GPU in the first slot. I'm honest curious, did you notice much if any throughput increase by moving the 1015? I can't imagine there being much but I would be interested to know.

I made it a mission to grab any and all PCI GPUs that I could at the old job specifically for boards that didnt have onboard video. They are almost always old enough to be compatible with the majority of generic drivers in most every OS from ~Win2k-current and I have yet to find a Linux/BSD distro which didnt work with them.

Any idea what killed the Controller on the mATX? Can you reset the BIOS XOR try a different single HDD? Any chance to pull the board out of the case to see if you have any bad/blown caps or burn marks from a short? Supermicro boards are typically just shy of bulletproof. Actually they might survive a gunshot depending on location...

 

[cullam3n]

On Server #1, the board only has open-ended slots on slot 2 and 4, so that's where I have to put the video card. The previous owner never actually tested the onboard, but he was kind enough to send me the video card he used in it after the fact.

I didn't do any benchmarks before and after. I did a hdparm test but it's about the same.

I'm not sure what killed the controller on the mATX. I flashed the BIOS recently to see if it would help with installing Server 2012R2, but I can't remember if it happened before or after that. I've tried multiple hard drives and cables and ports. I did a quick look at the time but I didn't notice any blown caps.

 

[cullam3n]

A few updates:

Most of the hardware from Server #3 is now in my bedroom for a second HTPC. I bought a GT 620 from the classifieds and put my old GT 520 in the second HTPC as well as a Crucial M4 60GB I picked up locally. Runs XBMC just fine on a 720p screen.

So Server #1 wasn't annoyingly loud, but I wanted to make it more quieter. For school, I am allowed to take tests at home which being watched with a school provided webcam. Since the server rack used to be right next to my desk, halfway through the exam I got interrupted by a technician who said my microphone was too loud. I figured I could make the server quieter while maintaining reasonable temps.

Got some quieter 80mm fans in and a fan controller.


Contrary to what I thought, the four 80x80x32mm fans I have on the fan wall weren't the loud ones. These two were, which are attached to the hard drive cages.


Swapped out the rear case fans.


I've highlighted those two hard drive fans I swapped out as well. These were actually really easy to do despite what looks like the lack of wiring etiquette. It's actually pretty organized.


Fan controller out front. It doesn't quite sit flush with the front, but this fan controller was inexpensive and feels pretty cheap, so I'm blaming it on QC.


LEDs are bright, but fortunately the case doesn't face me, and that the front door is tinted Plexiglas so it doesn't blind anyone. Fan controller works as promised at around 70%, although I'm not sure if it's a 0%-100% or not, as only one fan stops when all the knobs are off.


Got some more cables plugged into the switch to prepare for the return of the 1U (soon). I took it offline for a little while. Also noticed I move the PDU to the rear.


With about 5% CPU usage and a load overage of 0.15. Not too bad.

root@Proxmox:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0:       +50.0°C  (high = +80.0°C, crit = +90.0°C)
Core 1:       +49.0°C  (high = +80.0°C, crit = +90.0°C)
Core 2:       +50.0°C  (high = +80.0°C, crit = +90.0°C)
Core 3:       +47.0°C  (high = +80.0°C, crit = +90.0°C)

coretemp-isa-0004
Adapter: ISA adapter
Core 0:       +44.0°C  (high = +80.0°C, crit = +90.0°C)
Core 1:       +42.0°C  (high = +80.0°C, crit = +90.0°C)
Core 2:       +44.0°C  (high = +80.0°C, crit = +90.0°C)
Core 3:       +40.0°C  (high = +80.0°C, crit = +90.0°C)

Also I forgot to post earlier, but I installed the virtio drivers for the hard drives. I did the NICs earlier, but for some reason I thought I had done the hdds already.