- Joined
- Oct 25, 2001
This post is a continuity bonus to this thread.
As some of you are already aware, I have been in the process of building a new server for the clan I game with to use install in our host's colocation facility.
I've been at this on and off for a little over a month now. Most of the time has been spent waiting for parts to arrive, and about two weeks were wasted because I changed my mind about what motherboard I was going to use.
Finally, this week, I finished construction of the server. Here are the specs:
The parts all rolled in almost one at a time because I ordered from a bunch of different suppliers (mostly newegg.com and axiontech.com) so I had to let them sit before beginning construction. When I finally did, I had my camera handy. It's a Canon Powershot G3 if you're curious but I downsized the pics in Photoshop afterward so I doubt it matters. Sorry for some of them being blurry - shiny stuff messes up the AF lens.
Here is the CPU horsepower under the hood. Two AMD Opteron 64-bit processors. These are Opteron 246 chips, which run at a core clock of 2.0ghz. Each one features a dedicated, internal memory controller that runs at CPU clock speed, giving insane memory latency since there's no motherboard chipset to go through on the way to the RAM:
Since each CPU has the equivalent of half a motherboard chipset integrated into it in the form of an on-die memory controller, they use a lot of pins. 940 pins to be exact:
Whoa, that's a lot of pins. It goes in this socket:
Next step was to put the CPUs into the motherboard and attach the coolers to them. The next picture shows one of the Opterons in the socket for CPU0, while the other socket remains empty. The top of the CPU is white because I had already spread thermal paste on top of it in preparation for mounting the HSF on it.
You can kind of see how huge the Thunder K8W is for a motherboard in that picture.
The next picture shows the HSF mounted on CPU0 and the second CPU in the socket for CPU1. No thermal paste on CPU1 as of yet.
Here's the HSF mounted on CPU1:
Next step: RAM. For memory I used four 512mb modules of Corsair PC3200. Opterons require Registered RAM, and most Registered RAM is also ECC-capable since it tends to be used only in servers. Corsair makes excellent high-performance RAM, and they've recently begun manufacturing Registered RAM for high-end Athlon FX systems. The PC3200 that I used has a JEDEC rating of 6-2-3-2 at DDR400 speeds. If you don't know what that means then it can also be said to be fast and danger.
OH NOS, THEY'RE MULTIPLYING!@1
And hey, here's the motherboard populated with CPUs and RAM!
There are two sets of memory slots - one for each CPU. That's because this motherboard supports NUMA - Non-Uniform Memory Access. Since each CPU has its own memory controller, that means that if there is more than one CPU, you should be able to set it up so that each CPU has its own share of memory to talk to. NUMA allows this, and nothing else does. This way, if we have BFV running on one CPU and DC running on the other, each game will be isolated to its own memory. The normal memory bottleneck within the motherboard chipset is completely eliminated, while simultaneously memory bandwidth is doubled. NUMA kicks ***. Unfortunately only 64-bit OSes are NUMA-aware, and they're not out in force as of yet. But in the future, we'll upgrade to Windows XP-64 or Server 2003's 64-bit rendition (please correct me if I'm wrong about this, software is not my forte) and we'll get a very substantial performance boost. Remember too that each Opteron CPU's on-die memory controller is dual-channel DDR400, so it can address each stick of RAM at the same time within its own bank. Also, if one CPU is running a task that uses more than one gig of RAM, it goes out over the hypertransport tying the two processors together, through the other CPU's memory controller, and into the other half of the RAM. So, there's still 2 gigs of RAM there, it's just split between both CPUs.
Anyway.
Now that the motherboard is was full, it was time to put it in the case. But first, here are the other notable parts I used:
Those are 74 gig Western Digital Raptors. The Raptor is the first non-SCSI hard drive with a spindle speed above 7,200 RPM. Raptors turn at 10,000 RPM, which means faster seek times and higher data throughput. Seek time is what represents a majority of drive performance in most applications, especially with game servers. I used two 74 gig drives because they will be mirrored in RAID 1 using the Thunder K8W's onboard Silicon Image RAID controller. Mirroring means that each disk holds the exact same data as the other one so that if one undergoes a catastrophic failure, the server keeps running. It also means that the total array capacity is only 74 gigs, since one drive is used as the mirror, but 74 gigs is plenty for our purposes and I don't anticipate a need to expand in the foreseeable future.
That said, I didn't really take any more pictures until I had the system completely assembled because most of the rest of the build was just mundane generic stuff.
So, now, I present to you, the finished product:
There she is. I know it's kind of washed out because of the flash, so I took another one <i>sans</i> flash:
There's no floppy drive, as you can see. I don't see a need for one with future operating systems, although I do have one that I may add if necessary. I'll probably put it in before delivering it to ILAN.
The topmost bay holds a Digidoc for thermal monitoring. The bottommost 5.25" bay holds the CD-ROM. The two bays above that hold the hard drives. They have their own internal blowers for cooling and the digital display is the drive temperature (in Celsius there) read by a small probe attached to the bottom of the drive near the spindle motor. You can set them to alarm when the drive hits a certain temperature but I haven't messed with that. Aside from looking kickin' rad, I got them for another reason.
The Raptors are standard-sized 3.5" hard drives, but the motherboard was so big that I couldn't fit it in the case with the stock internal drive cages in it because the HSF for CPU0 stuck up too high. So what I did was I used the drive coolers to relocate the hard drives up into the 5.25" bays. I could do this because they use Serial ATA, and SATA allows cable lengths up to 48 inches. Old school Parallel ATA have an electrical cable length limit of 18 inches because parallel interfaces are prone to inductance (crosstalk) at high speeds.
After relocating the hard drives I drilled out the rivets for the drive cages and popped them out, which gave me enough room to fit the motherboard in the case.
You can see the light coming out from the window in the side. The window came in the case already, and I didn't look for one, but since it had one, I figured I'd stick a pair of white cold cathode fluorescent lamps inside the case to function as work lights so you can see inside. Also, they look kind of cool.
That is a cooling fan in the door.
The other angle is the rear angle. This may also be referred to as the RC angle. You can use the two nomenclatures interchangeably.
Note the absence of fan grills on the outside. The case had them, but they were those holes that had been stamped out of the steel. That means flat surfaces and jagged edges. Lots of air resistance and more noise. So I took my Dremel with the cutting wheel and sliced them out. I had to grind down the stubs left behind but I'll probably give it a touch-up since there are some burrs still left on there.
The case fans, by the way, are awesome. Comair-Rotron Flight II DC blowers are about twice (or more) the cost of regular 80x25mm case fans but they are built a lot better. They have double ball bearings, feathered blades, are manufactured to tighter tolerances, and weigh about twice as much as the el-cheapo fans that came with the case do.
Here is the inside of the case without the door in the way. It's nice and bright inside when the system is running. Too bad the CCFL inverter needs 12 volts or I'd have run it from the +5v standby line of the PSU, so the lamps could be on when the system is off.
That's the PCI graphics card in the bottom slot. I didn't even bother with an AGP card since this will only be a server, and everything else we need (SATA, Gigabit LAN, etc.) was integrated into the motherboard. The blue box on the floor of the case is the inverter for the CCFL lighting. The two thin red cables are the SATA cables going up to the hard drives, and the metal-sheathed round cable is the ATA cable for the CD-ROM. I did my best to zip-tie all the cabling and move it out of the way of the airflow. You can see how the air has a straight shot from front intake fans to rear exhaust fans with no impedements. You can also see how big the Thunder K8W is. Keep in mind this is a server case designed to accomodate WATX motherboards, and even so, I had to pop rivets and cut metal in order to get the motherboard to fit snugly - and it's really snug.
The lights are tucked away on the sides:
That sums it up. It's a pretty basic machine because it doesn't need stuff like a fancy graphics adapter, sound card, DVD burner, or any of that other stuff desktops do well with. Aside from all the work I had to do modding the case to fit the Thunder K8W the construction was quick and easy.
At the moment I'm running XP Pro on it, since it's the only OS I have right now. I downloaded a beta version of XP-64 the other day but haven't messed with it yet. I'd like to get a 64-bit OS on it ASAP before taking it to the colo facility so we can get the benefits of NUMA. Does anybody have a suggestion I could look into? Or is XP-64 coming out relatively soon? Again, I'm not superb at software, so you'll have to forgive my ignorance.
A lot of you guys helped me with the parts for this so I owe you my thanks: DaveB, diehrd, CPFitz14, Audioaficionado, cmcquistion, zachj, and anybody else who lent their advice to this thread can hit me up for a beer anytime.
Edit: sentence structure
As some of you are already aware, I have been in the process of building a new server for the clan I game with to use install in our host's colocation facility.
I've been at this on and off for a little over a month now. Most of the time has been spent waiting for parts to arrive, and about two weeks were wasted because I changed my mind about what motherboard I was going to use.
Finally, this week, I finished construction of the server. Here are the specs:
The parts all rolled in almost one at a time because I ordered from a bunch of different suppliers (mostly newegg.com and axiontech.com) so I had to let them sit before beginning construction. When I finally did, I had my camera handy. It's a Canon Powershot G3 if you're curious but I downsized the pics in Photoshop afterward so I doubt it matters. Sorry for some of them being blurry - shiny stuff messes up the AF lens.
Here is the CPU horsepower under the hood. Two AMD Opteron 64-bit processors. These are Opteron 246 chips, which run at a core clock of 2.0ghz. Each one features a dedicated, internal memory controller that runs at CPU clock speed, giving insane memory latency since there's no motherboard chipset to go through on the way to the RAM:
Since each CPU has the equivalent of half a motherboard chipset integrated into it in the form of an on-die memory controller, they use a lot of pins. 940 pins to be exact:
Whoa, that's a lot of pins. It goes in this socket:
Next step was to put the CPUs into the motherboard and attach the coolers to them. The next picture shows one of the Opterons in the socket for CPU0, while the other socket remains empty. The top of the CPU is white because I had already spread thermal paste on top of it in preparation for mounting the HSF on it.
You can kind of see how huge the Thunder K8W is for a motherboard in that picture.
The next picture shows the HSF mounted on CPU0 and the second CPU in the socket for CPU1. No thermal paste on CPU1 as of yet.
Here's the HSF mounted on CPU1:
Next step: RAM. For memory I used four 512mb modules of Corsair PC3200. Opterons require Registered RAM, and most Registered RAM is also ECC-capable since it tends to be used only in servers. Corsair makes excellent high-performance RAM, and they've recently begun manufacturing Registered RAM for high-end Athlon FX systems. The PC3200 that I used has a JEDEC rating of 6-2-3-2 at DDR400 speeds. If you don't know what that means then it can also be said to be fast and danger.
OH NOS, THEY'RE MULTIPLYING!@1
And hey, here's the motherboard populated with CPUs and RAM!
There are two sets of memory slots - one for each CPU. That's because this motherboard supports NUMA - Non-Uniform Memory Access. Since each CPU has its own memory controller, that means that if there is more than one CPU, you should be able to set it up so that each CPU has its own share of memory to talk to. NUMA allows this, and nothing else does. This way, if we have BFV running on one CPU and DC running on the other, each game will be isolated to its own memory. The normal memory bottleneck within the motherboard chipset is completely eliminated, while simultaneously memory bandwidth is doubled. NUMA kicks ***. Unfortunately only 64-bit OSes are NUMA-aware, and they're not out in force as of yet. But in the future, we'll upgrade to Windows XP-64 or Server 2003's 64-bit rendition (please correct me if I'm wrong about this, software is not my forte) and we'll get a very substantial performance boost. Remember too that each Opteron CPU's on-die memory controller is dual-channel DDR400, so it can address each stick of RAM at the same time within its own bank. Also, if one CPU is running a task that uses more than one gig of RAM, it goes out over the hypertransport tying the two processors together, through the other CPU's memory controller, and into the other half of the RAM. So, there's still 2 gigs of RAM there, it's just split between both CPUs.
Anyway.
Now that the motherboard is was full, it was time to put it in the case. But first, here are the other notable parts I used:
Those are 74 gig Western Digital Raptors. The Raptor is the first non-SCSI hard drive with a spindle speed above 7,200 RPM. Raptors turn at 10,000 RPM, which means faster seek times and higher data throughput. Seek time is what represents a majority of drive performance in most applications, especially with game servers. I used two 74 gig drives because they will be mirrored in RAID 1 using the Thunder K8W's onboard Silicon Image RAID controller. Mirroring means that each disk holds the exact same data as the other one so that if one undergoes a catastrophic failure, the server keeps running. It also means that the total array capacity is only 74 gigs, since one drive is used as the mirror, but 74 gigs is plenty for our purposes and I don't anticipate a need to expand in the foreseeable future.
That said, I didn't really take any more pictures until I had the system completely assembled because most of the rest of the build was just mundane generic stuff.
So, now, I present to you, the finished product:
There she is. I know it's kind of washed out because of the flash, so I took another one <i>sans</i> flash:
There's no floppy drive, as you can see. I don't see a need for one with future operating systems, although I do have one that I may add if necessary. I'll probably put it in before delivering it to ILAN.
The topmost bay holds a Digidoc for thermal monitoring. The bottommost 5.25" bay holds the CD-ROM. The two bays above that hold the hard drives. They have their own internal blowers for cooling and the digital display is the drive temperature (in Celsius there) read by a small probe attached to the bottom of the drive near the spindle motor. You can set them to alarm when the drive hits a certain temperature but I haven't messed with that. Aside from looking kickin' rad, I got them for another reason.
The Raptors are standard-sized 3.5" hard drives, but the motherboard was so big that I couldn't fit it in the case with the stock internal drive cages in it because the HSF for CPU0 stuck up too high. So what I did was I used the drive coolers to relocate the hard drives up into the 5.25" bays. I could do this because they use Serial ATA, and SATA allows cable lengths up to 48 inches. Old school Parallel ATA have an electrical cable length limit of 18 inches because parallel interfaces are prone to inductance (crosstalk) at high speeds.
After relocating the hard drives I drilled out the rivets for the drive cages and popped them out, which gave me enough room to fit the motherboard in the case.
You can see the light coming out from the window in the side. The window came in the case already, and I didn't look for one, but since it had one, I figured I'd stick a pair of white cold cathode fluorescent lamps inside the case to function as work lights so you can see inside. Also, they look kind of cool.
That is a cooling fan in the door.
The other angle is the rear angle. This may also be referred to as the RC angle. You can use the two nomenclatures interchangeably.
Note the absence of fan grills on the outside. The case had them, but they were those holes that had been stamped out of the steel. That means flat surfaces and jagged edges. Lots of air resistance and more noise. So I took my Dremel with the cutting wheel and sliced them out. I had to grind down the stubs left behind but I'll probably give it a touch-up since there are some burrs still left on there.
The case fans, by the way, are awesome. Comair-Rotron Flight II DC blowers are about twice (or more) the cost of regular 80x25mm case fans but they are built a lot better. They have double ball bearings, feathered blades, are manufactured to tighter tolerances, and weigh about twice as much as the el-cheapo fans that came with the case do.
Here is the inside of the case without the door in the way. It's nice and bright inside when the system is running. Too bad the CCFL inverter needs 12 volts or I'd have run it from the +5v standby line of the PSU, so the lamps could be on when the system is off.
That's the PCI graphics card in the bottom slot. I didn't even bother with an AGP card since this will only be a server, and everything else we need (SATA, Gigabit LAN, etc.) was integrated into the motherboard. The blue box on the floor of the case is the inverter for the CCFL lighting. The two thin red cables are the SATA cables going up to the hard drives, and the metal-sheathed round cable is the ATA cable for the CD-ROM. I did my best to zip-tie all the cabling and move it out of the way of the airflow. You can see how the air has a straight shot from front intake fans to rear exhaust fans with no impedements. You can also see how big the Thunder K8W is. Keep in mind this is a server case designed to accomodate WATX motherboards, and even so, I had to pop rivets and cut metal in order to get the motherboard to fit snugly - and it's really snug.
The lights are tucked away on the sides:
That sums it up. It's a pretty basic machine because it doesn't need stuff like a fancy graphics adapter, sound card, DVD burner, or any of that other stuff desktops do well with. Aside from all the work I had to do modding the case to fit the Thunder K8W the construction was quick and easy.
At the moment I'm running XP Pro on it, since it's the only OS I have right now. I downloaded a beta version of XP-64 the other day but haven't messed with it yet. I'd like to get a 64-bit OS on it ASAP before taking it to the colo facility so we can get the benefits of NUMA. Does anybody have a suggestion I could look into? Or is XP-64 coming out relatively soon? Again, I'm not superb at software, so you'll have to forgive my ignorance.
A lot of you guys helped me with the parts for this so I owe you my thanks: DaveB, diehrd, CPFitz14, Audioaficionado, cmcquistion, zachj, and anybody else who lent their advice to this thread can hit me up for a beer anytime.
Edit: sentence structure
Last edited:
