I also built a Plex Server (but not as cool as the other thread)

[wagex]

I'm going to do raid 10 across 8 disks. with 4x raid1
poolsize = vdev0+vdev1+vdev2+vdev3

whats good with zfs is that i can run different size vdevs. so i dont have to upgrade ALL drives, i can do 2 at a time.

So i'll start with 8TB across the 4x 2TB raid 1s
so in the future say I can pick up 2x 8TB drives at a good price
pop one drive out of vdev0, put in 8TB, then rebuild (copy, fast, no parity calc)
pop in second 8TB drive, rebuild, expand.

then ill be left with 2+2+2+8 = 14TB. and so on.

say instead i do 2x Raid5 with 8 disks. this needs 2x 4disk raid5 (raidz) vdevs. with both in the pool, it essentially becomes raid 50.
poolsize = 6TB+6TB = 12TB. i have 2 disk redundancy but only if they fail in separate vdevs.

rebuilds are significantly more time consuming calculating parity.
upgrading drives has to be done 4 at a time.
replace drive 1, rebuild whole array
replace drive 2, rebuild whole array
replace drive 3, rebuild whole array
replace drive 4, rebuild whole array
expand.

that a lot of stress on the drives. and that stress increases the probability of failure for the rebuild. then i lose everything.

and the striped 4xmirrors should be faster than the striped 2xraid5.

i dunno i'm still not set in stone but i think i like 10 better.

that's why they say raid is not a backup, you should always backup to an external storage device. it does indeed take a long time to rebuild a raid 5 though, going from 3x2tb to 4x2tb took 3 days, but thanks to my controllers online raid expansion nothing went offline and i was still able to cap my gigabit connection reading from it while it was expanding.

why do you feel the need to do two separate arrays then mirroring them striping, and how are you going to put 2tb drives in the same array with 8tb drives will that even work? i guess i dont understand how zfs works. but iirc any disks added that arent the same size as the others will be gimped down to the same storage capacity of the other drives so puting an 8tb drive in a raid 10 replacing a 2tb drive making it 2+2-2+8 would essentially still be 2+2-2+2 i dont think you can raid 10 an tb vdev with 3 other 4tb vdevs? idk i could be wrong though.

not trying to talk you out of doing what you are doing just trying to understand it.

 

[gsrcrxsi]

with ZFS, total pool size is the sum of your vdevs. they dont have to be identical, but perform better if they are. you can even add a vdev of a single disk if you didnt care about real parity.

so you definitely could have 1 vdev full of a 4-disk raidz with 2 TB disks and a second with 4TB disks. they are "kinda sorta" striped together this way. but from what i gathered it will only really act like raid0 if they are identical.

striping the raids improves performance. thats the only reason really.

and the convenience of only having to upgrade drives 2 at a time. if i build just one big raidz or z2 array, if i want more space, i have to upgrade ALL drives. one at a time, with a rebuild after each swap. that would take forever.

I know raid isnt a replacement for a backup. I will run my current 4-disk synology NAS as my "backup". I havent decided the exact config for it yet. but it will most likely have less total storage, but only store the very important stuff. the things that take up the most space (high def movies) are replaceable via physical optical media or otherwise.

- - - Updated - - -

this article was a main driver for me leaning more towards the "pool of mirrors" raid-10 route

jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

 

[Blaylock]

So plex server config info is stored in the registry under
Computer\HKEY_CURRENT_USER\Software\Plex, Inc.

You could try to remove that and reinstall and see if your server works then?

Just an update. Thank you Custom, cleaning the registry did the trick. So just for clarification or if anyone else runs into this here are the steps.

1. Download the latest Plex media server.
2. Uninstall Plex media server and Plex media player if applicable
3. Delete the Plex, Inc & Plex.tv(if applicable) from the registry path indicated above.
4. Reboot
5. Install latest Plex media server. Follow install directions until completion.

Badaboom Badabing.

 

[gsrcrxsi]

got new board. Supermicro X9DRi-LN4F+ v1.20a (old board was 1.10)
swapped in 2680v2s
board POSTs (YAY)
toss in hard drive with win7 64 installed (from previous board) for stress tests
hard drive wont boot. hmm
go into BIOS check boot and HDD settings. seems ok
get frustrated. put 40GB SSD (freenas install)
boots SSD fine, all my settings are the same, FreeNAS seems to work
get frustrated, still want to check thermals
lets check IPMI
damn, its got some random password set for the ADMIN (used board and all)
try to get back to windows, to reset the IPMI.
screw it, lets re-install
installer says it cant install to the disk because of BIOS settings (AHHHHHHH)
back to BIOS, try every combo of SATA and SCU configs, no dice
google problem. FAQ on supermicro says to try UEFI
tried UEFI, nothing.
reboot again and now the installer is letting me install windows again. dont know what i did but OK.
reinstalled windows to temp HDD, get all the drivers setup.
load up IPMICFG ..... no IPMI device found. huh.
back to google. IPMI not supported on windows 7, only windows server. grrrrr
while I'm here, stress tests. prime 95. max temp 60C on CPU2 with all fans on high.
notice that CPU only running 2.8GHz (should be boosting to 3.1)
back to BIOS. re-enabled steepstep and tweaked some other stupid settings that the previous owner setup
back to stress test, ahh all 20c/40t @3.1GHz, such bliss. max temp 65C on CPU2. good.
google how to reset IPMI again. found a good article to do it with the liveOS from the ubuntu install disc
download disc image, burn, reboot.
pop in disc, load up, a few commands in the terminal and i'm back in business. IPMI unlocked
put 40GB freenas boot SSD back in. wont POST WTFFFFFFF
unplug all peripherals. still wont POST. RAGGGGGEEEEE
google. SM FAQs suggest a ram issue.
powered off, reseated all 12 ram sticks.
reboot, YAY it works. back into freenas.
setup PLEX in a Jail, imported all my media from synology NAS to the FreeNAS drives.
configured the plex server settings, blah blah blah
everything working! even streamed 2x 4k files, with transcoding to various qualities, using about 50% total CPU.

I will say. the worst part of all of this was not all the small problems i ran into. the worst part was doing SO MANY REBOOTS. this thing takes AGES to post and boot. coming from my gaming PC which is at the OS login within about 10s of pressing the power button and this thing takes MINUTES. #firstworldproblems

 

[custom90gt]

Lol sounds like quite the adventure. I guess I can't complain about spending a few days figuring out overclocking the 7920x and the ram in my desktop.
Curious what Cinebench would be on that beast.

Also curious why it was only using 50% of your CPU for 2x 4k streams. Doing 4k HEVC to 1080p h264 will use 90% of my CPU with just one stream. Granted I have plex set to "make my cpu hurt."

 

[gsrcrxsi]

but in other news about the server config itself, here's the rundown

Rack: StarTech 12U 4-post adjustable depth rack
Switch: Cisco SG200-18 18-port gigabit managed switch
PDU: 1U CyberPower 15A 12 socket (6 per side) w/surge protection
Case: Supermicro CSE-835TQ-R920B 3U
MB: Supermicro X9DRi-LN4F+ v1.20a
RAM: 48GB (12x4) 1066MHz DDR3 ECC REG
CPU: 2x E5-2680v2
CPU HSF: 2x Noctua NH-D9L
Case fans: 6x Noctua NF-A8 PWM
PSU: 2x Supermicro PWS-1K28P-SQ 1U 1280w (1000w @ 115v)
OS drive: 40GB Intel 320 series SSD
Storage HBA: LSI 9207-8i
Storage: 4x WD 2TB drives (2 RE4, 2 green) "Raid10". waiting on delivery of 4 more RE drives to add
Backup storage: Synology DS416 4x2TB drives in SHR (raid 5)

Software:
OS: FreeNAS 11.0
Plex Media Server running in a Jail (kinda like a flexible low resource VM)

its crazy. everything is managed by web GUIs now lol.
FreeNAS has it's own IP and web GUI
Synology has it's own IP and web GUI
Plex Server (on the freenas box) has an IP and web GUI
IPMI is managed via it's own IP and web GUI
Cisco switch has it's own IP and web GUI

pretty neat. after it's all setup i don't really have to touch the hardware and can manage it from my couch on my laptop.

- - - Updated - - -

Lol sounds like quite the adventure. I guess I can't complain about spending a few days figuring out overclocking the 7920x and the ram in my desktop.
Curious what Cinebench would be on that beast.

Also curious why it was only using 50% of your CPU for 2x 4k streams. Doing 4k HEVC to 1080p h264 will use 90% of my CPU with just one stream. Granted I have plex set to "make my cpu hurt."

well i have nearly double the CPU cores available. (but a bit less GHz). i forget about that setting. let me see what it's at and if it makes a difference.

 

[gsrcrxsi]

tried it again with the make my cpu hurt setting.

cpu usage for 2 streams went up to a max of about 60% for 2 4k HEVC streams

something i noticed. when transcoding HEVC/H265 files, CPU usage is lower and performance is lower (transcodes are slower). its like it cant properly utilize the whole CPU power. it seems the higher the bitrate, the worse it is. less cpu used.

i took one file that was in HEVC, 4k res, ~68Mbps transcoding this file on my old 1 CPU system would use about 40% CPU. i ran it through Handbrake to re-encode to H264 at the same bitrate. the H264 file transcodes much faster and uses like 90% of the single CPU. that was when it was all on windows. with it now on FreeNAS, it seems to be working a little better. still less CPU usage on HEVC files, but at least it's not buffering

 

[custom90gt]

tried it again with the make my cpu hurt setting.

cpu usage for 2 streams went up to a max of about 60% for 2 4k HEVC streams

something i noticed. when transcoding HEVC/H265 files, CPU usage is lower and performance is lower (transcodes are slower). its like it cant properly utilize the whole CPU power. it seems the higher the bitrate, the worse it is. less cpu used.

i took one file that was in HEVC, 4k res, ~68Mbps transcoding this file on my old 1 CPU system would use about 40% CPU. i ran it through Handbrake to re-encode to H264 at the same bitrate. the H264 file transcodes much faster and uses like 90% of the single CPU. that was when it was all on windows. with it now on FreeNAS, it seems to be working a little better. still less CPU usage on HEVC files, but at least it's not buffering

Interesting. My xeon 18 core had ~80% cpu usage with h265 transcoding a single file if I recall correctly. If you optimize a 4k h265 file around what speed does it go (not very scientific but I'm curious). Just eyeballing mine it stays around 4.8x with my cpu at 80%.

Also I love the hardware btw, it's a lot different than your small setup but it looks like a ton of fun. Do you have a kill-o-watt or anything that you could use to measure power draw from that beast?

 

[gsrcrxsi]

i dont have one, but i can monitor the power consumption from the PSU itself via the IPMI interface.

idle is about 215w
transcoding 1 4k movie is about 300w
transcoding 2 movies was about 375w

one day i'll pop the windows drive back in and see how much it's pulling while running prime95 for an absolute max number.

are you still running the plex server off of your desktop rig?

i'm having an issue with subtitles now. i can transcode a 4k movie with about 35%-40% cpu usage, and thats all fine and good. but once i add subtitles to the mix, the cpu usage drops to 10% and it starts struggling and buffering. can you test that out? make sure you try it with PGS subs as i think PGS is forcing it to be single threaded. only using 1 core of each CPU. it's essentially like trying to transcode a 4k movie on a dual core lol.

 

[custom90gt]

i dont have one, but i can monitor the power consumption from the PSU itself via the IPMI interface.

idle is about 215w
transcoding 1 4k movie is about 300w
transcoding 2 movies was about 375w

one day i'll pop the windows drive back in and see how much it's pulling while running prime95 for an absolute max number.

are you still running the plex server off of your desktop rig?

i'm having an issue with subtitles now. i can transcode a 4k movie with about 35%-40% cpu usage, and thats all fine and good. but once i add subtitles to the mix, the cpu usage drops to 10% and it starts struggling and buffering. can you test that out? make sure you try it with PGS subs as i think PGS is forcing it to be single threaded. only using 1 core of each CPU. it's essentially like trying to transcode a 4k movie on a dual core lol.

Yeah my desktop is now my server and gaming rig all rolled into one. It saved me a good amount of money to sell the server and upgrade the desktop.

I will try with subs tonight if I remember, I'll have to see if I have anything with PGS subs.

 

[gsrcrxsi]

here's some more data and benches about the rig.

max power consumption while running P95 was about 470W (IPMI reading)

heres PassMark and Cinebench R15 Scores:






Here's my gaming rig in my sig for comparison

 

[wagex]

idk what kind of router you are running but with all those management url's it would be neat to redirect something like admin.com to a local web page with links to each of the management pages to keep easier track of them.

 

[gsrcrxsi]

i just bookmarked them to a folder on Firefox.

 

[wagex]

i just bookmarked them to a folder on Firefox.

that works too lol, just figured since you are already going overboard might as well make a nice ui for all your management

 

[gsrcrxsi]

small update.

server has been working fine!

gave it a 16GB ram upgrade. i wanted a nice clean 64GB.
i also installed my LSI 9207-8i HBA. plugged into the SM backplane and its working great.
and got my remaining 4 2TB drives from cw. the ZFS pool is now setup with 4x mirrors. total space 8TB. as drives fail or i can get screaming deals on larger capacity, i'll replace them. but i'm content for now.