Cheap 10gb LAN

[Nimblor]

I'm looking for advice and assistance on a 10gb LAN plan. I want to connect my unraid server and windows PC and have room for a future NVR where I might back up to the unraid server. I'm thinking Mellonix 10gb nic for the server and PC. These will fit in open pcie 3.0 x4 slots in each. These are older xcat 3 models that seem available and cheap and the x 4 slot is all that is available in the server.

Plan is to run lc fiber between the PC and one of those 4 port sfp+ switches from Mikrotik through a wall lc keystone connector so it looks nice. Then switch to the server with 10 gb dac. Then a 1gb dac from the switch to the router which has a 1gb sfp port. This is leaving a port open for the future NVR on the switch.

I have less than 1 gb Internet. And my router is a Mikrotik 3011. Cheapness is a priority.

Am I missing anything here? Should this work and improve my transfer speeds between the server and my PC?

All input is appreciated.

 

[infinitevalence]

Good Plan!

My suggestions:

Mcx312b-Xcct for sale | eBay

Get the best deals for Mcx312b-Xcct at eBay.com. We have a great online selection at the lowest prices with Fast & Free shipping on many items!

www.ebay.com

These are getting long in the tooth but are still well supported in Linux. You can pick up the 4 version for not much more if you feel like it.

For cheep switches you have kinda 3 teirs, Aliexpress, Mikrotik, TP-LINK/UNIFI

https://www.aliexpress.us/item/3256806038981360.html

https://www.aliexpress.us/item/3256805928917037.html

Mikrotik is a good tinker's switch as it has router and powerful layer 3 tools but it is not what I would call easy.

https://mikrotik.com/product/crs310_8g_2s_in

Unifi and TP-Link offer you a higher level of support than Mikrotik and easy to use GUI's, but at a cost.

Switch Pro 8 PoE - Ubiquiti Store

An 8-port, Layer 3 switch with PoE+ and PoE++ output.

store.ui.com

Omada 8-Port 2.5G L2+ Managed Switch

Omada 8-Port 2.5G L2+ Managed Switch

www.tp-link.com

As for connecting, I would use DACs (Direct attach cable) to and from the switch to server, and only use fiber if you need to extend beyond 15ft/3m from your switch.

FS.com - HPC, Data Center, Enterprise, Telecom

FS is a global technology company focused on high-speed network system solutions, providing product R&D, solution design and test, global warehouse and more.

www.fs.com

 

[Nimblor]

Thanks for the feedback! I was able to find a Mikrotik switch and some NICs for a good price. They came with a 2M DAC which is nice. Mikrotik routerOS is a learning curve but I should be able to put it into switch mode without too much trouble.

I am going to have to use fiber for one of the runs. I am not excited about having a fiber run along the baseboard of my room to get to my PC but I am also trying to minimize holes in the walls.

Have you had any experience with LC-LC keystone connectors?

 

[infinitevalence]

They are usually fine you just don't want to use to many as they add to the total signal loss.

 

[Nimblor]

It looks like the longest run will be 27 m plus the connector. The modules I got are good for 300 m. Maybe I should post some before and after pictures. Idk if anyone is intersted in this stuff.

 

[Nimblor]

I got it working. 10G is fast. I am seeing 1.08 G/s transfer speeds with the server. Previously it was 115M or so. For reference, now Win11 iso moves in 5 sec vs about a minute.

Here are a few tips. The cheapo Mellanox Connectx-3 MCX311A have some problems for PCs. They are perfect for linux based systems. My server is unraid and none of these issues applied to my server.

There is no way to update the Mellanox firmware on PC without booting linux. I do not know if this is even necessary and may have inadvertently caused problems. IDK, I like current firmware. The windows 10+ drivers do not work well. I had to disable CPU-E cores to make the card work. Not ideal but it is working. I ordered a different 10G sfp+ card for my PC. I may have been over emphasizing the need for a PCIE x4 card. I was trying to use a non- graphics card interface and I only need one SFP+ plus port. There should be plenty of bandwidth in a x4 electrical slot with an x8 card and that is what I am trying next. I have ranted previously about low numbers of PCIE lanes in modern motherboards.

Some research here would have saved a headache. The headache entailed switching the cards in the server after updating them and troubleshooting the issue with the card in the PC. It seems to only like the graphics card slot. That is where I got it working anyway. The drivers caused BSOD bootloop which I think is everyone's favorite.

SFP+ modules are like USB-A, they only fit one way and you will push them in wrong every time the first time. The connection feel is completely erotic. Similar with the fiber to module connection.

I am a huge fan of Mikrotik but there is a learning curve there. There are an unlimited number of options. It took a bit to get things set right but everything is perfect now. I have cat6a working which is fine for now between the switch and the router and will be trying out the 1G DAC (1.25G) when the server is not in use.

By happenstance, the lone POE port on my router and switch hooked up in an unplanned way. Seeing the switch just fire up when I connected the ethernet threw me for a loop.

Parts - $298 inc shipping

Fiber keystone connectors -$11 - bc I am bougie
2 MCX311A cards and10G Dac - $53 - package deal
Mikrotik CRS305-1G-4S+IN - $122 - ebay deal
LC to LC OM3 20M - $18 - lots of options for these
LC to LC OM3 6M - $13 - same here
2 10GBase-SR SFP+ LC Transceivers - $34 - could have gotten cheaper modules but I wanted new
0.5 M DAC - $13
Intel 10G SFP nic - $34 - overpaid for this but I wanted it nowz

I could have done this cheaper with a direct Peer to Peer connection with the server, or maybe by using existing cat6A wiring with rj45 SFP+ modules, but I went the switch route for expandability in the future.

Thanks for helping me with this project!

 

[infinitevalence]

I have never tried MCX3's on windows so sorry I could not have warned you about a challenge there.

Nice to see the performance meeting expectations!

 

[Nimblor]

Its not a big deal. Hopefully my endeavor is helpful to others. Maybe someone here will buy the MCX3.

 

[infinitevalence]

Its not a big deal. Hopefully my endeavor is helpful to others. Maybe someone here will buy the MCX3.

o i will, i always need more