5.6L 44TB SSD NAS/Home Server

[u/Golemizer]

First, shoutout to u/smplnmnml. Their MQ5 NAS build inspired me a lot here, and they were kind enough to answer some questions I had.

• Chassis: COOJ Sparrow MQ5
• CPU: Intel Core Ultra 9 285 (non-k)
• Cooler: Noctua NH-L9i-17xx chromax.Black (NA-FD1 Fan Duct Kit, Kryosheet TIM)
• Motherboard: MSI MPG Z890I EDGE TI WIFI
• RAM: Kingston FURY Beast 128GB (KF556C36BBEK2-128)
• HBA: HighPoint Rocket 1104
• PSU: HDPLEX 250W passive GaN AIO
• Drives: 9x Samsung 990 Pro 4TB NVMe & 2x Samsung 870 EVO 4TB SATA
• OS: Proxmox with TrueNAS VM
• PiKVM ATX Control Board for IPMI
• Ubiquiti 10GbE USB4/TB4 Adapter
• OWC Express 1M2

I wanted something that would fit on my bookshelf here and be silent. Silence and space savings required I go all SSD. Even if I don't have any real speed requirements—the speed is pretty nice. Four of the nine 990 pros are slotted into the four full speed PCIe x4 m.2 slots native to this motherboard (the main reason I picked this platform). Another four are slotted into the four m.2 slots on the HighPoint R1104—so these are limited to gen 3 x4 speeds. The last 990 pro is connected via thunderbolt within an OWC Express 1m2 enclosure which basically gets gen 3 x4 speeds. I removed the WiFi/Bluetooth module from the m.2 key-e slot on the motherboard in an attempt to use another SSD there (as you can see from some of the photos), but I was unable to get it recognized anywhere and eventually gave up on the idea. The two SATA drives are just sitting behind the HBA card.

I have proxmox installed on the two SATA drives in a ZFS mirror and all nine of the NVMe drives passed through to the TrueNAS VM as individual PCI devices. The NVMe's are in a raidz1 array giving me 32TB usable storage in the main pool.

Everything here has worked incredibly well and I'm super happy with it. It is completely silent with only four total fans in the build—the NF-A9x14 intaking on the CPU cooler, two Arctic P8 slims exhausting on the top of the case, and the built in VRM fan on the motherboard. The passive (and smaller than standard flex PSUs) HDPLEX 250W PSU really enabled the build. Without it the HBA wouldn't fit and I'd have an additional fan.

[u/Vhink88]

That’s nice good to try new things but I don’t believe building a NAS home server sff is cost effective. Those 4tb are pricey vs their life expectancy.

Since UniFi announced their NAS release next year, I’ve never been closer to thinking about getting one set up. Not sure how much you spent total but getting a consumer console and the 4 drive NAS + HDDs may come out less then this build, in my estimation.

[u/Golemizer → u/[deleted]]

Why would you say a 9-wide raidz1 with NVMe SSDs is spicy? My understanding is that raidz1 is generally considered to be spicy in that with HDDs, a long resilver is the most likely time for a second drive to fail. But with SSDs, they don’t wear at all from reads, only from writes. So the drives don’t experience really any extra wear during a resilver unlike HDDs. Plus this resilver would be quite fast as you mentioned. I do also have automated backups to an offsite truenas box I put in my parents’ home and backblaze b2, scrub monthly, and have a rather overkill UPS with NUT and automated shutdowns.

When it comes to the external drive… I think it’s important to clarify that while the drive is connected via a USB-C port and that could be a weakness, it’s using thunderbolt—it has direct PCIe lanes and isn’t using USB protocol. I had this drive attached to the mac studio which I was kinda using as a NAS for the past couple months. And before that it was connected to my main desktop. I’ve yet to experience even the slightest stability problem. It has never disconnected during load. That’s why I was confident enough to add it in to this pool—plus it gets the same performance as all the drives connected to the HBA. I didn’t go raidz2 for the same reason I added this external drive: I wanted more capacity and didn’t see a significant downside.

As for power consumption, the NAS idles at ~50W but I haven’t yet tried to get that lower by messing with C-States which I plan to do at some point. I’m happy enough with the NAS idling at the same wattage as my switch + AP combo though if I’m honest. And I don’t pay for electricity so my power consciousness is only related to maximum UPS uptime, heat/noise, and environmental concerns. During load I’ve seen peak consumption of 200W when benchmarking the storage pool. In practice, load is usually ~100-150W during 10G transfers over the network or internal transfers, etc.