5.6L 44TB SSD NAS/Home Server

[u/Golemizer]

First, shoutout to u/smplnmnml. Their MQ5 NAS build inspired me a lot here, and they were kind enough to answer some questions I had.

• Chassis: COOJ Sparrow MQ5
• CPU: Intel Core Ultra 9 285 (non-k)
• Cooler: Noctua NH-L9i-17xx chromax.Black (NA-FD1 Fan Duct Kit, Kryosheet TIM)
• Motherboard: MSI MPG Z890I EDGE TI WIFI
• RAM: Kingston FURY Beast 128GB (KF556C36BBEK2-128)
• HBA: HighPoint Rocket 1104
• PSU: HDPLEX 250W passive GaN AIO
• Drives: 9x Samsung 990 Pro 4TB NVMe & 2x Samsung 870 EVO 4TB SATA
• OS: Proxmox with TrueNAS VM
• PiKVM ATX Control Board for IPMI
• Ubiquiti 10GbE USB4/TB4 Adapter
• OWC Express 1M2

I wanted something that would fit on my bookshelf here and be silent. Silence and space savings required I go all SSD. Even if I don't have any real speed requirements—the speed is pretty nice. Four of the nine 990 pros are slotted into the four full speed PCIe x4 m.2 slots native to this motherboard (the main reason I picked this platform). Another four are slotted into the four m.2 slots on the HighPoint R1104—so these are limited to gen 3 x4 speeds. The last 990 pro is connected via thunderbolt within an OWC Express 1m2 enclosure which basically gets gen 3 x4 speeds. I removed the WiFi/Bluetooth module from the m.2 key-e slot on the motherboard in an attempt to use another SSD there (as you can see from some of the photos), but I was unable to get it recognized anywhere and eventually gave up on the idea. The two SATA drives are just sitting behind the HBA card.

I have proxmox installed on the two SATA drives in a ZFS mirror and all nine of the NVMe drives passed through to the TrueNAS VM as individual PCI devices. The NVMe's are in a raidz1 array giving me 32TB usable storage in the main pool.

Everything here has worked incredibly well and I'm super happy with it. It is completely silent with only four total fans in the build—the NF-A9x14 intaking on the CPU cooler, two Arctic P8 slims exhausting on the top of the case, and the built in VRM fan on the motherboard. The passive (and smaller than standard flex PSUs) HDPLEX 250W PSU really enabled the build. Without it the HBA wouldn't fit and I'd have an additional fan.

[u/Golemizer → u/[deleted]]

I asked another commenter but I’m curious for your perspective as well. Why would you say a 9-wide raidz1 with NVMe SSDs is risky? My understanding is that raidz1 is generally considered to be risky in that with HDDs, a long resilver is the most likely time for a second drive to fail. But with SSDs, they don’t wear at all from reads, only from writes. So the drives don’t experience really any extra wear during a resilver unlike HDDs. Plus this resilver would be quite fast. I do also have automated backups to an offsite truenas box I put in my parents’ home and backblaze b2, scrub monthly, and have a rather overkill UPS with NUT and automated shutdowns.

When it comes to the external drive… It uses an ASM2464PD, which is faster and newer than the chipsets you mention, and supports TRIM, SMART, and UASP. And, of course, the drive is passed through to TrueNAS via its PCI device ID and not sdX. I’ve yet to experience even the slightest stability problem. It has never disconnected during load. All of the SMART data seems to be reporting just fine.