SmartOS on AMD Ryzen with ECC RAM

This blog post summarizes my ongoing efforts to build a relatively robust rack mountable SmartOS system using an AMD Ryzen processor with ECC RAM and NVMe storage.

I will append chronologically until I'm satisfied, then rewrite to put the summary at the top.

Here we go:

Specs:

• AMD Ryzen 9 3900X 12-core, 24-thread unlocked desktop processor with Wraith Prism LED Cooler
• 128GB of Unregistered ECC DDR4 RAM using 4 x Crucial/Micron MTA18ASF4G72AZ-3G2B1 32GB EUDIMM (note, must be unregistered ECC which is far less common than typical server registered ECC RAM. I linked to ShopBLT which is the only place I could find this part number for sale.)
  
• ASRock X570 Taichi motherboard
  
• VisionTek Radeon 5450 1GB DDR3 (Dvi-I, HDMI, VGA) Graphics Card for console
  
• FSP Group Twins ATX PS2 1+1 Dual Module 700W 80 Plus Gold Hot Swappable Redundant Digital Power Supply
• Chenbro Rackmount 4U Server Chassis RM41300-FS81

Notes:

• You need to use unregistered ECC RAM if you want the system to even make it to the BIOS screen.
  
• As of BIOS version 3.40, the ASRock X570 Taichi board still needs Compatibility Mode turned on for booting SmartOS successfully and if using all four DIMM slots, the memory clock frequency may need to be switched from Auto to 2666 MHz so that the motherboard doesn't try to run it at max speed. This can cause ECC correctable errors and can also cause SmartOS to report a faulty CPU cache and disable one of the virtual processors at boot time.
• I also possess but have not yet tested an ASUS AMD AM4 Pro WS X570-Ace ATX Workstation Motherboard. In theory it should also work but neither ASUS nor ASRock have actually tested their boards with 128GB of unregistered ECC despite claiming support for both ECC and 128GB. There may be other boards out there but these seem to be the ones people point to if you want the X570 chipset for PCIe 4.0

I currently have SmartOS happily booting with the above specs and using a single Samsung 860 EVO 4TB SATA SSD for zones pool. I have recently put a pair of Sabrent 2TB Rocket NVMe 4.0 drives in the system and while both appear in the BIOS, only one is showing up in SmartOS when I boot the installer or when booting to my zones pool on the Samsung 860. That is the next problem to solve.