Is it safe to run zone-wide NFS Primary Storage on the Management Server node itself?

[tuanhoangth1603]

Hi everyone,
We are running Apache CloudStack 4.22 with Ceph RBD as the main block storage for VM disks.
For simplicity, we have been hosting the zone-wide NFS Primary Storage share directly on the Management Server node itself (like /export/primary).
My question is very straightforward:
If the Management Server node goes down (reboot, crash, power loss, etc.) or simply the NFS service stops or becomes unreachable, what exactly happens to the whole zone?
From our previous painful experience (when we accidentally put NFS Primary on a compute node and took it offline), the entire cluster froze because every KVM host has that NFS share mounted with hard,intr options.
Will the same thing happen if the NFS share lives on the Management node?
Will all compute nodes hang? Will running VMs (with disks on Ceph) freeze? Will System VMs die?
We want to understand the exact failure mode before deciding whether to keep this setup or move the NFS share to a dedicated/HA server.
Thank you in advance for any clarification!

[jgotteswinter]

Expect that if the NFS server goes down and one of its shares is somehow in use (an idle mount might survive) this will cause hanging tasks. If this is productive, the NFS should be definitely on a dedicated machine. HA NFS needs careful testing.

[tuanhoangth1603]

Thanks for you answer @jgotteswinter. If that's the case, while we're trying to HA everything, isn't the NFS primary storage still a single point of failure?

[jgotteswinter]

ACS just has the option to use a NFS, its not the scope of ACS to provide HA for this. There are multiple options for HA NFS. Like DYI Linux/BSD/TrueNas/zVault setups. HA can be archived for example with Pacemaker or the commercial RSF-1 Cluster. These solutions need careful testing. Or ready-to-use boxes like a Netapp, which will mostly work out of the box.

So, yes. It could be a spof depending on the implementation.

[tuanhoangth1603]

Thank you.
One final question: is at least one primary NFS storage required for the ACS cluster (or pod/zone) to operate in a healthy state?

[jgotteswinter]

If you got Ceph as primary storage, thats sufficient. You need at least 1 primary storage, thats the only requirement.