I’ve done single-GPU passthru in NixOS. It’s not any different than on i.e. Arch, just that you configure some basic setup things such as kernel parameters declaratively using NixOS options.

I highly doubt your issues are NixOS-specific, so I’d recommend you ask about them in the general channels.

I’ve had VFIO passthrough setup on a NixOS host before without issue. While this guide isn’t the one I followed it is an accurate and good guide.