Hi all!
I will soon acquire a pretty beefy unit compared to my current setup (3 node server with each 16C, 512G RAM and 32T Storage).
Currently I run TrueNAS and Proxmox on bare metal and most of my storage is made available to apps via SSHFS or NFS.
I recently started looking for “modern” distributed filesystems and found some interesting S3-like/compatible projects.
To name a few:
- MinIO
- SeaweedFS
- Garage
- GlusterFS
I like the idea of abstracting the filesystem to allow me to move data around, play with redundancy and balancing, etc.
My most important services are:
- Plex (Media management/sharing)
- Stash (Like Plex 🙃)
- Nextcloud
- Caddy with Adguard Home and Unbound DNS
- Most of the Arr suite
- Git, Wiki, File/Link sharing services
As you can see, a lot of download/streaming/torrenting of files accross services. Smaller services are on a Docker VM on Proxmox.
Currently the setup is messy due to the organic evolution of my setup, but since I will upgrade on brand new metal, I was looking for suggestions on the pillars.
So far, I am considering installing a Proxmox cluster with the 3 nodes and host VMs for the heavy stuff and a Docker VM.
How do you see the file storage portion? Should I try a full/partial plunge info S3-compatible object storage? What architecture/tech would be interesting to experiment with?
Or should I stick with tried-and-true, boring solutions like NFS Shares?
Thank you for your suggestions!
If you want to try something that’s quite new and mostly unexplored, look into NVMe over TCP. I really like the concept, but it appears to be too new to be production ready. Might be a good fit for your adventurous endeavors.
This is just block device over network, it will not allow the use cases OP is asking for. You will still need a filesystem and a file-serving service on top of that.
I agree, but it’s clear that OP doesn’t want a real solution, because those apparently are boring. Instead, they want to try something new. NVMe/TCP is something new. And it still allows for having VMs on one system and storage on another, so it’s not entirely off topic.