Honestly, taking the time learn Docker and then learn more about the specific containers that you want to use is probably going to be the easiest way forward in your position. If you have any specific questions about Docker or the containers you’re looking at, I can try to help.

When it comes to network mounts, I’ve found it a lot easier to use rclone for that purpose, and that’s currently what I use for the backend of my Plex server.

This website has all the infos you need. You probably won’t get anymore spoonfed than this, good luck :)

I installed CasaOs on Debian with a one line command on a laptop with a brocken screen, then from another laptop on my network I installed jellyfin within Casa Os app store, I plugged my multimedia hard drive, now everyone at home can watch movies and shows from their device

1. Disregard Docker. You’ve got NixOS, you don’t need Docker. Thank god.
2. Configure the services:

{config, pkgs, ...}: {
  # Jellyfin
  services.jellyfin.enable = true;
  
  # enable the other *arrs, whichever you want to use
  services.sonarr.enable = true;

  # qbittorrent user and service
  users = {
    groups.torrent = {
      # put users that should be allowed to access torrented media
      members = [config.services.jellyfin.user "you"];
    };

    users.torrent = {
      isSystemUser = true;
      description = "qbittorrent user";
      group = "torrent";
      createHome = true;
      home = "/var/lib/torrent";
    };
  };
  
  systemd.services.qbittorrent = let
    qbittorrent = pkgs.qbittorrent.override {guiSupport = false;};
  in {
    enable = true;
    description = "qbittorrent daemon";
    documentation = ["man:qbittorrent-nox(1)"];
    wants = ["network-online.target"];
    after = ["network-online.target" "nss-lookup.target"];
    wantedBy = ["multi-user.target"];
    serviceConfig = {
      ExecStart = "${qbittorrent}/bin/qbittorrent-nox";
      User = "torrent";
    };
  };
  
  # VPN configuration
  networking.wg-quick.interfaces = {
    mullvad = {
      # Insert options for Mullvad
      address = [...];
      dns = [...];
      peers = [
        {
          publicKey = "...";
          allowedIPs = ["0.0.0.0/0" "::0/0"];
          endpoint = "...";
        }
      ];
    };
  };
  
  # file server, SMB unfortunately works the best for all the operating systems
  services.samba = {
    enable = true;
    shares = {
      storage = {
        # where do you store your stuff?
        path = "/path/to/linux/ISOs";
        browseable = "yes";
        "read only" = "no";
        "guest ok" = "yes";
        "create mask" = "0644";
        "directory mask" = "0755";
      };
    };
    extraConfig = ''
      workgroup = WORKGROUP
      server string = ${config.networking.hostName}
      netbios name = ${config.networking.hostName}

      guest account = nobody
      map to guest = bad user

      # No printers
      load printers = no
      printing = bsd
      printcap name = /dev/null
      disable spoolss = yes
      show add printer wizard = no

      dos charset = CP850
      unix charset = UTF-8
      unix extensions = yes
      ; mangled names = no
      map archive = no
      map system = no
      map hidden = no
    '';
  };
}

This is a minimal config that doesn’t set up specific stuff like qbittorrent’s file storage location or network interface, I’d tell you how to do it but I don’t actually have such a setup. This is just copied from what I have/had in my configuration and looking up services on https://search.nixos.org (very useful site if you don’t know about it).

I think you bit a big mouthful if you’re just starting out on the NAS game. I would suggest breaking things down into smaller pieces:

1. Prepare a standalone container only with the VPN.
2. Try to set up a torrent client container on its own.
3. Learn how to set up docker networks for the 1st and 2nd container so that the torrent client will always use the VPN.
4. Try to set up a Jellyfin container on its own.
5. Move on to the *arr stack.

nixOS also has a bit of a learning curve and it would’ve probably been easier if you started with something else. Up to you if you want to stick to it. IMO it’s mostly overkill for an OS that will simply serve as the base for a docker setup.

Use Void as the basis. Most of the tools you’ll need are in the repo (add the non-free repo as well), the ones that aren’t, are in xbps-src and you compile from source (it’s only a few commands after you clone the repo).

Regarding samba, holler, I’ll share what I use. Also, don’t forget to install nss-mdns for discoverability. Also holler if you need help getting the services to autostart on boot up.

For the VPN issue: Transmission + wireguard or Transmission + OpenVPN containers will do what you need.

They don’t allow transmission to access the network unless it is through the VPN. You can test by using the torrent magnet at ipleak.net

For the *arrs, you will have to import all of your existing files manually, assuming they don’t have the standard format like mine didn’t (movie.title.year.encoding-uploader)

You do not need a vpn on the *arrs and jellyfin. They are not doing anything illegal. However, you should only run jellyfin on your local network or have a router/server-based VPN to log into from the outside. Hosted domains and proxies have a variety of TOS that will generally be violated if you stream jellyfin. The standard containers would be fine.

If you are only running on your local network anyway, just find someone’s raw docker-compose that is all set up (avoid traefik/ngnix/proxmox unless you want to dive into server management) and learn the 6 lines or so you will have to change to get it set up on your own system like volume paths for your media. Then you can just ‘docker-compose -f path/to/compose.yml up -d’ and everything will be running. Then you need to add the DOCKER IP addresses and API keys to each *arr from transmission in the GUI and they have a test button that will show that it is ok. Done.

Transmission: https://git.egor.wtf/me/my-little-flake/src/branch/main/hosts/common/optional/transmission/default.nix (I just noticed a have secret public, taking down it🫢) Jellyfin: https://git.egor.wtf/me/my-little-flake/src/branch/main/hosts/common/optional/jellyfin/default.nix

You can take both and smash them into your configuration.nix, change the hostname, and everything should work