Saves follow you

A console that holds your progress inside it is a console you can never rebuild. This part is the smallest amount of infrastructure that fixes that: the saves leave the box, they go somewhere I control, and the box can be wiped at any point without losing an evening’s play.

The problem: progress that only exists on the machine in front of you

The save file is the only irreplaceable thing in this whole setup. ROMs can be re-fetched, metadata can be scraped again, a BIOS can be found a second time — but the 40 hours inside a .srm exist in exactly one place if you let them, and that place is a filesystem on a Pi that I reflash whenever I change my mind about how it is built.

There is a second, quieter version of the same problem. RetroArch writes a battery save when you close the content properly. A hang, a crash or a power cut does not close anything properly, so everything since the last clean exit is simply gone — and a crash-prone N64 core makes that a real risk rather than a theoretical one. The fix is unglamorous:

autosave_interval = "30"

Every thirty seconds the save is flushed to disk whether or not the exit was clean. That is the whole difference between “an evening lost” and “thirty seconds lost”, and it costs one line.

RetroArch already has cloud sync. Point it at your own server.

RetroArch ships a cloud sync feature, and the part that matters is not the storage — it is the timing. It syncs around content load and content close, at the moments the save is actually different, and it is conflict-aware rather than a blind overwrite. That is exactly the shape of the problem, so I pointed it at a WebDAV endpoint on the VM instead of a third-party service.

Pointing it anywhere turned out to be the difficult part, because two of the settings I reached for do not exist, and nothing says so:

• The destination key is webdav_url. There is no cloud_storage_url — that name is not a RetroArch setting at all.
• Savestates are covered by cloud_sync_sync_saves. There is no separately named toggle for them, so the setting that sounds like it only handles battery saves is the one that syncs everything.

Both mistakes fail the same silent way. RetroArch drops configuration keys it does not recognise when it rewrites its own config file on exit, so my invented keys were politely deleted — and the store stayed empty while everything looked correct: sync enabled, credentials accepted, no errors anywhere. The lesson is not “read the docs”, it is that a configuration file which rewrites itself will quietly absorb a typo and then act as if you never wrote it.

        templates."retroarch-cloud.cfg" = {
          owner = "guest"; # RetroArch runs as guest and must be able to read the overlay
          content = ''
            cloud_sync_enable = "true"
            cloud_sync_driver = "webdav"
            # NB: the destination key is `webdav_url`, NOT `cloud_storage_url`
            # (which is not a RetroArch setting at all). RetroArch silently drops
            # keys it doesn't know when it rewrites retroarch.cfg on exit, so the
            # old name left webdav_url = "" — cloud sync ran enabled, with valid
            # credentials, and uploaded nothing. Same trap on the saves toggle:
            # the real key is `cloud_sync_sync_saves` (it covers savestates too).
            webdav_url = "https://${hetznerApp}:8600/"
            webdav_username = "retroarch"
            webdav_password = "${config.sops.placeholder."retroarch-webdav/password"}"
            cloud_sync_destructive = "false"
            cloud_sync_sync_saves = "true"
            # Don't sync retroarch.cfg to the share: RetroArch bakes the merged
            # config — webdav_password included — back into retroarch.cfg, so
            # leaving this on (its default) would upload the password too.
            cloud_sync_sync_configs = "false"
          '';
        };

WebDAV over the tailnet, not the internet

The endpoint itself is a single container running rclone serve webdav over a directory on the data disk — not the root disk, because saves are user data and belong with the rest of it. It listens on a loopback port, and the only way to reach it is through the Caddy instance on the same VM, which terminates TLS for a tailnet name. HTTP basic auth is handled by the WebDAV server itself, with user and password injected from sops into an environment file, so the password never appears in a container argument where ps would find it.

The whole module is also inert until its secret file exists, which is what lets the VM keep evaluating on a fresh checkout: no secret, no endpoint, no broken deploy. That pattern repeats throughout this setup, and it is the reason the repository does not need placeholder passwords that eventually get committed by accident.

        sops.secrets."retroarch-webdav/password".sopsFile = secretsFile;
        sops.templates."retroarch-webdav.env".content = ''
          RCLONE_USER=${webdavUser}
          RCLONE_PASS=${config.sops.placeholder."retroarch-webdav/password"}
        '';

        virtualisation.oci-containers.containers.retroarch-webdav = {
          image = images.rclone;
          user = "0:0"; # write to the root-owned /data volume
          cmd = [
            "serve"
            "webdav"
            "/data"
            "--addr"
            ":8080"
            # short dir cache so a save written by the Pi is visible on its next read
            "--dir-cache-time"
            "5s"
          ];
          environmentFiles = [ config.sops.templates."retroarch-webdav.env".path ];
          volumes = [ "${dataDir}:/data" ];
          ports = [ "127.0.0.1:${toString hostPort}:8080" ];
        };

What is actually in there

Here is the store as it stands, read off the VM when this page was built. The console writes two kinds of thing: battery saves, which are the game’s own memory card, and savestates, which are a snapshot of the whole emulator that only RetroArch understands. Both sync, because both are things you would miss.

Making the saves show up in RomM

The library part put RomM in charge of what exists; it would be a shame if it could not see what I had played. RomM speaks neither WebDAV nor RetroArch’s cloud-sync protocol, and it files saves under its own database key — user, platform, rom id — so the two stores can never be lined up by path alone. A small job bridges them: every fifteen minutes it walks the sync store, matches each filename back to a row in RomM’s database, and uploads it through RomM’s HTTP API with a token that can write assets. It keeps a hash of what it has already uploaded, so the steady state is “nothing changed, nothing to do”.

It only ever goes one way, and that is deliberate. RetroArch’s cloud sync is conflict-aware about its own store, which means it can be trusted as the only writer into that tree; a second process putting files back in could silently overwrite a save I had just made, and I would find out weeks later. RomM is a mirror of my progress, not a second source of truth for it.

      config = lib.mkIf have {
        sops = {
          secrets."romm-save-import/client_token".sopsFile = secretsFile;
          templates."romm-save-import.env".content = ''
            ROMM_CLIENT_TOKEN=${config.sops.placeholder."romm-save-import/client_token"}
          '';
        };

        systemd.services.romm-save-import = {
          description = "Import RetroArch saves/states into RomM";
          after = [
            "podman-romm.service"
            "podman-romm-db.service"
          ];
          path = [ pkgs.podman ];
          environment = {
            CLOUD_DIR = "/mnt/data/retroarch-cloud"; # what the Pi syncs to
            ROMM_URL = "http://127.0.0.1:8095"; # RomM's loopback host port
            STATE_FILE = "/var/lib/romm-save-import/state.json"; # upload hash cache
          };
          serviceConfig = {
            Type = "oneshot";
            StateDirectory = "romm-save-import";
            # Two rendered env files rather than one: MARIADB_PASSWORD comes from
            # the same template romm-db/retroarch-export already use, so the DB
            # password stays defined in exactly one place (flake/vm/romm.nix).
            EnvironmentFile = [
              config.sops.templates."romm-db.env".path
              config.sops.templates."romm-save-import.env".path
            ];
            ExecStart = "${pkgs.python3}/bin/python3 ${./romm-save-import.py}";
          };
        };

        # More often than retroarch-export's hourly sweep: saves change every play
        # session, and the Pi uploads on content close.
        systemd.timers.romm-save-import = {
          wantedBy = [ "timers.target" ];
          timerConfig = {
            OnBootSec = "5min";
            OnUnitActiveSec = "15min";
            Persistent = true;
          };
        };
      };

What this does not solve

Progress is safe now, and that is a real change: I can rebuild the console whenever I want and lose nothing. What I still cannot do is watch what the box is doing — whether the sync actually happened, whether a service quietly died, whether the fan is screaming because a core is stuck. The last part is the machinery that answers those questions, and the interesting thing about it is how little of it is about monitoring and how much of it is about the one machine I do not have physical access to.