daily checkpoint

docs/plans/mandos-wake-on-demand-build-host.md

@@ -0,0 +1,125 @@
+# Mandos as a wake-on-demand build host
+
+## Goal
+
+Mandos is primarily an interactive living-room machine, but it is also a strong candidate for handling remote Nix builds when it is idle. The goal is to make that dual use practical without keeping the machine powered all the time.
+
+## Current context
+
+On `main`, Mandos is configured as an interactive gaming machine:
+
+- `systems/x86_64-linux/mandos/default.nix`
+  - `sneeuwvlok.hardware.has.gpu.nvidia = true`
+  - `sneeuwvlok.hardware.has.audio = true`
+  - `sneeuwvlok.desktop.use = "gamescope"`
+  - `sneeuwvlok.application.steam.enable = true`
+- `homes/x86_64-linux/chris@mandos/default.nix`
+  - user-facing application set for an interactive machine
+
+This makes Mandos a poor fit for "always running random infrastructure", but a reasonable fit for "available for work when needed".
+
+## Desired behavior
+
+- Mandos remains an interactive machine first.
+- Mandos can be used as a remote build worker when no one is actively using it.
+- Mandos should not need to stay fully on all day just to be eligible for builds.
+- Waking and idling down should be automatic enough that the machine can participate in builds without turning into a maintenance burden.
+
+## Recommended model
+
+### 1. Use wake-on-LAN as the activation mechanism
+
+Mandos should support being awakened by another machine on the same LAN.
+
+Requirements:
+
+- BIOS or UEFI wake-on-LAN support enabled
+- NixOS interface configuration enabling wake-on-LAN
+- one low-power machine that is effectively always available to send wake requests
+
+In this repo, `ulmo` is the obvious candidate to act as the coordinator, but the pattern should stay generic: one machine is always reachable, and one or more stronger machines can be woken on demand.
+
+### 2. Prefer suspend-first over shutdown-first
+
+There are two main power states worth considering:
+
+- **Suspend on idle**
+  - faster resume
+  - generally better user experience
+  - often easier to make reliable for wake-on-LAN
+- **Shutdown on idle**
+  - lowest power draw
+  - more fragile in practice because firmware support for wake from soft-off varies
+  - longer time to become available again
+
+Recommended rollout order:
+
+1. Prove the concept with suspend on idle.
+2. Only consider full power-off later if the hardware and firmware behave reliably.
+
+## 3. Add an explicit availability policy
+
+The interesting lesson for tagging is not "Mandos should have a build tag". The interesting lesson is that some machines have a deliberate availability policy that affects how safely they can participate in automation.
+
+A future host-level setting could encode this policy directly, for example:
+
+- `always-on`
+- `wake-on-demand`
+- `manual`
+
+That setting would be a better source for any computed operational tag than current workload or ad hoc tags.
+
+## 4. Idle detection should be policy-driven
+
+If Mandos becomes a build worker, idle shutdown or suspend should depend on signals such as:
+
+- no local interactive session activity
+- no active build job
+- no long-running system task that should keep the machine awake
+
+This should not be a blind timer that powers the machine down every X minutes regardless of context.
+
+## 5. Build orchestration needs a coordinator
+
+Wake-on-demand only works well if something else can wake the machine and wait for it to become reachable. In practice, this means:
+
+- a coordinator sends the wake signal
+- the build client retries until the machine is reachable
+- the remote builder participates only after it is actually ready
+
+The exact implementation can vary, but the architectural point is the same: a wakeable build worker is not self-sufficient.
+
+## Risks and caveats
+
+- Firmware wake support may be unreliable, especially from full shutdown.
+- Build latency increases because wake and readiness checks take time.
+- A machine that users expect to be immediately available should not surprise them with power-state transitions at awkward moments.
+- Interactive workload detection matters; otherwise the machine will feel hostile as a living-room device.
+
+## Recommendation
+
+Treat the Mandos idea as a good pattern, but generalize it:
+
+- some machines are **interactive**
+- some machines are **wakeable on demand**
+- some machines are suitable for **interruptible background work**
+
+Those are more reusable concepts than "Mandos is the build server".
+
+## Implications for the tag strategy
+
+This investigation strengthens a small part of the `operational:*` space:
+
+- `operational:availability:always-on`
+- `operational:availability:wake-on-demand`
+- `operational:workload:interruptible`
+
+These should not be assigned by hand if they can instead be computed from explicit machine settings that describe availability policy.
+
+## References
+
+- Clan inventory tags and dynamic tags docs: `https://clan.lol/docs/25.11/reference/options/clan_inventory`
+- NixOS Wake-on-LAN wiki: `https://wiki.nixos.org/wiki/Wake_on_LAN`
+- Home-lab wake-on-demand discussion and patterns:
+  - `https://dgross.ca/blog/linux-home-server-auto-sleep`
+  - `https://danielpgross.github.io/friendly_neighbor/howto-sleep-wake-on-demand.html`