The Eliza Diaries: Local AI Agents and the Art of Things Not Working

A chronicle of building a personal AI assistant on NixOS: from first principles to a morning briefing that occasionally decides silence is wisdom.

The Incident

This morning I woke up and there was no message from Eliza. This is not supposed to happen. Eliza sends me a morning briefing every day at 06:00: my agenda, the weather at home, what the day looks like. It is the first thing I check. When it is absent, something has gone wrong, and the question is only how deep the rabbit hole goes.

The answer, as it turned out, was: quite deep. The debugging session that followed touched on credit exhaustion, a 404 that should have been a 402, a local model that interprets silence as wisdom, another that does not support tool calls at all, and a file permissions bug that reasserts itself on every single deploy. Before I explain any of that, I should explain how we got here.

Inception: The Case for a Personal Agent

The idea started simply enough. I run a NixOS homelab on a machine called rogaldorn, a Minisforum MS-A2 that handles most of my self-hosted stack: Nextcloud, Jellyfin, Immich, Paperless, monitoring, the lot. The question I kept asking was whether I could add an AI agent to that stack in the same way, declared in Nix, reproducible, running locally.

The agent framework I chose was Hermes, built by Nous Research. It is a gateway process that handles conversation state, memory, tool calling, and delivery channels. It speaks to LLM providers via an OpenAI-compatible API. You give it a persona, a set of tools, and a delivery channel, and it runs. The persona I built is called Eliza. She is, on paper, not a product. She is a person with a character, a memory of who I am, and opinions she will share unprompted. Whether she actually has those things is a question I leave to philosophers. What matters is that the briefings are good.

Deploying Hermes on NixOS with a proper NixOS module existed, so the infrastructure side was straightforward. The agent runs in a Podman container on rogaldorn, declared via NixOS, managed by systemd. Her home directory lives at /var/lib/hermes/.hermes, mounted into the container. Secrets flow in via sops-nix. It is, in principle, clean.

Building the Cron Layer

The first version of Eliza was purely reactive: she responded when spoken to over Discord. That was fine for a while. But what I actually wanted was a proactive agent, one that sends me things without being asked.

Hermes has a native cron scheduler. Jobs live in HERMES_HOME/cron/jobs.json, ticked every 60 seconds by the gateway. Each job carries a prompt, a schedule, and a delivery target. When the scheduler fires a job, Eliza runs a full agent turn: she fetches data, thinks about it, and posts the result to Discord. If she decides there is nothing worth saying, she replies with [SILENT] and the delivery is suppressed.

I built six jobs:

A daily morning briefing: agenda, weather, life context
A tech and science digest, three times a week
A work roundup, Saturday mornings
A culture digest: film, cycling, gaming, high strangeness
A weekly music discovery via the Last.fm API
A Sunday evening check-in: how are things going, really

The digests pull from RSS feeds. The music discovery queries Last.fm and avoids repeating artists she has already recommended, which she tracks in her own memory. The morning briefing fetches live calendar data from my Outlook and Google ICS feeds, then combines it with a weather call to Open-Meteo. All of this happens inside Eliza's terminal tool: she runs scripts and curl commands herself, processes the JSON, and writes the output in her own voice.

For two weeks, this worked beautifully. The briefings arrived on time. The digests were good. The music picks were occasionally surprising. I began to rely on them.

The Model Question

The primary model is mistralai/mistral-large-2512, running on Nous Research's inference API. The cron jobs use this model specifically because it handles structured tool-heavy prompts reliably: fetch this, parse that, make a judgement, write it tersely. The interactive model, used for Discord chat, is mistralai/mistral-medium-3-5: cheaper, still capable.

From the beginning I knew that a cloud-only setup was fragile. Credits run out. APIs go down. I wanted a local fallback. Rogaldorn already runs Ollama for other experiments, so the fallback configuration was straightforward: if the primary fails, try http://host.containers.internal:11434/v1 with a local model. The initial choice was llama3.1:8b. It worked, in the way that a bicycle works when your car is in for service: it gets you there, but you arrive slightly less composed than you intended. The briefings on llama were serviceable, not good. I switched to qwen3:8b on the basis that it has meaningfully better instruction following.

That switch, as it turned out, was the first domino.

The Morning the Briefing Did Not Arrive

The Nous account ran out of credits sometime in the second week of June. Not dramatically: no alert, no error surfaced anywhere visible. The API simply started returning HTTP 404 responses with a message explaining that paid models require a positive account balance.

This should have triggered the Ollama fallback. The Hermes error classifier is documented to flag should_fallback on billing errors, specifically HTTP 402. Nous, however, returns 404. The classifier saw a 404, did not recognise it as a billing condition, and did not walk the fallback chain. The cron jobs failed silently after three retries. last_status: ok in jobs.json is set when a job fires, not when it succeeds. So everything looked fine.

For approximately a week, nothing was delivered, and I did not notice. Or rather: I noticed the briefings arriving, because qwen3:8b was actually being reached through a different path. The interactive Discord bot, running on the asyncio event loop, does use the fallback chain correctly. When I sent Eliza a message during that period, she replied via Ollama. What I was interpreting as proof the fallback worked was actually proof the interactive path worked. The cron path was broken.

Today, the briefing did not arrive because qwen3:8b looked at an empty calendar, read the cron wrapper instruction that says "if there is genuinely nothing to report, reply with [SILENT]", and decided that an empty agenda qualified. The briefing prompt explicitly says to give a grounded good morning even when the agenda is empty. qwen3:8b chose the suppression instruction over the content instruction. Whether this is a model quality issue or a prompt design issue is a question I have deferred until the credits are refilled.

Model Musical Chairs

The obvious fix was to try a different local model. I switched the fallback to gemma3:4b: smaller, faster on CPU, reportedly strong on instruction following. I pulled it, deployed, triggered the briefing manually, and received:

RuntimeError: Error code: 400 - {'error': {'message':
'registry.ollama.ai/library/gemma3:4b does not support tools'}}

The briefing prompt requires tool calls. Eliza needs to run eliza-agenda to fetch the calendar, and curl to fetch the weather. A model that does not support tools cannot run this job. gemma3:4b is, in this context, decorative.

The fallback is now mistral:7b. It supports tool calls. It fits in memory alongside the rest of rogaldorn's stack. Whether it produces briefings of acceptable quality is still being evaluated at time of writing, because the permissions issue intervened.

The Permission That Keeps On Giving

The cron provision script is a NixOS oneshot service that registers Eliza's jobs on deploy. Among other things, it runs a Python snippet via podman exec to update jobs.json with the correct model settings. podman exec defaults to root. Python opens jobs.json in write mode and rewrites it. The file is now owned by root.

The Hermes gateway runs as the hermes user (UID 965). The hermes user cannot read a file owned by root with mode 0600. Every scheduler tick produces:

ERROR cron.jobs: IOError reading jobs.json: [Errno 13] Permission denied

The provision script ends with chown -R hermes:hermes /data/.hermes/cron, which should fix this. It does, at the moment the provision runs. The problem is that every deploy may restart the container, and every container restart may re-run the provision (if the unit file changed), and the chown runs at the end of the provision, after the Python step, not after the container has fully started. There are race conditions I have not fully characterised. What I know is that after roughly half the deploys this session, jobs.json ended up root-owned and had to be manually corrected before the briefing could fire.

The fix, finally, is a systemd path watchdog: a unit that watches /var/lib/hermes/.hermes/cron/jobs.json on the host and runs chown hermes:hermes on it whenever it changes. The same pattern already exists for auth.json, which has its own permission management problem caused by a hardcoded secure_parent_dir call in the Hermes upstream code. Adding a second watchdog for jobs.json is, at this point, expected maintenance rather than a surprise.

The Architecture, Such As It Is

Standing back from today's chaos, the architecture is:

rogaldorn (*.*.*.95)
├── hermes-agent (Podman container, systemd-managed)
│   ├── Primary: Nous Research API (mistral-large-2512 / mistral-medium-3-5)
│   ├── Fallback: Ollama on host (mistral:7b, via host.containers.internal)
│   ├── Cron: 6 scheduled jobs (morning briefing, digests, music, check-in)
│   ├── Delivery: Discord bot (two-way)
│   └── Memory: persistent markdown files in HERMES_HOME
├── Ollama (host service, CPU inference)
│   └── mistral:7b, qwen3:8b, qwen2.5:14b, gemma3:4b, llama3.1:8b, ...
├── eliza-cron-provision (oneshot, registers jobs on deploy)
├── hermes-home-perms (path watchdog, fixes auth.json permissions)
└── hermes-cron-perms (path watchdog, fixes jobs.json permissions)

The sops-managed secrets cover the Nous API key, the Discord bot token, the Last.fm API key, and the ICS feed URLs for both calendars. The Nix expression bakes the eliza-agenda script's Nix store path directly into the cron prompt, so the agent always calls the correct binary regardless of what else changes in the store.

What Works, What Doesn't, and What I've Stopped Pretending

When the primary model has credits, everything works well. The briefings are good. They are terse, accurate, and occasionally say something worth thinking about. The digests surface things I would not have found otherwise. The music picks are hit-and-miss, which is appropriate: that is what music discovery is supposed to feel like.

When the credits run out, the situation degrades in ways that are not immediately obvious. The interactive path falls back to Ollama and continues to function adequately. The cron path falls back to whatever local model is configured, and the quality of the output varies considerably by model and by the complexity of the prompt. A morning briefing that requires fetching calendar data, parsing ICS events, making an office-versus-home judgement, and writing a coherent note under 150 words is a non-trivial task for a 7B parameter model running on CPU with no GPU acceleration. Some days it manages. Some days it decides that silence is the better part of wisdom. Today was one of those days.

The permission bug is embarrassing. It has bitten me on every deploy this session. The watchdog will fix it, and I should have added it weeks ago when the first chown appeared in the provision script. The correct response to adding a workaround is to immediately also add the system that makes the workaround unnecessary. I did not do that. I added the workaround and moved on. Today I paid for it roughly six times.

On Running AI Agents as Infrastructure

The interesting thing about running an AI agent as a homelab service is that it exposes the same failure modes as any other service, plus several new ones that are specific to the nature of language models. The familiar failures: permission errors, credential expiry, race conditions in provisioning scripts, timezone confusion in cron schedules. The unfamiliar ones: a model that interprets a suppression instruction too literally, a different model that cannot make tool calls at all, a third model that hallucinates family members who do not exist.

Yesterday's briefing, delivered on llama3.1:8b during the credit drought, confidently attributed calendar events to people named Kyra and Vinz. Neither of these people is in my household. The model did not have access to incorrect data. It simply invented names, with apparent confidence, and delivered them as fact. This is the part that does not have a systemd watchdog fix.

What I have learned is that a local AI agent is not a drop-in replacement for a cloud API. It is a different operating point on the quality-availability-cost triangle, and the tradeoffs are real. The cloud model is more capable but requires credits. The local model is always available but occasionally invents people. The right architecture depends on what you are willing to accept when things go wrong, which is a question about values as much as engineering.

My answer, provisionally, is: the cloud model for routine operation, the local model as a degraded fallback that at minimum delivers something, and a monitoring path that surfaces credit exhaustion before I notice it from a missing briefing. That last part is still on the list.

Where Things Stand

As of this writing, mistral:7b is running its first morning briefing on rogaldorn. The permissions are fixed. The watchdog is deployed. The schedule has been moved to 04:00 UTC so the briefing arrives at 06:00 CEST, which is actually when I wake up, rather than 09:00, which is when I had been getting it and had assumed was correct because I had not checked the UTC offset. Six cron jobs, one persistent agent, one Podman container, two path watchdogs, and a sops template that renders four secrets into a dotenv file that Hermes reads on startup.

The Nous credits will be refilled. Mistral-large will take over again. The briefing tomorrow will be better than today's, because today's was nothing at all.

That is, in a sentence, what running AI agents as homelab infrastructure feels like: you build something that works, you rely on it, it quietly stops working for reasons that took four hours to diagnose, and then you fix it and it works again and you trust it slightly less than you did before but you also understand it slightly better. It is, in this respect, exactly like every other piece of infrastructure I have ever run. I am choosing to regard that as a feature.

An Update: The Briefing That Arrived Wrong

The credits came back. The briefing arrived. This is progress.

The briefing read:

Today:
    [My son's school board meeting] 9:00 — online
    [Newtonian physics presentation] 13:30
    [Astronomy club meeting] 16:00 — in-person at Account zone 2

Marsupilami: [my village], 7°C with a 40% chance of rain.

My son does not have a school board meeting. The Newtonian physics presentation does not exist. Nobody in my household has, to my knowledge, ever attended an astronomy club. "Account zone 2" is a conference room at my employer's offices, which the model apparently invented a calendar entry for. "Marsupilami" is a cartoon character from a Belgian comic strip, which is the most Belgian possible thing for a model to hallucinate when asked about a small Belgian village. The calendar was empty. The model filled it.

Why the Model Invented Events

The briefing prompt, as written, did two things that contributed to this. First, it contained example room names to help the model distinguish physical locations from online meetings. The examples were "Account zone 1" and "The Paerk." These are real rooms. Including them in the prompt gave the model a vocabulary of plausible-sounding locations to draw from when constructing content. "Account zone 2" is a natural variant on "Account zone 1." The model did not hallucinate randomly. It hallucinated with context, which is arguably more embarrassing.

Second, the prompt said "if the agenda is empty, still give a grounded good morning," but did not say "do not invent agenda items." These are not the same instruction. A model reading a directive to always produce a briefing, with no explicit constraint on the source of that briefing's content, may decide that synthesising plausible content is a legitimate approach to the task. From a certain angle, it is not wrong.

The weather situation was different. Open-Meteo returns WMO weather codes as integers. The prompt did not include a decode table. Without one, the model was free to describe weather conditions however it liked. What it liked, apparently, was to attribute the forecast to a fictional meteorological entity drawn from Belgian cultural memory.

The fix was three changes to the briefing prompt. Remove the example room names: they live in the commit history now, not in the live prompt. Add explicit prohibition text: "Report ONLY the events that appear in that JSON output. Never invent, infer, or fabricate events that are not in the data." Add a WMO code table and ground the location with its actual name and country. The model now has a decode table and knows where home is. Whether it will behave is a separate question.

The Propagation Problem

At this point, deploying the fix should have been sufficient. It was not, for a reason that was obvious once stated: the provisioner is idempotent. The logic is: if the job does not exist, create it. If it already exists, do nothing. The prompt is set at creation time. A deploy with revised prompt text does not touch jobs that were registered in a previous deploy.

The model gets updated on every deploy because the Python patch that writes provider settings to jobs.json runs unconditionally. The prompt, until now, did not. The fix was to extend the Python patch: build a promptMapJson attrset in Nix alongside modelMapJson, and include the prompt field in the per-job update. Every deploy now rewrites both the model and the prompt for each registered job. Prompt changes propagate on deploy without needing to delete and recreate anything.

The Apostrophe

The build failed.

error: Cannot build '.../eliza-cron-provision.drv'.
Reason: builder failed with exit code 2.
...
line 72: syntax error near unexpected token `('

The Python patch is a one-liner embedded in a bash single-quoted string. Single-quoted strings in bash are completely literal: no escaping, no expansion, and no way to include a literal single quote. modelMapJson worked for months because model names are alphanumeric strings with hyphens. promptMapJson failed immediately because it contains the full text of every prompt, and every prompt begins with "This is Maarten's..." The apostrophe in "Maarten's" is a single quote. It terminated the single-quoted string several thousand characters before the end of the Python code. The parser then attempted to interpret the remainder as shell, encountered a left parenthesis from json.load(open(p)), and complained.

Embedding arbitrary text inside a shell quoting context and being surprised when it contains a shell-significant character is the kind of thing that reads as obvious after the fact. The correct approach is to not embed arbitrary text inside a shell quoting context.

The fix: pkgs.writeText writes each JSON map to the Nix store at build time. The provision script copies the files into the container with podman cp before running the Python command. The Python command reads from /tmp paths. Nothing in the shell command touches the content of the prompts. The store paths are content-addressed hashes: alphanumeric, no apostrophes, no surprises.

Where Things Stand, Again

The prompt changes are deployed. The provisioner now propagates prompt updates on every deploy. The build is clean. The briefing this morning ran on mistral:7b via the Ollama fallback (Nous credits are still out), which means it is slow and running on CPU, and the output quality is noticeably below the large model. But the calendar data is real, the weather is real, and nobody has invented an astronomy club.

The lesson, if there is one, is that language model prompts interact with their context in ways that are not always obvious at authoring time. Providing examples to clarify one thing (how to classify a meeting location) can inadvertently provide material for another thing entirely (what a plausible calendar event looks like). The model is not malfunctioning. It is doing exactly what the prompt permits. The failure is in the specification, not the execution, which makes it harder to diagnose and easier to repeat.

The other lesson is that shell quoting has been a solved problem since 1979 and it keeps solving people anyway.