Midnight Saci: an AI test engineer, built in a week with coding agents

Every project has a pile of work that is important but never urgent. For Wanaku, that pile was the test plans. We have hundreds of test units written in Markdown: set up a namespace, deploy the operator, call the CLI, check the output. A person can follow them. Nobody wants to follow them every night.

So I tried something. I wrote down what I wanted, handed it to coding agents, and kept the role of reviewer and product owner. Six days later I have Midnight Saci: an AI test engineer that runs Markdown test plans against real environments and shows the evidence for every verdict.

This post covers what Saci does, and what building it taught me about working with coding agents.

The name

The Saci-Pererê is a trickster from Brazilian folklore. He hides things, ties knots and causes small chaos for fun. Our Saci uses that mischief for good: he looks for bugs at night so you do not have to.

What Saci does

You write a test plan the way you would write it for a colleague: phases, tests, commands and the results you expect to see.

### Test 1.1: Read an existing order

```bash
curl -sS -w '\nstatus %{http_code}\n' "$SERVICE_URL/orders/1"
```

**Expected result:**

- The output ends with `status 200`.
- The order's `status` is `shipped`.

Saci carries out the plan step by step. An LLM decides what to do, and a small, bounded set of tools (shell, HTTP and MCP) does it. Each run gets its own isolated worker, as a Podman or Docker container or a Kubernetes Job. Credentials are redacted from the evidence. Any OpenAI-compatible model works, from OpenRouter to a local Ollama.

The important part is what Saci does not do. It follows the plan. It does not invent repair steps, and it does not patch your code. Every verdict ties each expected result to the command output that supports or contradicts it:

1.2 Create an order [test]: failed
  expected result 1: contradicted (mechanical; ev-2): the output ends with status 500, not 201
  expected result 2: contradicted (mechanical; ev-2): the body is {"error":"orders database is read-only"} and has no id

You check the reasoning instead of taking it on faith.

Watching a run

The web console shows a run while it happens. Here Saci runs the Wanaku MCP CLI test plan: 29 units, against a real Wanaku instance.

The Activity tab is the live log. Every shell command links to its evidence, and every attempt reports how many model requests and tokens it used.

The Plan performance tab answers the question I asked most often in the first days: where did the time go? In this run, the model took 97% of the time and the tools took 2%. That tells me where to optimize, and it is not the shell.

The Environment tab records the worker image, the OS and tools in the worker, and the exact Saci build. When a run from last Tuesday fails, I know what it ran on.

After the run

A test report is only useful if someone acts on it. Saci adds three things after a run.

Outcome actions. After a run finishes, the backend can call MCP tools: open a ticket, post to chat, send an email. Actions can wait for a person to approve them in the console, and they never change the verdict.

Investigations. When a unit fails, Saci can start a second agent that clones the source code, reads the code that serves the failed step, and ends with hypotheses that cite file and line. It can file a bug, or say why it did not. Other agents can request an investigation over A2A, so Saci fits into a larger agent setup.

Example investigation using the documentation’s sample data.

The script runner. saci-script-runner exposes administrator-defined scripts as MCP tools. A Saci worker, Wanaku or any other MCP client can call them. The first example emails run results with a plain bash script.

Saci works with your coding agent too

Hundreds of existing Wanaku test units did not have the exact format Saci expects. Most of them had their expectations in echo "PASS ..." lines instead of an Expected result section. Turning those into criteria needs judgment, so a converter that only applies rules would have to guess.

So the conversion is a skill for your coding agent, plus a validator:

saci skill install --all --agent claude   # or codex, cursor, copilot, gemini, opencode
saci check --strict plans/operator-basic-functionality.md

The agent converts the plan, and saci check tells it, offline, what is still wrong. Other skills start a backend, run a plan and investigate a failure. Saci was built by agents, and it is designed to be driven by them too.

An experiment with small models

Most steps in a test plan are simple: did the command exit 0, does the output contain this line? Asking a frontier model about each of them is slow and expensive. Saci can use a "System One" judge, a small and fast decision model, to pass the obvious steps. It falls back to the LLM for everything else.

We tried Julia 1, a 144M parameter open model, served by a small Rust module with ONNX Runtime. To be honest: on real Wanaku runs it was rarely confident enough, so the LLM still judged almost every step. The plumbing is in place, and fine-tuning is the next experiment. I include this because not every idea works on the first try, and that is fine.

How it was built

The repository started on 2026-10-17. By 2026-10-23 it had about 40,000 lines of Rust across eight crates, a React web console, an OpenAPI contract, Helm charts, multi-arch container images, and 130 issues and pull requests. I did not write that code. Coding agents (Claude Code, running in cloud sessions) did. My job was different, and these are the things that made it work.

Write the plan first. Before any code, I wrote a system plan: the goal, the principles (simplicity, trustworthiness, adaptability), the technical baseline and the shape of the system. The agents broke it into tickets. Each change traces back to an issue.

Write the rules down. A CLAUDE.md file holds the standing rules: no unwrap or panic in production code, no Python or Perl for tooling, meaningful tests for every major feature, documentation in Simplified Technical English, no abstractions before a second real use. Agents follow written rules well. They do not read minds.

Let CI be the judge. make check runs format, Clippy with warnings as errors, and the full test suite. Agents fix their own CI failures before I look at a pull request. My reviews are about design, not about lint.

Run many threads at once. Each piece of work got its own agent thread: the investigations design, the Julia 1 judge, the script runner, the console pages. They ran in parallel and opened pull requests that I reviewed and merged.

Use the product to test the product. Saci has test plans for Saci itself, and it runs them.

Stay the product owner. The agents asked good questions: should plan conversion be a CLI or a skill? Should the judge run in the worker or as a separate service? Those decisions were mine, and they shaped the project more than any line of code.

What is next

Saci is a new project, and it shows. Some areas are rough, and the Julia 1 judge needs work. But it already runs real Wanaku test plans, and it already saves me time.

More than that, it shows what is possible now. One person with a clear plan, written rules and good CI can build a real system in a week. The skill that matters is not typing code. It is knowing what you want, and saying it clearly.

The code is at github.com/orpiske/midnight-saci. The walkthrough runs everything on one machine with a scripted model, so you do not need an API key to try it.