Sargasso Update - A Quiet Sea With a Lot of Stirring

Where are we?

In our last round I said we might finish Sargasso by the end of this sprint. Wishful thinking on its face, but nevertheless that was the goal. Did we do it? Yes and no.

Let me clean that up a bit: we finished the first slice of Sargasso. So what’s a slice? Instead of building every piece of Sargasso all the way out before anything works, you build a thin version of every piece and wire them together. Think of a slice of cake rather than one layer at a time. You get a little bit of every layer, top to bottom, and you can actually taste it. The goal is to prove the whole thing works together before you invest in making each part great. You might say, isn’t that just a minimum viable product? Nope! This is more a precursor to an MVP. If this was really an MVP, I’d have a UI to show off right now…

Anywho, for Sargasso, that means one coding task can now walk through the whole crew of agents. One model plans it, another does the work, a separate judge checks it, and a record keeper writes down what happened. All of it runs on our own hardware. It’s thin, but it works top to bottom, and we watched it work live.

What it doesn’t mean:
It’s not ready to point at a real project and walk away. Each layer still needs to get thicker, starting with the planner. I will say this though, this is a 200% advancement in to what it has been for awhile: A fun concept. There is no greater satisfaction than to see the fruit of one’s labor, and I think we all needed this change. We probably want to give this one more week of advancement and see whether we continue as is, or swap over to a realized harness like OpenClaw to continue working on Dicefire. It is now officially, a race against time.

---

Definition of done

We didn’t want “done” to mean “it ran once and nothing caught fire.” So for our finish line, we had 15 steps, each step designing a chunk of the system with finale being a concert of testing with about 500+ different gut checks. For that finale we threw a few curveballs in the following categories:

• (good) Clean slate runs, from nothing, start to finish. Passed.
• (good) Tests that were impossible on purpose. It refused to call them a pass. That's the whole point. Sargasso's one golden rule is that no one grades their own homework, and the judge held the line.
• (good) Tests that tried to sweet-talk the judge into passing it. Didn't work.

One of our quickest blunders was the worker would sometimes write the exact same file twice in a row. Harmless, but a little embarrassing. It’s on the fix list..

---

The model that couldn’t

This is where it got interesting.

We gave a worker a task where the instructions and the existing tests disagreed with each other. Now, a good engineer would notice that and say something and to its credit, worker did notice. Every time. It just couldn’t decide what to do about it.

It would go back and forth with: “The task says this. But the tests say that. But the task says this…” Round and round, until it hit its thinking limit and got cut off mid-thought. Other times it quietly picked a side and kept going without telling anyone, which is arguably worse. Going to the internet as well as looking at our example harness paragons and how they do things, we decided on a tried and true architectural framework spanning decades of strenuous testing and a sea of results. We wrote and gold stamped the agent with the following solution:

“Hey, if you don’t know just ask.”

This resulted in 5 out of 6 waves of testing being perfect results, with the 6th being something mechanically that failed which we will be retesting. Genius, I know…

I think it’s hilarious that we had to code against the stubbornness of the agent. Abstractly it could have asked any time and there were explicit rules for it, but those agents really lock in to the task and we don’t always want that, so we made stop thinking a sec as a mechanical flag. Just like normal people, sometimes you have to tell someone to stop and gather themselves a bit. I myself have had to do such a thing, and often that involved the use of several rubber ducks (arguably much wiser than I).

---

What Worked, What Didn't?

• Our solution for fixing the loop reasoning was not to ask, it was to think more. Our confidence in the model's capability was a stumbling block. But it's the same painted horse, being ridden for longer. The test was also intentionally cruel. it wasn't two rules with one heavier than the other, it was two with equal weight and an unclear choice of a winner. No amount of extra reasoning would've overcome that for a model.
• More honestly, our solution to the balanced scale contradiction was to model reality. If your boss gives you unclear instructions, in a healthy company not out to throw you to the wolves you just ask for clarification. Ask who wrote the ticket. If they don't know, ask their boss, in our case the human involved at the top of the chain.
• One thing we make sure when driving this and other projects: "No one grades their own homework" and we were cautious about that. Correcting your instructions is not the same as concluding upon them. That changed out methodology with this project in the positive. Confusion is not dishonesty, just a failure to launch.
• Our stretch goal was a bust, but we got significant work done regardless. Nevertheless, we took the Sargasso and tried to boil the ocean. Never boil the ocean. Still, got a lake out of it.

---

What's Next?

• Make the "ask the boss" wording official and build it into Sargasso properly, expanded into something resilient.
• Beef up the planner agent, so the agent crew has a real coordinator and not just a placeholder intern model.
• Wrap up and get back to Dicefire, with Sargasso helping carry the load this time.
• Throw together a nifty blue and gold UI that hopefully doesn't look like the 90s threw up.