Sargasso/Dicefire Update - Stop, Code, and Roll

Where are we?

This update, a testament to our imperfect scheduling, is a double whammy: Both Sargasso and Dicefire get to hog the limelight.

Last round, we got one coding task through a thin slice of Sargasso on our own hardware. We could call it barely passing but not ready for prime time. That, with these last two weeks of grinding, has significantly shifted. The reason we’re highlighting both is because we aimed Sargasso directly at Dicefire for our POC to see if it could make art.

…and it did! Kinda. We’ve begun a heavy research phase into how we want Sargasso to work and taking notes from other harnesses.

In that research, we spent a lot of time finding out what breaks on the first, second, tenth, and twentieth try. We’ve gone through books of articles, how tos, propositions, ponderances, and gripes looking at our own shape. In that time we’ve ended up building what will become the CLI interface, an active terminal for Sargasso…design wise. It’s still in beta.

Now then, on to the fun stuff..

---

Pain, in 16k...

Hermes and OpenClaw, our go to inspiration harnesses, are the first places we looked when we wanted to see how another agent harness solved a problem. We do this every step before going online and looking at other 3rd party designs. This time though, we had an issue where the answer was a little painful. The Hermes build we studied expected at least 64K. OpenClaw would let us run at 16K, but its local llama.cpp examples also started at 64K. Our two go-to references were telling us, in different ways, to give the model more room.

But you know what, not everyone has the extra room for 64k, especially on a personal machine where you already need to quant your models to get things decently running. This grievance in mind, we decided to make 16K our working window. We want Sargasso to be something more people can try on hardware they already own.

Making that happen got us a fairly rude lesson in what that costs. The test itself was clear enough, take the phone sprite asset from Dicefire, the one that broke both Astra and Claude trying to make, and make it make…something at all. Didn’t have to be perfect, just make something. 146,839 tokens is the price of that something, as we found out.

---

What Worked, What Didn't?

Here’s what our failure looked like:

• Thinking saturation. Requiring so much context from instructions and reasoning that you never get to the problem cause you blow the budget thinking. That happened two ways, immediately on heavy prompts, and over time shortly after doing things.
• Limited resources. Not being able to offload thinking to another agent due to our 16k window. Not being able to increase the budget because of that 16k window as well.
• Design flaws. We saw several key places both in security and in elegance where our architecture paled in comparison to other like models. Not going in to detail, but we've patched after discovery. Nothing escaped labs.
• Instruction handling. This was a weird one and we're still investigating the why. We had a model try to determine that the image of the phone needed revision. It said the image should fail, and then on paper it marked it pass. We think we caught the bug and its in code.
• Panic. Our models, it turns out, can get caught with analysis paralysis. We've given them an out, and it's holding up so far but more testing needs to be done.

In one particular test, as an example, the worker spent twelve turns reading files and never wrote the sprite. It was stuck trying to decide what to do, and didn’t have the bandwidth to reach a conclusion quickly. We’d reserved limited space for thinking and answering that the next turn could lose the very thing the agent had just read. So it read it again.

These issues pushed us toward the approach we’re testing now. This is where we are with more to come:

• We keep the recipe, renders, checks, and decisions on disk. Each agent gets the part it needs for the job in front of it, and we count the whole request before sending it. This not only reduces our need to use RAM, it also lets us index and speed up how fast our models think.
• We pick and choose who thinks. Our observer models and verifiers get larger thinking windows then say something like a grunt agent.
• We hot swap models. Due to limited seats, we learned we can get a model to reason, freeze it, swap it with another model, and let them pick up the call from the previous model. Then, we can swap it back to the older one with no loss of context reasoning. That gives us unlimited, but rotating seats. We tested this extensively, researched it, and we've yet to see a hard issue.
• Assembly line architecture. Since there's limited budget, we treat agents like temporary workers. Come in, grab the context they need, make their changes and leave. This recursive looping lets us do something a lot of clever harnesses do: work in the boundaries of their limited window while being able to reach massive token numbers. Work in progress.

With the failures we hit and the changes we’ve made, it’s taken up the whole 2 weeks of work. But we did, eventually, get something that kind of sort of looked like a phone!

Fun fact, we already have the phone sprite. We weren’t expecting anything we’d depend on. We just wanted the model to complete in 16k, and like we stated, it did it in roughly 143k tokens with a 16k VRAM budget in about 15 min. Good, but we will get better.

---

What's Next?

• Increase performance of the agents where possible.
• Test on more Dicefire assets with even harder tests.
• Finish the debugging log we're building up as we go.
• Fine tuning our pass/fail gate for the verifier agents.
• And if this takes too much time, probably split the team and have one side work on Dicefire in parallel.