Why The Harness Matters More Than The Model | YC Paper Club

transcript

Harnesses often get dismissed as just scaffolding, just prompt engineering, and not real research. But that couldn't be farther from the truth. The same model weights that score 30% on ARC-AGI score 95% with a better harness.

So we gathered a group of researchers and founders working at the frontier to do a deep dive into the state of harnesses.

We cover how we got to this point, the case for making your harness as expressive as possible, and what YC learned building an agent for every employee in the company.

00:00 - @FrancoisChauba1: Why harnesses matter 04:27 - Building an auto-researcher by accident 07:13 - A five minute history of harnesses 13:56 - Self-improving harnesses 18:35 - @sethkarten: Prime Agent, a self-improving RLM harness 21:50 - Context as an L1, L2, L3 cache 24:51 - From Turing machine to von Neumann computer 28:33 - Messaging between agents 30:04 - ARC-AGI results 33:09 - Emulator Bench and GPU kernels 37:30 - @JonSaadFalcon: OpenJarvis, personal AI on personal devices 38:26 - How far behind are local models 39:21 - The five primitives of a personal AI stack 42:47 - Letting cloud models optimize your local stack 43:53 - 800x cheaper than the cloud 45:58 - @josh__france and @jbellregan: QM, YC's agent harness for work 47:29 - A history of YC's internal agents 49:24 - OpenClaw and a fleet of 50 agents 51:04 - Pulling the brain out of the sandbox 54:43 - Letting the agent choose its own sandbox and model 57:16 - The grind tool: budgets on goals 58:50 - Agents don't understand social context

Full verbatim transcript of the attached YC Paper Club / Harness Night talk, from the official X video English captions for posterity. The same talk is also published on YouTube.

Welcome to YC Harness Club I worked really hard. It took me like 10 iterations with Gemini to have Harshit of riding a lobster Gemini didn't want to do it but we figured it out Okay, welcome to Harness Night First order of business how does everyone like the new YC Paper Club look? Well this is like an idea that I kind of blurted out at Ev our head of design at YC and two days later she came back with this and I'm like, this is amazing So, Ev, in the back there please take a bow Very exciting All right Why harnesses?

I mean, it's just a wrapper This is just scaffolding This is just like prompt engineering Why would this be at all worthy of a knight? Here's a great little Reddit that was only a month ago which is actually the most aggressive I'm not sure this kind of prompt engineering belongs at a top tier machine learning conference Here's another great one Context engineering is not a research problem So I think harnesses have long been belittled as subpar research Yet it is literally gives us an 18% bump in the difference between harness one and harness two and as Seth will tell us is the difference between getting ArcAGI to work and not.

And so this is obviously worthy of some some amount of research And I think if you look at this the classic meter plots release date, and how long, an agent can be running literally we go a lot of this progress has been because of harnesses. And so I call this the static harness era where there isn't self-improvement on the harness And then this later latest maybe the last six months has all been on the self-improving harnesses and we'll get through all these And it's I actually saw this plot in a presentation by the CEO of Tra of Trajectory.

I actually really liked it It was talking about how the we, we keep measuring perplexity and you know, somewhat correlates to IQ and how and how intelligent the model is And so you keep pushing more and more up this intelligence IQ dimension but we're not leveraging test time experience very much. So we're generating all this test time experience. We have a new domain and, and it does and we don't really quickly adapt If you remember one of the first YCP clubs we did I've been doing this experiment where the, as I increase the number of samples online how do you actually learn from just batch size one?

We don't really don't have that structure. We have ICL and then once ICL gets saturated after just loaded like 40 or 50 it doesn't actually improve on the Valsa at all Then you have to go to LoRA small rank, then you go to LoRA big rank, then you go to SFT And it's kind of weird that we have this like Training procedures. And so I think that's really where harnesses are shining right now And what ArcAGI exposes that's kind of the main point of ArcAGI is how quickly it adapts to a new problem, a new a new distribution and does well in it And so ArcAGI actually went through the batch with me in winter 26 and we helped Greg you know look at this and like the amount of thought and attention as I mentioned last time that goes into these these these games to make sure they're all orthogonal skills from game one to game two so that it's isolating this fluent intelligence measure really, Claude, Opus was one of the the first that was actually verified on the the holdout on the private that no one else has access to other than Greg and Sholay and the best that they got was 30% And just with, some harness this thing that doesn't deserve any research just some wrapper and some scaffolding we can get to 95 and Avio from NVIDIA got to 100% so Prime Agent and NVIDIA which both recently just came out And the other thing I want to add so when, Carpathy launched his auto researcher thing in March I want to say it was I forked it and I was playing around with it.

And all I wanted to do was make like a little user interface to kind of see what's happening and track it And And I ended up building a harness by accident I didn't mean to but it was just like I wanted to see it And, and basically what it is is you specify a purpose And in this example which is actually a true one I gave is like diffusion LMs don't beat AR LMs But maybe if I ensemble if I shard the diffusion LLM into a bunch of different ones because there's such high arithmetic intensity per GPU on a diffusion model versus AR that I can actually in aggregate, by sharding them, I can get better better results And so I just give this to a as the purpose to the to my, to my suite of agents my swarm of agents.

I give it some seed ideas I want to vary the ensemble size shard at different amounts A hundred times, ten times, five times, whatever And then I specify a Val metric and maybe I want to do GSM8K and like GPT-2 setup or something like that And then I have a scoping agent that will kind of look up papers that have that are similar, look up GitHub repos We'll give it to a PI agent named Chris Ray who will then give it to a research agent named John Southcott There he is and then he'll work on it And then he'll start doing some stuff give it to a counsel for some help That's where I come in and me and Yasser will give you some feedback And then you work on it a bit more And then whenever you're ready and Chris Ray will kind of keep tabs on you keep nagging you every every hour or so then it goes to this author agent to say okay, now start, stop, freeze the idea, start writing ablations and and then start writing the paper And so then what this has turned into so this is the scoping agent and then you have this cockpit that kind of you can view from anywhere We do the tail scale up so you can actually view this URL from anywhere and see how it's progressing and and talk to it right there it can send you email updates Everybody, the whole 1B thing in here And then it actually starts publishing some papers And like, I've read the papers They're actually like good.

They started out in March April, like not so good like, you know, May, whatever But now I just basically give eight ideas to eight H100 nodes and each one has eight H100s and they just keep going And then I check in and they give give me back papers. And it's kind of wild what we're what we're dealing with there And what's changed really is just the scaffolding that you put on top of it to allow it And I I didn't spend an enormous amount of time on this but this is largely how I I do a lot of my research at least the, the initial idea and so, yeah so I'll just give this six different ideas of things I want to try and just leave it alone and let it rip And so the things that we can now do just because of harnesses on the same exact weight file is just wild So I look on I spent the, the weekend just in prep for this reading a whole bunch of the classic literature from self-refined to reflection to Voyager to all the tool former And I just wanted to just do like a my, my shot at a five minute like history, how did we get here? cause I think it was it's kind of important.

And I I don't think a lot of people that are entering AI now kind of like have the context So this is not going to be in chronological order. It actually doesn't make sense to to to teach it that way or show it that way I'm not I'm going to skip a lot of papers and I may coarse grain the papers absurdly So apologies So the initial harness the GPT-2 February 2019 is a walnut end of end of sequence loop. That's it And then we basically have top P sampling and then we have the environment and that's the harness And there's really not that much There's no tool calling there's no skills, there's nothing like that And so that's the V0 harness And so basically, if you have this GSMAK example you're a math, remember this system prompt you are a math teacher This is like the persona stuff we used to have there's a context, Susie has five bucks she spends three How much does she have now?

And then there was just no chain of thought It was just like four hashes and two and a same and a sequence That is, the GSMAK format We measure, what happens after the four hashes We get accuracy and you get a plus one or a minus one if it's wrong And then the entire last six years has been giving more functionality into the harness into a static harness And so we said okay, well, what if we give that back into the context a bunch of examples like that?

Like, well, here's an example And let's say now we'll switch it to four and one And hopefully we'll have From that previous example we'll say, oh, okay, this makes sense I can, we'll help to learn And actually, this is with a few shot learner's paper back in July and 2020 And then we had chain of thought and says well, directly predicting hash, hash, hash, hash two is maybe difficult. What we'll do is we'll smear the the compute, the logic over many more tokens and we'll train it to get to that two not rather than just output the two And so that was the chain of thought idea very cool And that was all context innovations and, and output space innovations action space innovations and then then we came up with these tool former and WebGPT.

WebGPT actually came out first then tool former, which is this idea of giving it tools and it can call a tool That's just really just adjacent object with a bunch of things specified But in this example it was just, let's say, subtraction Instead of me calculating in the weight file what is five minus two? I can just call Python and call sub five three and it'll tell me two And so that's pretty cool And you can expose The tools in the system prompt and that's where tools come from And then we figured out MemGPT which is a big one of the coolest tools is being able to read and write to your own context And so before then all we could do is just append append, append, append to the context Now we said, what if I actually give you create read, update, delete on the context itself almost separate just this one little chunk called a memory that you'll be able to update And then Voyager said okay, well, we have these tools but like what if I want to chain together these tools to achieve a task?

And then I learn it and then how do I distill it back into the system prompt to make it learn forever? And this is the skill where skills kind of came about And they did this on Minecraft and this is the Voyager paper and it's very cool paper but this is like largely now what a skill is. And so I have this skills.md and I have a name of it that I can go and search and here's the procedure. And then intercode this idea of like again, in action space innovation if I can actually output code that I can basically now I have on-the-fly tools or on-the-fly skills I actually don't know if it's considered a tool because a tool is to an API Am I outputting a function?

Is it a tool or a skill? I actually am still unsure And then the react came first then self-refine, then reflection But this idea if I have multiple agents that have different roles and they can help self-improve on the context then I can get smarter and smarter So I'll take some action And this is examples I flipped the three and the five Whoops I sent it to an internal evaluator to say is this right? No, it doesn't look right I can either keep being here or I can go to the actual environment to get back a reward signal I go here to the reflection They'll say, hey, you actually flip these two go back, and I can improve the result And this is one of the first ideas of being multi-agent letting the agent reflect on its own output and improve it And then this idea of multi-agent goes even further where I can actually spawn one of the Repos could be I can spawn a set of sub-agents and they persist in these persistent repos and they'll be able to be running and I can interact with them and I'll have the sub-agent list I can invoke those ones and I can keep adding to the launch sub-agents And an RLM, what even crazier to allow this in a recursive fashion so that, and I exposes this RLM query so I keep recursively calling the RLM query to solve a larger class of problems and at the leafs whenever I want I can call the LM query to spawn that LM agent as well And then I have this main orchestra agent that's running all of that And that's what I call like harness V1 This whole thing is like static harnesses I'm not improving on the system prompt I'm not updating the harness itself And this is this you know, of course we're going to have to have G stack on the top.

He's the one that lets us do all this. So we thank you Gary Love G stack But there's some other ones that deserve a call out as well And basically what that means to summarize all this you have some agent spec you have some system prompt here You say, how many turns am I allowed? How many tool calls am I allowed? You don't want to allow infinite You specify a tool list you specify a skills list the sub-agent list And that's largely what the V1 is And then you put it in a loop and this can be spun on a prompt if I'm asking it to do something in my Slack channel which we'll hear about QM which is like I use it every day It's the team that made it is here It's super exciting And or it's on a cron and just wakes up every hour and it decides to do work just like you know, anyone else And so there's some session management There's a loop.

There's this context compilation We're actually creating the context I'm putting all that into an LM call I get back the action and then I may or may not have some tools that I need to invoke and append back into the context And that's basically harness V1 And the cool part this is the most exciting part where we're seeing a lot of advancements where you're letting the harness itself learn Either we're learning the system prompt or we're learning the harness itself which is very trippy And so one of the famous ones that I actually wanted them to talk but they're actually running a 150 person DSPY meetup tonight in San Francisco.

They couldn't make it but I've done some podcasts with them before They're a super great community Is this DSPY. So demonstrate search, predict They have this idea of basically taking a train set a small set of examples and then learning the optimal system prompt So I keep iterating iterating, iterating I can't back prop through that process but I can do something called genetic programming where I'm finding candidates I'm merging candidates with some merge rule I'm evaluating, seeing what happens And I keep working and working And it basically gives me crud over the system prompt itself and allows me to choose any system prompt And then Darwin-Godel machines actually go a step further Not only are you allowed to change the system prompt but you're allowed to change the harness itself the harness code that is actually running And so you can imagine basically what happens is you have this archive Of many different, agents, which is harness and system prompt and you sample from them you push them, through, you'll actually evaluate how how they did on some fit fitness function you'll, add it back into into this archive state And I skipped over the the self-modify.

You can actually you, you have a, a meta harness that actually allows the the agent to modify its own harness so that it can become a different harness And then that basically loops around loops around, and eventually you get better and better agents over time And then the meta harness which is the main harness is to produce harnesses right? Which is a really meta concept and so this is the output space where you're configuring multi-agent context compilation and you're you're doing this and you have allow CRUD on all of it And so it just keeps adding more and more CRUD into the harness code all the green that you see here the, meta prompt, the system prompts All the agents, how many agents there are and you kind of grow this meta-harness over time And then one of the authors of this paper the lead author is actually here tonight super exciting.

You have to drive down and talk to him a lot a bit about continual harness which is, I love it And they go even a step further where one they add some extra color on the classes of memory And so they add this history thing It'll go into a bunch more detail on that memory breakdown But the coolest part I think, is for the classic RL people I see Robert back there he definitely would enjoy this dagger style online learning where you can actually update the weight file itself.

So you're actually doing a test time training on the LLM based on a small amount of examples that you just learned which I think is actually a huge huge important research direction that we should get working Anyway, that's it. How do you do? All right All right. So, tonight we have, three authors Bien got food poisoning this morning so he couldn't make it very sad But we have three tremendous authors. Seth, student under Shi, Shi, Shi Chin is that how I say it right? at Princeton researcher at Prime Intellect and the author of Prime Agent John, who is the first time we've had a a callback, presenter.

Very excited about that PhD under Chris Anazalia. and he's gonna be he's the author of Open Jarvis with Ivanka from my lab, Hazy, which is a personal Open Jarvis. And so I think it's really cool. And then for Josh and Regan who, Josh just got promoted to be head of YC Labs which is really exciting which I think it deserves a little round of applause as well we'll be talking about QM And so QM, at YC we had this general agent that we use and they came out with QM a month ago and it is like meaningfully better and is so functional and I use it every day All right, that's all I got Thank you so much Hi everyone great to be here tonight my name's Seth, and I'm going to be presenting Prime Agent which is a self-improving LLM harness my job here tonight I think is to try to convince all of you to take a very first principles style approach to thinking about how you build your harness So we're going to be very very basic here to begin with If you think about and I love the introduction, right, from, of all the background literature which I thought was so cool I think it's very complimentary the how I think about this as well but if you think about just the raw LLM itself it's actually just this sequential processor that you know, has some fixed weights and has some visible context and it's it's taking some tokens in and it's putting some tokens out to make the next decision we don't really think of an LLM that way these days.

We have a set of files that it has access to We give endless programs and tools for it to use you can even Other sessions of LLMs are going on and create sub-agents in order to do all of these very cool things But in its basics it's just tokens in tokens out. It's a neural network making a prediction The harness itself is the layer between the LLM and the world that adds things like this persistent state tools, and compute Here we have a diagram of how we think about prime agent from the human's perspective So you open up your prime agent on your computer just like you would cloud code, codex, pi, et cetera And it gets you into this agent's view And this agent's view is an overview of all the agents that you have going on for all your parallel sessions with like a very tight like tight summary of what you're using And then you can hop it to one of those and check it out There you're going to be at this root session And this root session is your basically the project orchestrator over all of these different sub-agents that it's controlling And you don't have to ask it to start sub-agents.

It will leverage sub-agents when they're useful And these are all programmatically called because this is all based on the It's a language model principle where everything is in this IPython shell All your tools, all your memories, all your sub-agents and we expose for further coordination these messaging paradigms so you can manage all of these And then these agents can directly interact with your environment which could just be like the programs or files on your computer or running any 200 node cluster maybe for your auto research Each of these agents are then backed by this persistent daemon on your computer This is so that you know when you close your laptop or you control C out of the session it's still running in the background You have to actually stop the session so that make sure that you're continuing to running And then we also have these other features we expose from continual harness where it's able to provide like live CRUD operations on all of the components that we mentioned in order to manage its memory skills sub-agents persistently and prompt its own system prompt persistently The way I like to think about all of this context that we're Is that we have this sort of almost like like here I have like L1 L2, L3.

It's like a cache right? It's like what is the most accessible information that we're working with? And at the very like fastest like readily available information you know the models to be able to retrieve that really quickly It's the model weights. So everyone always wants to get all the information in the model weights But then we said okay well maybe we don't have all the information because we don't want to have to fine tune every single time to update because that's very expensive So we have this active input context So we're using lots and lots of tokens on the input We might have some in context examples like we've seen previously in order to add to these different capabilities But at a certain point we run out of context And so the very like earliest form of harnesses that we've seen that are still used to this day even by those who say we want the most minimal harness possible is compaction It is a compaction is a very generalized tool for the agent to be able to summarize its own Context history in order to work past its context length working window.

You can think about once we go beyond like, what are directly, like, inputs and outputs from the the model here into this L2 L3, you might be familiar with the L3 which is more of the disk-back state So if you're working with a file system you can read and write from main memory. if we're at the L2 which is, I could think, at a means in between what the active context is and working with your file system you might have a live REPL, which could just be running things directly in Python an IPython shell like you're in a Jupyter notebook And all those variables are saved directly in your RAM.

Your agent can then programmatically manipulate them and run all sorts of programs directly on the information there saving tons of tokens rather than putting it directly into context. you can also create sub-agents And it's the same thing You're basically saving context here because you can task the agent with a specific set of information in order to perform some operations and report back at the end What is interesting here is we talked about compaction for the active context right? You have your context history This is helping to update it over time so you can continue to leverage this But once we go beyond this we need to be thinking about how are we doing these update?

We talked about CRUD How are we doing beyond just creating reading? How are we updating and deleting our context over time beyond our or the state over time beyond the context link? I like to think of this at the REPL as this agentic garbage collection where we're just cleaning up the variables in our state as well as like what subages could be used And then so to make sure that our RAM doesn't crash my computer laptop every day And then on top of that we have this notion of refinement where we're updating and deleting the skills and memories and prompts that are stored on your system You can think of that so that way you don't crash your actual out of space on your hard drive as well And so this very much is a here's how we express this thing and here's how we revise it over time The other perspective that I really like to think about and I really try to push because harnesses are almost going towards this like agentic operating system that we're creating is when I think of it more metaphorically here is that when you look at the raw LLM it kind of looks more like a Turing machine where you have this ticker tape and you have all these instructions that are going in and then it's performing some set of operations and going out But when you look at a harness it's looking a lot more like a von Neumann computer You're able to do these read and write operations on external memory And that makes it much more powerful and another class of problems than just what a Turing machine is able to express on its own.

And so yeah, the idea of like how do you build a good hard disk? You want it to be the most expressible thing you can imagine So some some like early harnesses before it gets into like data flywheel where the models can do themselves are very specific Plan, act, critique, do these exact specific steps Well, now the models are able to do that themselves You can imagine like we we don't have like a React loop that we necessarily need to explicitly impose The models kind of have natively figured this out But what they haven't figured out is how to you know they have to be able to have the expressibility to call compact They have to be able to have a Python REPL so they can run programs They have to have the ability to programmatically create sub-agents and access state and have different feedback mechanisms Those are model controlled expressibility features And if you removed one of those You're actually removing a capability that it won't be able to do otherwise The way we manage and I'm sure you're all familiar with the RLM paper from my co-author Alex fantastic bit work What we do beyond what was in the RLM paper is we think about the sub-agents as these persistent sub-sessions So each, the parent station can create spin up a new RLM sub-agent and each of these are then emitted they run some tasks and then they finish and report back to some end state to the parent session These are then idle.

They're still working in your RAM At any point the parent session can then send a message to one of the sub-agents to continue working and it has all that good context that you built up over time so that you're not missing information or have to reuse information that was already developed in a prior context And then, of course, you know we don't want to use a lot of RAM so we can move them offloaded in an inactive state which then can be called back at any time by messaging them in this persistent sub-agent setup I talk a little bit about continual harness which we have in a prior paper at mine which talks about cutting the entire harness state This is another feature that we want in our coding agents Leverage all of our prior history So you can imagine like some set of trajectories where they have some actions and outcomes or something happened at each turn And so we just kind of want to expose the ability for the agent to leverage all that information in order to update what the future harness is going to look like Do we need to change our system prompts?

Do we need to create some skills? When skills I think of as a set of instructions or a program in order to achieve some specific goal Memory which could just be long-term storage about things that are important as well as the sub-agent specifications that we talked about in this very persistent manner. Were there certain sub-agents that we want So you reuse at a later time because the context is useful And just having the ability to do this kind of reflection or refinement over time it's very powerful for the models to have They're not perfect at this right now but this is one of the capabilities that you want to build your harness such that it is a bit better than what the current models are able to do So then you can get those reasoning traces and use that to leverage your next iteration of model and they'll be able to handle the harness and be able to bootstrap themselves into a higher and higher performance One of the coolest features that we have in Prime Agent that we've had Since the beginning of when I was working on this this is one of the first things I added is the ability to message between any any two agents within like some nuclear family setup parents, children, siblings And the reason why I did this is because I was I was constantly trying to figure out what's the best way to like myself to manage all of the agents I have doing everything for me in five different directions five billion different directions every day And it would be so much better if they could just like share their context directly with each other and coordinate And it turns out that's fantastic for like typical software engineering and long horizon jobs as well The last thing that we look at when it comes to how did we want to design our harness is we were really thinking about long horizon performance I want to go run some jobs and I don't want to have to babysit my agents the entire time And when, when I'm ready to come back and check in I can check in with them and see what's going on And this is a perspective that I also really lack seeing in a lot of the evaluations that we're looking at A lot of times if you run a model for nine Time or say oh, well, the model stopped working after this amount of budgets but then this other model kept working with using more budgets.

Well first of all you're not even using the same fixed expenditure to compare the models But second of all that could also be hiding performance that you're missing The way that I look at long horizon performance eval is that I want to see what's the practical plateau At what point will we only get incremental gains in performance as I throw more test time tokens at it? I have a couple of experiments that I'm going to show after we've shared design philosophy here about how we created a prime agent We're going to talk a little bit about test time scaling and our RKI results as well as looking at you know, does is it actually helpful and why is it actually helpful for our information management for the REPL that we're working on these long contexts?

And then when we have these really really long, like almost ultra horizon long horizon tasks how do we sustain these like multi-day work and like what actually goes on when we have these refinements over like these settings that can last like a week at a time? So this is a result that you've probably all seen. We actually have one additional data point that we added here that we didn't include in our original result just to compare across harnesses We solved this We went out we're trying to figure out what is the best evals that people care about these days when we're running our harnesses And we're like, oh, we should do ArcAGI It's like, oh, yeah, yeah I remember I ran some results with Continue Harness and we got 20% with Gemini Flash Sorry, Gemini Pearl So I think we can get at least 20% People think that's really cool that our general harness that didn't even wasn't even structured for ArcAGI did really well So I went online I was like, okay I need to find a good system prompt because I don't want to make sure that we're losing information.

So I found another community leader board called Prolong and I just grabbed their system prompt And I was like okay, I'm going to grab their system prompt forget the rest and I'm just going to throw this directly into Prime Agent And then I ran this and I was like oh, my God The first run that I got it hit 99.9% And then I looked at the logs and I was cheating. Okay. So I was like okay, I got to do proper sandboxing here Like let's set this up properly and then so I spent another day on this and that and then I went back and I was like oh my God I got 78% with GPT soul Like this is going to be a great result and then I went back and it's like oh, let's let's compare a couple of other ones And so it, again, we just took the prompt a general prompt that's basically says use a world model to solve RKG I3 here are the actions that you can take you have, and then the general system prompt for prime agent which is like, you have a REPL you can call sub agents you can use the it programmatically and, and so we went through I went through the traces and it's basically doing a bunch of different like calls of the coding in order to like check out these different scenarios and analyzing the images and doing like image processing And it's a lot of really cool stuff that it seems like it was doing reasonable reasoning while leveraging the the, the REPL that we had as like one of the main things that was able to enable this so I went through and I ran a couple other ones.

We did GPT-Terra 25.7% which is really cool. You can see that compared to like what were the the, like the week before we did this OpenAI was like, guys the harness matters a lot when you're doing evaluations. We used the responses API This is the result that we got and we, we ran Terra and and got almost like We we didn't run to completion this one but we we got really good results in comparison And then we go that that we're already achieving higher than some of like the GPT soul extra high which was crazy And then we went and we did Opus and hit 95.5% We're like that's insane.

We also compared to a lot of the other harnesses So some people asked me like did you run this with Claude code? I did unfortunately, the results weren't very good and so rather than having bad results I just deferred to the the original Claude results and some other people have run it with similar configurations to Prime Agent and gotten much better results since then but what's interesting is that a lot of the really popular harnesses don't necessarily do well when Prime Agent does well.

So like for Hermes Agent we spent a lot of money very quickly and we had to cut it off because it felt like $5,000 without making much performance not saying this is the best they could do but it costs a lot of money to do so so I think that the cost to performance ratio is very important. And one of the things that does save money is being able to Programmatically work with your context we ran a bunch of long horizon evals as well, like Oolong and some coding emulator bench, which is going to come out soon which is a program bench alternative And we found that it was mainly parity or slightly better than these other harnesses like you, across different models versus doing like PyMono Cloud Code, Codex with GLM 5.2 Opus 5, and 5.6 as our setting another one I thought was really cool is we have this like A program bench alternative called Emulator Bench where we're trying to reproduce entire emulators of computer systems or in this case creating like a Game Boy Color and check that out.

And we found that what's really interesting is because it has this REPL access in the RLM it's able to use these programs in order to kind of do these like out of experiment loop designs in order to try things out in a lot more expressible and free way before submitting the final solution to the grader. we also tried this with GPU kernels, and we got about parity results across, different, both Sol and Kimiko One does better, one does worse about parity which, so we, we're not overfit to like any one particular evaluation here what's interesting for the long horizon stuff is we had some auto research experiments that we did with the nano GPT speedrun but we scaled it up We said, let's give it, eight eight by H two hundred for a week and see what happens And you might be like okay, prime agent is going to be so much better right?

Cause it's able to do all this programming it's a little high variance We can't attribute any of the benefits to the harness versus the model there because it's a very hard task But what we can do is inspect a lot of the behavior that we see And what's really interesting is that we're seeing models like DeepSeek V four GLM five point three and Kimi K three you can tell these were done a little more recently than our first results and we took these and they were doing like what we call out of loop experiments So we were trying to say how can I run experiments on like the CPU and like look at the parameterization and they do hyperparameter search and analyze the data so that I don't have to spend like all my time running expensive H two hundred experiments cause that takes the majority of the time So it's it's running experiments that are not the main Experiment in order to optimize them I think that's really cool behavior that we're seeing as we shape what would be what kind of things we need for the expressibility for prime agents so you can use like really good auto research Because you can imagine if it's good at auto research it'll be good with you it'd be even better with a human in the loop to bootstrap your experiments And finally, we also streamed a seven-day factorial run which used a total of 633 agents across 23 million output tokens in order to make like steady technological enhancement across the tech tree to continue to progress over time And here it one of the main benefits is they can use like the sub-agents that can divvy up into different tasks in the factory in order to research and build and gather resources and build the next items to design the factory as well as it can use the refinement to leverage what happened in the past in order to help in the future over these very long contexts so it doesn't get stuck And one of the most interesting things here Is that it does not get stuck and it continues to make technology progression even at the end of our stage This is more like a Gemini place Pokemon kind of conclusion here.

If there's one thing that I find interesting today but like what takeaways you should actually add to your own harness I think that you should think about agentic context management You should think about swarms and looking into further depth RLMs and trying to run standardized evals All of the results that we can they showed today can be run with our verifiers package that we have at Prime Intellect And shout out to my collaborators who are fantastic and I love working with Thanks All right, next up we have John Hey everybody I'm super excited to talk about a project that we've been working on at Stanford I've been working on this with Ivanka and Ryan my co-lead author, as well as our advisors Azalimir Hosseini and Christopher Ray So Personal AI is everywhere but it's mostly cloud-bound today We see lots of different harnesses and projects focused on making daily writing research, coding, and scheduling But projects like OpenClaw and Hermes Agent typically rely on cloud LMs for most of the intelligence and for most of the queries What does this mean?

It means that it's pretty costly You're getting thousands and thousands of dollars in API costs if you aggregate it over a year It's not private. You're often sending your most personal data to LMs up in the cloud and you don't necessarily know where all that data is going It also requires you to rent your intelligence as opposed to just simply owning it out of the box And finally it tends to consume orders of magnitude more energy than just running these LMs on your laptop And so the local LMs are finally good enough to actually run a lot of these queries that people care about And so we see that the current LMs of today are only six to 12 months behind whatever Was the state-of-the-art frontier models of before So you see, LMs today, such as Qwen 3.8 27B that achieve roughly the same performance as like Cloud 4.6 Opus back in the day.

So that was kind of the state-of-the-art model back in August 2025. and that gap seems to be closing more and more as the hardware accelerators that we have for our laptops and for our workstations get better and better just this week, we saw a new release from Apple with the new Mac Mini And so we're seeing this renewed focus from Apple as well as NVIDIA to build accelerators specifically for personal use cases And so with this project we wanted to explore the the main question of can we build the core of a personal AI stack namely the model inference the agent execution, the memory, the learning basically the parts that are mostly reliant on the cloud today entirely on device while staying competitive with these cloud-only stacks And so we decided to propose Open Jarvis name needs, no, needs no explanation Wanted to explore just how much of this we could run on device completely for free while preserving security privacy, and quality.

And so to construct Open Jarvis we wanted to create the simplest set of primitives for which you define any sort of harness or personal AI stack The first one is whatever user interfaces you need to use The second one is the actual agentic logic around composable reasoning and using different kinds of intelligence and tools For the intelligence it's whatever LM you're using as your engine for keeping everything going So this could be Qwen GBD-OSS, Gemma 3n And then whatever actual inference engine you need to run it So this could be Ollama Llama CBP, VLM, SGLang including whatever hardware you're running it on So this could be Apple Silicon NVIDIA, whatever you need For actually making all of these agents and intelligence useful you need some set of tools and memory that can be run through a standard MCP protocol And you need some sort of set of primitives for actually doing learning whether it's prompt-based techniques Like Jetbar or DSPy whether it's way-based techniques like GRPO and SFT and LoRA you need some way to actually get the agent to improve over time and actually be able to make it more personal and more effective.

And so to kind of walk through like what Open Jarvis looks like we tried to go with all of the standard interfaces that people are already accustomed to So we wanted to give people the ability to interact with it through a desktop and actually just run it as they would normally expect but then see all of the savings that they're getting in terms of dollars and energy We also wanted to give people the ability to run different kinds of continuous agents so different kinds of agents that are persistent in terms of cron jobs and being able to run standard protocols day after day Basically we just wanted to make this like plug and play with all of the workflows that people are already accustomed to running And we wanted to make this something that can get people to have their first experience with LMs on device the same way people had their first experience with ChatGPT or Claude back in the day.

So yeah and so yeah, to step through a little quicker but yeah here's like a nice way to like set up new persistent jobs We have all of these different components We need some way to actually optimize it And so we wanted to get out of the way of the LM as much as possible by just creating a simple spec of these five primitives by which they could go through the optimization And what we found is that by going through this whole optimization loop we were not only able to get significant dollar costs dollar cost reductions but also significant latency reduction and significant improvements to the overall quality on these tasks And so this configuration is meant to simplify down to just the five main things that people care about when they're building these LM harnesses So the intelligence, the engine, the actual agentic logic around it the tools or learning systems required for running it and the whole optimization for the whole spec as a whole And so something that we thought could be interesting to help bridge this gap between local and cloud LMs is to actually have the cloud LM go through and automatically optimize whole LM, the whole local stack And so this is a nice way of taking advantages of the capabilities of cloud LMs to diagnose propose changes and gate to create improved solutions for these local LMs while not incurring the cost of those cloud LMs when you actually deploy these local stacks at inference And so what we found is that these open Jarvis jobs that were these open Jarvis configurations that were actually optimized by cloud LMs like Cloud or ChatGPT were much more effective than local stacks that were just deployed out of the box because you could actually cater to the specific LMs the specific harness that was needed for your different kinds of workloads And what we found is that even with the on-device LMs of today we can rival cloud LMs on different workflows around personal AI personal use cases, coding, agentic tasks While there remains like many tasks for which like local size LMs are not enough the gap is surprisingly closing After a month as these LMs become better distilled more effective and also we get better accelerators for running them.

And so even with the LMs of today we can get 800x lower costs in terms of actually running them as well as a significant reduction in latency What we also found is that no matter which cloud LM we picked it was useful in terms of optimizing the whole agentic loop for these local open Jarvis configurations We found that the Opus series the Opus 5 as well as GBD 5.6 Sol were naturally the best But what's interesting to see is that you could pick Gemini you could pick other larger LM families like Kimi and GLM and use them to optimize these local configurations so that you could capture those efficiency gains capture those performance gains for local inference later What we also found is that the whole Open Jarvis harness was cheaper to optimize than alternatives which might require more data and more LM calls We found that like this set of specs and this set of primitives was most effective for local LM settings because it got the whole optimization loop and the whole set of LM abstractions out of the way of the cloud LM to just optimize the whole system and just make it make it really fast and really effective Looking forward we're excited to keep building out this project We think in the very near future you're going to see a huge a huge proportion maybe even a majority of people's daily inference calls going to local devices and on-prem laptops or on-prem workstations as opposed to the kind of standard of today where everything's being pushed up to the cloud We think these trends are only going to continue because the accelerators keep getting better and the LMs keep getting better And so if you're excited about anything in the stack whether it's better local LMs better accelerators, better inference engines for deploying beyond data centers please reach out.

We'd be excited to chat Include a QR code of the project If folks are around here afterwards we'd love to chat. Thanks All right, now we have real Arlen YC We have Josh and Regan Hi, I'm Josh, and, this is Regan and we're working on QM which is YC's open source agent harness for work QM is one system that gives every employee at YC an open claw-like assistant that's like fully customizable and available in Slack or via web UI which is, what we're looking at here each person works within QM in their own personal context that has its own sandbox files, and crons, and it can al they can also work with QM in a multiplayer setting like a sock channel People use QM for a pretty broad range of things like a lot of automations like email triage, like legal and finance workflows It's really good at editing documents and pulling data out of our internal database it can also spin up live internal web apps and help with stuff like planning events but it's designed to be broadly helpful for the range of tasks that someone might encounter at YC on a day-to-day basis So, you might be wondering why we built this And it's really the result of a string of internal Asian projects that have kind of unwound throughout the years and all of which were really riding this tailwind of increasingly capable models The first one we built was in like January of 2025 We internally refer to it as the quote unquote like general agent but it was pretty straightforward just kind of a system prompt with tools in a loop It was one size fits all sort of like everyone was talking to the same thing and it was pretty straightforward architecturally but like still very or surprisingly good at answering data questions interestingly like the scope of what the general agent was good at just increased I guess unsurprisingly as the under underlying models got better and we eventually hooked it up to Slack We added crons and gave it a a few more tools so that it could be more it'd be a capable across more domains in June of 2025 By then like a lot of our engineers started using Cloud Code and Codex and we realized that you could pretty easily run these in a VM And then, we hooked that up to a Slack tag which was a pretty powerful medium for people who just wanted to like, run a one-off code change we also configure it configured it to run our CI pipelines and then spin up dev environments for testing and so people could come in like, describe a bug or something they wanted to see happen and the bot would go off and actually solve it which was, like, a pretty powerful, thing for someone who like, maybe hadn't made a code change before in their life even but we also, on top of that had a small loop going where we would observe sort of how the bot failed where it went wrong and then update the agent.md which was present in the codebase at the time To make sure that the thing got better as as, we like observed the usage And so in January of this year a lot of the partners started using OpenClaw And one thing to know about YC partners is that they're incredibly busy between like office hours they get tons of inbound email they're always reading applications So like any tools that can give them additional leverage are incredibly valuable to YC So OpenClaw in particular was useful because it was the first agent that a lot of them had used that had their own computer that had its own computer And so this made it like very customizable in a way that the previous paradigm of agents was not and it functioned almost like a personal assistant And so in April the question became like could we provide this to every employee at YC without, like buying everyone a Mac mini effectively?

And so we ended up provisioning a fleet of like 50 plus Hermes agents that were running in VMs And these were definitely pretty helpful but they required a lot of configuring for people to get value out of them And it was just like inherently kind of difficult to manage this fleet it was sort of like a whack-a-mole situation where I would have to sort of like SSH into these individual instances and fix them And so the follow-up question became like we've got we've gotten a lot of value out of these agentic systems like let's build something that tries to address some of the downsides of running this big fleet of Hermes agents while still maintaining the personalizability and some of the like the stuff that people were really getting value out of So yeah, okay.

So basically, there's a pretty clear trend from that you can see there from what Josh was showing you Basically, the models are getting better exponentially and we were starting to see just increasingly impressive returns from giving them more and more capabilities So OpenClaw gives the agent its own computer and we start to see really impressive returns from that So around May this year we started thinking just like how far can we push this if we just keep pulling on this thread? we really like the this lens of sort of unhobbling I don't know if you guys read Situational Awareness when it came out in like 2024 but that was kind of an era when like test time compute was just starting to become a thing.

And like you know we were starting to give agents tools for the first time And there's sort of this intuition that what agents can do like there's a little there's kind of more intelligence in the models than we're than we're using in a lot of cases and it's like if we really push the frontier in terms of just like what we're offering up the agents as as capabilities. abilities that they can make use of like magic can start happening so the first way that we do that with QM is by essentially pulling the brain of the system up out of the sandbox So with, you know, with Hermes and with OpenClaw you effectively have, the agent has its own computer which is super powerful but it's also trapped inside that computer So that causes a few issues just from you know, us trying to administer that system whether, you know, even a few dozen of these things it starts to become unwieldy almost immediately but the other issue with that is that you all, all of the, like all the sessions that you would have they're trapped inside that computer And so what we did instead is we we just offload everything into Postgres So everything is centralized from all the agent conversations that people are having and then we expose those to the agent itself.

So it can look at all the context that's sort of aggregating from across the system And then the other thing we do is we start thinking about Sandboxes less as this home where the where the agent lives and where it's kind of stuck in a lot of ways And sandboxes become more of this thing more of a resource that the agent can dip into and use as needed but it's, it's, it's a lot less limiting the other thing that this starts to open up is this idea of you're accumulating this large eval set of all the traces that you have from the conversations that people are having with the agent And in principle you can think about going in and hill climbing on that it's sort of having this automated improvement loop we've had sort of mixed results with that I would say.

I think typically if you're just dispatching this like torrent of agents that are supposed to fix all of the bugs that they're encountering when you have the LLM as a judge you start to get this kind of like main character syndrome where the agents are making fixes that are you know, only seeing their, their their piece of the elephant effectively Like they're, they're really, they can be sort of yeah, not, not seeing the whole whole system. And so having the human in the loop there has continued to be Really important Although we're really looking forward to this working, all the way around so the other major thing we do it's pretty obvious is just wire the agents to all of the resources across the company that we can we already happen to have a CLI at YC that worked really well that wired a lot of systems together but anything that wasn't in there we basically allow, adding just arbitrary API keys that sort of thing.

And then we also want to ensure that we have parity with just an employee working on their laptop So, you know, device code off, OAuth we go ahead and ingest that into a keychain and then refresh it for you. So it's it's ideally supposed to imitate the experience of a person on their computer We mostly keep this to to be read-only, in the database but we do allow for writes via, human-reviewed bulk upstarts So the way that works is the agent will put forward a plan to edit the database that a person can Give a once over and ensure it's not doing anything crazy before the write actually happens One thing we've observed with this is that we've started just kind of rubber stamping these It's a little bit like I think if you guys use cloud code in the early days like you might have been reviewing the tool uses very closely And eventually you start to build up more trust in the agent.

So this is something that we're looking at very closely over the over the next few months Yeah, so sort of like I was saying the sandboxes in this system we like to think of as a resource for the agent So and unless where the agent actually lives So in QM, the agent can basically by default it's going to be using a particular sandbox So that's going to be one that's been allocated to the user that it's talking to But in general there are there are environments that the agent can kind of converge on and and can collaborate with And then the other key thing is that if the agent is working on like a like a heavier dev workload it can go and reach For a machine that has more resources if it's working on something that's simpler it'll just go for a sandbox that's less powerful And so pushing that decision into the agent itself rather than the harness has been a really really powerful thing Similarly allowing the agent to tap into its own runtime So basically if it can choose the provider that it's working with you start to get out of situations like I'm sure you guys have run into this with Fable If you try to do AI research if you try to do cybersecurity anything, you'll get a bunch of refusals So what we can do when the agent can control its own runtime it can just pop out into another model when it needs to avoid a situation like that Similarly, like in the earlier situation it's often useful to pop between different sandbox providers And so that's something that we can do quite easily And as a general rule what we've tried to do is keep the harness extremely thin We think of the kind of core of the system as being these three tools where you have execution in a remote sandbox reading and writing from object storage and then publishing internal apps a pretty simple sort of get-backed system And then we have other tools for interacting with memory and crons and that sort of thing but we really think of these as of temporary papering over rough edges in the system And really the core of it is is these three up here it's, we we really try to keep it as small as we can Yeah, we're, we're sort of trying to be this like AGI anticipating harness, although, since we aren't there yet there are a few things that we've run into one of these is that the agents have been we've tried to put them in this really capable environment where they have all these tools available to them but, they often give up way too early so one thing we've experienced with especially over the past month or so has been setting, we call it like a grind tool or basically we set budgets on goals So the agent is not allowed to give up on its task before a certain amount of like walk clock time so like a couple hours or a certain amount of token spend And so, what that can accomplish is just like really a lot better you know, research outputs, better reports that sort of thing and it's been really fun actually to see like OpenAI and Anthropic you know crack some open problems in math with like a very similar technique but it also works for just you know, normal office work stuff too the other thing that we've seen a lot of is so this harness is supposed to work it works in multiplayer it works in Slack but because of the the artifacts of its of its training, effectively, what we see is that the agent gets can get very confused about the situation that it's in even if we specify this pretty clearly in the system prompt So having like local affordances for this has been something that's been that's been really important So yeah The other thing that has been a problem is that agents really don't understand social contexts To make this a little more concrete like if I tell Regan a piece of information he intuitively sort of knows or at least has like a good mental framework of where it is okay to share that information But it takes some actual work to recreate this with an agent Like privileged information can very easily just leak into these contexts where it should not be And so the information that you can put in the brain is effectively like bounded by how good your permission system is And so YC luckily has an existing software system with like fine-grained permissioning that has been built over the years but a lot of people just don't have that And so it takes work to allow for knowledge sharing in a nuanced way So thanks everybody for listening You can try out QM it's open source And coding agents are pretty good at standing it up If you run into any problems feel free to put up an issue and we'll look at it We're also hiring.

So if any of this resonated with you or would be exciting then feel free to send us an email