We won, what now?

transcript

Thank you so much everybody. Welcome to Bug Bash. Uh those of you who are new here, welcome. Those of you who are old here, welcome back. Uh I want to just emphasize something that Akshay said because I think there's a very common misconception about this conference. And the common misconception is that since this is a conference hosted by Anthesis, which is a software testing company, at least for now, therefore it is a software testing conference. And that is not actually true. This is a conference about building reliable software by any means necessary, however we need to achieve that. And you know, using whatever techniques we need to make that happen. My slides are not advancing. Ah. Okay. What's the Should I just tell you when it's time to advance?

Should I just try pressing this? There we go. Okay, great. See, reliable software, very important. Um And so basically, I think that this task is hard enough and important enough and the tools we have at our disposal are primitive enough that we need everything. We need to bring together lots of different people who are using lots and lots of different techniques from different communities that maybe historically don't talk to each other that much for kind of dumb reasons. And we need to take the best of the ideas from all these places in order to achieve this goal. And so that's what Bug Bash is about, making these conversations happen, bringing you all together, and you know, I hope it's I hope it's really great. But something is a little bit different since last year.

I don't know if the rest of you have felt this. There's a new contender on the field. Nowadays, you can just go and you can ask You can ask your friendly neighborhood Eldridge abomination to make your to make your code better. And I think a year ago, people were very skeptical that this would work because basically what this amounts to is taking a fundamentally unreliable system, which is what AI is, and trying to use it to make software more reliable. That feels almost paradoxical. And yet, actually, to the surprise of many, this does work quite well. There are a lot of people who are trying to do this in a lot of different ways. And so I think we need to add a new Palantir.

Somebody else is joining the party. And I I think it's going to I think it's going to fit right in. Um But But But But kidding aside here, like people are actually using AI in all kinds of extremely creative ways to increase software reliability. People are using AI to auto - formalize programs and then prove things about them. People are using AI to drive up their test coverage in all kinds of creative ways. People are using AI in the observability space in all kinds of creative and terrifying ways. And then, almost most surprisingly of all, people are also just asking AI to improve their code for them. And this is the one that I would never have told you would have worked. Because AI is, on average, actually very, very bad at writing code.

And yet, there's this crazy paradox where it can actually make your code better. And the person who first clued me into this was my colleague David McKeever, who's sitting right over there. He coined a term which I love, which is vibe quality. And the idea of vibe quality is that when the cost of something goes down drastically, and maybe the cost, you know, maybe that thing is semi - automated refactors or cleanups that touch every file in your repo, or, you know, whatever random yak shaving project you have, you can do a whole lot more of it. This is related to Jevons' paradox in economics. And so, the really cool thing is like people are able to do this, too. And I think this is a very exciting idea.

I've now seen it work with my own eyes. Um and I think there's going to be a lot of people today and tomorrow who are here to talk about that. So, one more difference from last year and then I'm going to get to my actual talk. One of the most controversial discussions we had last year was about software infrastructure versus physical infrastructure and whether software engineers are just fundamentally bad people because, you know, because physical infrastructure doesn't just randomly collapse, buildings don't just randomly fall down. I thought it would be really great to get some actual experts on this topic to come and talk to us this year and so we have two new people Brian and Deb who are both actual experts in physical infrastructure and the built environment.

Um Deb is the author of How Infrastructure Works. Brian is the author of the excellent construction physics Substack. I'm super super psyched to have them here. So, if you thought last year was a big tent, if you thought it was ecumenical, this year will be more of both. Uh, it's going to be rad. I hope you enjoy it. Okay, one last thing before I get going. Um, for the remainder of this talk, I am mostly not going to be speaking to you from my perspective as an Antithesis employee. I am mostly going to be talking to you as somebody who has cared about all of this stuff for a very long time and who has worked at a large number of different companies trying to build reliable software.

And so, when I use the word we in this talk, which I'm going to do kind of a lot, I want you to imagine that I'm including all of you. I'm talking about we the software correctness community, not we Antithesis employees, okay? Um, so I want you to imagine that we, all of us, are fans of this really really niche band. This really super this super weird musical act. Um, you can tell they're niche because they still put out cassette tapes. Um, so imagine we're all fans of this niche band. Nobody else has heard of them. We all love them. What's that feel like? > > It's awesome. > > Yeah, it's great. Means we're ahead of the curve, right? We get something that we have excellent taste.

Nobody else understands like how great this is. We're special. This is like a great substitute for having a personality. Uh You You've always got something to talk about. Uh you know, it's true. Maybe some people get tired of you talking about this all the time, that makes you kind of a weirdo and a misfit in certain circles, but if you meet somebody else who's into this thing, man, it is very energizing. You have something in common. You're not just a You're not just a group, you're like a little cult. You're like a little You're like a little conspiracy against the world. Um that's a very special feeling. And it kind of makes up for the fact that you're a bunch of weirdos. Now, I want you to imagine what happens when this band that you are a super fan of suddenly becomes freakishly popular.

They are number one on the global charts. They are sweeping the world. What does that do to your day - to - day lift experience? In some ways it's really great, right? This is probably a tremendously validating moment for you. You were into them before they were big. Now everybody else has acknowledged that they're actually amazing. That's That's cool. It's true that you now need to get a new personality, right? Being Being into these people no longer makes you very, very special. Um but, you know, maybe that's maybe that's okay. By the way, you can probably also cash in on this in some ways, right? If you like maybe you will get hired by record labels as a talent scout to find other great musical acts before they get big.

Or maybe you can make a lot of money as an expert in this band because you were into them early. Or maybe if you own, you know, their original T - shirt from when they first WENT ON TOUR. MAYBE YOU'RE GOING TO BE ABLE TO SCALP THAT T - SHIRT for a lot of money. Um, that's really cool. You know, also, there's probably going to be a bunch of other people who are producing this kind of music now. And if you're really into this kind of music, like, you know, maybe these guys come along and they're not they're not quite the same thing, but you love this sort of music, they're similar enough. There's now a whole lot more demand for this thing and so there will also be more supply.

That's also an advantage. And by the way, if you thought that it was only a few people who love this, now there's tons of people who love this. You'll be able to find others who are into the same thing. That's really, really great. You know, suddenly you're having a whole lot more conversations with people who are into this thing. But there's a problem. Suddenly, somebody being into this thing is no longer a very high signal fact about them. Because some people actually love this band for their music and they have now been exposed to it, but some of them just like things that are popular. And they don't actually like this music at all. And that might be a slightly weird experience for you. Um, there's a guy who wrote about this weird experience in a very different context.

His name is John Evans and he wrote this essay back in 2015 for TechCrunch called Beware the Pretty People. And it was really controversial when it came out. It was phrased in like the most trollish and incendiary way possible, which, you know, which is normal for TechCrunch, I guess. Um, but it actually made a number of good points. And basically what he said is, look, the tech industry used to be this incredibly niche thing that you could only possibly get into if you were obsessed with technology. And then it became the nexus of a tremendous amount of money and power and status. And as soon as that happened, it changed. The people that he calls the pretty people showed up, right? These are the people who would have been lawyers or financiers or bankers or consultants in a different life.

They flooded into tech because now it was the place to be. And Evans talks about how that fundamentally changed the character of the industry, and in his mind it was mostly for the worse. And I think he overstates his case. Like I actually lived through this transition, and I can tell you that the tech industry got a lot better in a lot of ways, too. Um it got a lot more diverse. A lot of the people who were flooding in had really interesting new perspectives. A lot of them understood how the world outside of technology worked in a much more immediate way than tech people. It produced a flood of new ideas, and I think without this happening, the tech industry could never have lived up to its potential, and it could never have had the transformative effect on the world that it has had.

But it is also true that Evans is onto something, right? Along with all the good stuff, there did come a flood of scammers and grifters and sociopaths. Because those people, sad to say, exist in the world, and they're disproportionately attracted to things that are popular and things that are high status. And so if your niche band suddenly becomes really popular, you might also attract some, you know, imitators, some losers like these guys. Like I don't even know what this is. Like they misspelled the title of their album. Like this is just like these people are not on board at all. They are not producing our kind of music. And if we are a tiny niche community, we may not may not have evolved the immune reaction or the memetic defenses or, you know, the the social practices for dealing with bad actors.

Okay. So I want to summarize a little bit what niche fields are like and what popular fields are like. And I want to emphasize here that neither one is good or bad. They're just very different, and the transition between them can be traumatic. Niche fields tend to be relatively elite. They're elite just by necessity. It's a niche. It's hard to get into. It's hard to find out about. And so you have to put quite a lot of effort to get into it. And so the quality of the average participant is very high. On the flip side, they tend to be kind of elitist, which is not an amazing quality, but one that often goes along with being super elite. They also tend to be kind of cozy.

This is a double - edged sword, right? Cozy is good. I know many of the people in this room. Like, hooray, we're all cozy. This is very warm and inviting feeling. But cozy can also mean groupthink. Cozy can also mean a lack of openness to new ideas. It can mean incestuousness. Um and then lastly, it can often have this really cool defiant feeling. It's us versus the world. We're a plucky band of rebels. They don't understand us, man. Um I personally really enjoy this feeling, but it like anything, if you overdo it, it can become super unhealthy. Popular fields, in contrast, are energizing. When the tsunami of money and talent, when like the firehose of money is like aimed at you, um it's like no other feeling in the world.

You will feel completely wired and energized 100% of the time in those rooms. A lot of that money and talent is actually Like a lot of it is fake. It's scams. It's grifts. It's bad stuff. You need to start to develop filters for all of this. But a lot of it is actually genuinely new and exciting as well. A lot of it is really good new ideas that the new people are bringing to your field. And a lot of it is good new ideas that the existing people in your field are having because of all this new energy that's coming in. And then lastly, it can get kind of ridiculous. Um when people feel like the money train is going to last forever, they start doing crazy stuff.

Some of it is harmless and crazy, and some of it is like no, really, really bad. And then And like 10 years later, when your field crashes and Michael Lewis writes a book about it, he will use all these as anecdotes, right? Um, so, you know, that's that's the thing. And obviously, I am not here to talk about this awesome niche musical band. Obviously, I'm talking here about the thing that all of us are doing, which is software correctness. Because you may not have realized this yet, but you belong to a cult. You belong to a tiny, tiny niche field. The man on the right always thinks he's on the left. Um No, we seriously, like the vast majority of engineers actually don't care about correctness very much.

I I I hope you all know this by now. This does not mean that they are bad people. This is true even of engineers at very, very good companies. It just means they have a slightly different set of values. And I think it says bad things about our industry and our economy that they do not feel more incentives to care about correctness, but I don't think it's on them. Um this is just a fact of life. And I think this is probably a fact that everybody here has felt. Certainly, I have felt it. When Dave and I started our company, Antithesis, this was like part of the plan. I remember very, very, very vividly writing down on a big sheet of paper a list of all the reasons that the company would probably fail.

And the number one item on the list, which I remember underlining, was most people don't care about correctness. And basically, basically, that was just we just assumed that that was true. So, the plan was to go and try and make contact with the tiny, tiny niche group of fringe weirdos who cared whether their software worked, and try and get them excited, and build a little bit of momentum, and you know, some of them would become customers, whatever, and then, when the moment was right, to launch a brutal, grinding, uphill war of attrition to make the rest of the world care as well. And then something really weird happened. About 6 months ago, just as we were getting ready to launch that war, suddenly we didn't have to.

Suddenly it felt like something changed in the world. Something something shifted and the rest of the world did care about all this stuff in a way it never had before. This was a very disorienting experience for me. Um It's not just me, by the way. Here's the Google Trends for property - based testing. That is a That is an insanely shaped graph. It's Yes, we're all laughing because it's zero up until But now it's not zero, right? And it's not actually just property - based testing. Here's Google Trends for formal verification. Um I can keep producing like niche weird software validation things that people in this room have looked on worked on and the graph always looks the same. Everybody has just started caring all of a sudden.

I can tell you on a personal level that I receive a phone call from a venture capitalist roughly once a week asking me my opinion of some exciting new software validation startup that they are considering funding. I assure you that did not used to happen. Um So what's going on? What happened? I think it's pretty obvious, right? It's AI. But how has AI caused this to happen? I think that is a good deal less obvious. Um the conventional story, the one that I have heard hundreds of times when talking to people, is that it's because AI agents don't write very reliable software. They're not very reliable proxies for human beings and so they produce software that's not very good and so people feel like they suddenly need backup.

They suddenly need some kind of powerful guardrails to keep them on the straight and narrow in order to be able to trust the results that they produce. And I do not believe the story at all. Um I don't think they're lying to me. I think they're lying to themselves. And the reason is that we have had agents that are very unreliable at writing software for decades. And they're are called human beings. Um, I can I can tell you that as a human software engineer, I am not capable of writing reliable software unless I have very very strong guardrails, unless I have a powerful type system, unless I have supercharged testing. I just can't do it. And I think that's true of most human software engineers, actually, aside from a few really truly exceptional people.

Um, and yet, our industry has just been okay with this, right? For decades and decades and decades. We've been okay with it as these human beings have produced disastrously bad software with bugs in it that have lost ruinous amounts of money, that have actually killed people in the real world. And as an industry, we are just okay with that. And then you're telling me that all of a sudden AI shows up and it is marginally worse at writing software than people, and who knows how long that will stay true, by the way. And now, all of a sudden, we've got this big problem and everybody cares about software verification. I don't believe it. I don't believe it for a moment. Um, I think something else is going on.

I think it's actually it's actually more obvious thing is going on. Um, I think it's just Amdahl's law. So, this is a this is a room full of distributed systems engineers, and so I can just put this equation on the thing, but for those of you who are not distributed systems engineers, it's much less scary than this looks. Amdahl's law is just a fact about performance optimization, which is really kind of intuitive. Basically, what it means is that if you're trying to make some overall system faster, you should not worry about the part that's already pretty fast. You should not worry about the part that's already easy to parallelize. All of the improvements are in the part that's slow. Um, and the part that is slow, the part that's single - threaded, that will become more and more of a bottleneck and that will dominate the overall run time of the system.

And I think that something like this is basically exactly what's happening in software development everywhere everywhere on Earth right now. I used to go into sales meetings with prospects and I would tell them that hey, 50% of your team's time is being spent testing, triaging, debugging, et cetera. And they would laugh at me and they would accuse me of making that up or of lying. And these days I go into the same kinds of meetings with the same kinds of people and they I say the same thing and they get like this haunted look in their eyes. And they're like, " No, no, no, dude, it's not 50%, it's 99%. " How is How is that possible? It's literally just Amdahl's law, right? Like, we used to have a process where you spent 50% percent of your time writing code and 50% making sure the code was fit for use and then suddenly Claude comes along and let's say it makes the writing code part infinity times faster and now you are actually spending 99% of your time on the remaining part, which is still slow.

And there's a couple of very striking things about this, all of which are just just literally Amdahl's law in a nutshell, right? One is we've made part of this process infinity times faster. And yet our overall throughput has only doubled. And note also that any further improvement we make in this green bar is not going to affect the shape of the overall process at all. All of the remaining marginal improvements must come in the red area. And that is why this is suddenly getting popular. That's why the hordes are heading our way. It's why the tsunami of money and talent that is going to come and totally swamp this community and transform it fundamentally is heading for us. Um and there's a couple consequences of this that I think we as a community should try to get ahead of and be prepared for.

I want to name three of them for you. The first one um is that because this is why this field is getting really hot right now. Um there's there's a sort of superficial culture clash that is going to occur, which I want to like preemptively tell everybody not to freak out about. Um so imagine that you have a trade - off between software quality and development speed that looks like this. And we all know in this room that this is a lie, right? And like Tiger Beetle has written very movingly about how actually keeping a super high - quality bar, you know, is enables you to move faster and so on. And I totally believe that, but the world believes in this trade - off.

And when you're working in a very short - termist way, this trade - off is true. And a lot of the world thinks in a very short - term way. Okay? So what are we all doing? Everybody in this room, each in our own way, each with our own approaches, our own research, whatever, is trying to shift this frontier outwards. Trying to make this trade - off less stark. That's what technological progress is. And because of our backgrounds and how we got into this super weird niche culty field, we all look at this graph and we're like, " Sweet. For the same amount of effort, I can make my software better. " And everybody who's about to show up is going to look at this graph and go, " Sweet.

I can keep the same level of quality and go faster. " And then some people are also going to be like, " Sweet. I can have some of both. " And the thing that I want to say is that this is not some great holy war that has to be waged. This is not some fundamental conflict of values. These all three of these responses to a technological improvement are 100% valid, depending on what it is that you are trying to achieve. Um these are all totally okay ways to reap the surplus that we are all going to create together. And note that all three of these camps are also allies. Like anybody who manages to shift the frontier outward benefits all three groups. So when a whole lot of people flood into the field and it seems like they want something very very different from what you want, they actually don't.

And things that they invent may end up helping you. Um I want to mention another event that happened. This one was before my time, but I've read a lot about it. So back in 1994, the internet was ruined. Um basically internet service providers did something extremely inconsiderate. They allowed anybody to get on the internet uh if they paid $ 30 a month. And this was interpreted by the people who were already on the internet as basically like the apocalypse. And if you read accounts of what happened, they were kind of right. It kind of was the apocalypse. Basically all of these communities, all these Usenet groups that had arisen were drowned in this flood of new people who trampled all over their cool norms, completely changed the character of things.

It's really hard and they they made by the way cool t - shirts like this, the internet's full, go away. Um And if you read accounts of this event, it's very hard not to feel tremendous empathy for the people who wore these t - shirts. Um At the same time, they were wrong, right? Like at the same time, the internet has done some good things in the world in the last 30 years. It has, you know, produced a lot of value for a lot of different people, a lot of different groups. So like it was actually kind of a good thing that the nice, beautiful, awesome niche thing they had going was destroyed and swallowed up and inundated. Like sometimes unfortunately that is what progress looks like.

In the same way that sometimes your cool park getting bulldozed and replaced with a hospital that can only go there is actually a good thing. And so, it is hard to feel that way at the moment that the destruction is happening. It's very, very hard. And it's going to be very, very hard for all of us. But we need to remember that looking back from advantage point 30 years in the future, we may I I I believe we will feel the same way about what happened here. Um And that's what brings me to my my third my third thing. Um this is Rembrandt's depiction of Oh, wow, you really can't see it on the screen. That's too bad. You should look up this painting. It's beautiful.

Um this is Rembrandt's depiction of the parable of the workers in the vineyard. If you haven't heard this story, it's a very good story. Basically, there's a guy who owns a vineyard, and he hires a whole bunch of people to come and work in it. And, you know, they're working there all day long in the hot Middle Eastern sun. And at the very end of the day, a bunch of other guys show up. And then, at the end of the day, the guy who owns the vineyard pays them all the same amount. And the people who've been working there since the very start are like, " Dude, what gives? That's completely unfair. " And the guy who owns the vineyard is like, " What's wrong with you?

I paid you exactly what I promised I would pay you. I've dealt fairly with you. Why are you sad that somebody else has received a good thing as well? " And I think this is like a very profound story that speaks to something very deep in human psychology. But basically, some of the people in this room have been toiling in this vineyard for like literal decades. And a lot of new people are about to show up at the very last minute, and some of them are going to become fabulously wealthy, and some of them are going to become very famous, you know, building on the work that we all have done. And in that circumstance, it will be very, very tempting to feel resentful, like the people in this story.

And what I want to encourage everybody is not to feel resentful, but to feel happy. Because actually Well, first of all, it's going to happen whether you like it or not. Um But actually like this is what winning looks like. Like other people showing up and co - opting your thing and profiting it from it is generally how winning looks. Um there's like a there's like a very, very wrong understanding that people have of winning cuz they watch too many Hollywood movies. In Hollywood movies winning looks like your plucky band of rebels blowing up the Death Star, you know, taking over, convincing the world of something important. What winning actually looks like usually is the world initially not caring about what you do and then realizing that oh crap, those guys are actually right.

And then the world comes in eats you and it digests you and it destroys your distinctiveness and you are forgotten forever. But But the world has moved marginally closer to your position in doing so. That's what winning actually looks like and that's that's that's good. That's actually that's actually us achieving our telos and actually creating the change in the world that we wanted to see, which was people caring about software correctness. Like we used to have a problem which was nobody cared about software correctness and it was a really big problem and we're about to have a new problem which is that everybody cares about software correctness but not in the way we like. And that is a that is a that is a different problem, but that is a better problem.

But it's also a problem that demands a very, very different reaction from us, a very different response from us, a very different set of behaviors from us. Um and you know, it's also kind of our time to shine. Like the masses are coming, they're going to pour in. They are hungry for the knowledge and experience and ideas that this community has been incubating in our long, long, you know, time as a niche fringe group. And, you know, they need us to go out and teach them. And, you know, there's a lot of people who are very, very worried about AI job loss right now. And, you know, the good news for you all is if there's any set of people in the world who do not have to worry about this, which is unclear, but if there's any set of people, it is actually the people in this room.

Um, because the world for what we have to teach them. And I think we just, you know, this is our moment to shine. So, like I said, that's going to require us to change in a lot of different ways. That's also going to require Antithesis to change in some ways. We have historically been a tool for the few, for the elite. It's been hard to adopt. It's been expensive to adopt. It's required a lot of time to adopt. And so, we're going to be trying to change our approach as well in a pretty fundamental way. And to tell you a little bit more about that, I want to welcome my colleague Carl. > > Thank you, Will. Thank you, everyone. And let's keep it going for Will for a second here.

No, we have to do tech. Sorry, guys. What is going on? Play? Ah, there we go. All right. Thank you, everyone. I was here last year at Bug Bash, and I did not imagine I would be standing up here presenting Antithesis stuff to all of you this year. And I'm so excited to be here. Um, today I'm going to be talking about a lot of what Will talked about. Software engineering has changed. And I'm going to be trying to sort of motivate that change through a fun project called Aardvark Arena. Any of you wondering like, what the heck? Like, what is what is this like children sort of game looking thing on the screen that's obviously AI AI generated. No gravity. That's not how Battleship works.

Like there's a lot of things going on. But, this is the world we're living in, y'all. So, Aardvark Arena is a scrappy little team of game developers. Um on the on the left of your screen, Tavi is game developer. Tavi cares so much about making amazing games from her childhood. And Moss is a back - end engineer. He loves multiplayer game systems, scalability, stability, uh all the good sort of distributed systems things that we all many of us love. And they recently added another person to the team. This is Zippy. Zippy's their AI agent. Uh Zippy uses all the latest sort of greatest uh frontier models under the hood and basically helps them write code faster. Um and together they have built something really cool.

Uh let's see if this plays. Yeah, there we go. Um this is Aardvark Arena. Alt Aardvark Arena is a multiplayer game server. It has a matchmaker system that queues up players, matches them by Elo. It has a game server system, a cluster of game servers that allow players to play their favorite children's games. You'll see Battleship here, you'll see Connect Four, you'll see Tic - Tac - Toe. And all these are running in a distributed system. This is an application that maybe many of us have seen before. Many of our maybe our customers build this type of app. This is sort of a normal application with lots of interesting problems. But with Zippy, as much as Tavi has seen a huge pace uh in in in development, they've been able to write more games than ever or ship more features than ever.

They've also noticed that there's been a lot more gaps in the tests. They found a lot more sort of little issues here and there that they sort of wish that they didn't have to deal with. And like any software engineer, Tavi cares very deeply about this and started to do research, started to understand what it was that made their tests so easy to find gaps Like, or so bad at finding these gaps. Tavi learned about example - based testing. Tavi Example - based testing is something you may have all been aware of through the The most common name is unit tests. Unit tests or integration tests. Example - based tests are single - shot walks through your state space. Think of it as a set of steps that are taken and it either passes or fails.

The problem with example - based tests when you start scaling up the amount of software that's being produced by agents is that you only end up testing what you think about testing. And agents are really good at being very confident about what they think they're testing or what they're writing. And this leads to a lot of gaps. It's very hard to sort of verify software like this. So, Tavi went searching for maybe an alternative. And she learned about property - based testing. Property - based testing is a sort of different way of imagining a test. Rather than sort of a scenario like a specific set of steps, with property - based testing, we want to define a set of properties or invariants about the system.

And then we're going to throw thousands or millions of adversarial inputs at the system to try to find ways of disproving those properties. Find counterexamples in the system. If we think about it like state space exploration, property - based testing is this sort of like example - based testing in since you have these scenarios, but these scenarios explore a lot more of your state space. I like to think of property - based testing as like a warm hug for your code. And it sort of is like a seatbelt that protects you as you're driving really fast with agentic development. The thing is is that as much as property - based testing sounds amazing and all of these sort of formal verification and software verification methods sound really amazing, it's also something new to learn that not everyone has the time for or the interest in.

But, luckily, Tavi has Zippy. And Zippy's here to help. Zippy's recently installed a bunch of skills released by a company called Inthesis. And one of those skills is called the Inthesis research skill. Research is a skill that's very generic. It teaches agents how to think in terms of properties, write to like understand the topology of your system, and even understand really really high quality test design to be able to drive the system into interesting portions of the state space. Looks like this when you use a skill if you haven't used one before. You go into your favorite AI agent, maybe that's Claude. You say, " Hey, use the Inthesis research skill. " Uh it spends about 30 minutes churning through with many sub agents and so on trying to really really discern and distill the essence of your software into this sort of catalog of information.

Um you can think of it as this is almost like an ultra plan mode. The output of the research skill, as I said, is a set of documentation which can then be used to write property - based tests. Now, Tavi maybe ran into another problem at this moment. There isn't amazing property - based test libraries available in every language. We have a couple of amazing libraries in the industry, but if you just want to sort of adopt property - based testing, you have a very few set of choices. And for that, Inthesis has decided to sort of help solve that problem by releasing two open - source projects that we're excited to announce today. We have Hegel and we have Bombadil. And together these form what we're calling universal property - based testing.

Now, what do I mean by universal? Hegel is a client - server architecture. The server side is based on Hypothesis. Hypothesis is the best property - based testing library for Python. And about it sometime in the last year, we hired the two of the engineers who actually built Hypothesis to help us imagine what the how do how do we take all that is good in Hypothesis and make it available to everyone. And that was resulted in Hegel. Um Hegel is shipping front ends in all these different languages, and we're even seeing community front ends already pop up in PHP and Haskell. We hope that Hegel will form the foundation of property - based tests in all of the languages that you love to write. The second project is called Bombadil.

Bombadil had to reimagine property - based tests through the lens of how you would apply this this uh style of testing to a web interface, or maybe in the future any user interface. This requires teaching the system how to sort of autonomously explore a user experience and find interesting property violations. Something to think about with with user experiences is a lot of those properties are sort of very temporal in nature. You might say something like if this happens, I want this to eventually happen. And so we really had to think hard about the user experience around how we can discern and and and define those properties. Like all visual tools, Hegel or Bombadil is best seen rather than just explained. So in this demo, I'm taking Claude code and I'm asking it to run my Bom - Bombadil tests.

If it finds any bugs, be verbose about it, and then try to fix the fix the issue, and then show the result. Uh what we can see here is Bombadil's autonomously exploring the Aardvark Arena UI. Uh eventually after exploring and clicking through, it actually finds an interesting sort of state violation where our local client state has gone out of sync with the server. Because of Bombadil, Claude is able to look at the causality of what actually happened, the series of events that went into that point, quickly find the bug, fix the bug, and then rerun Bombadil tests. Now, it's important that we have Claude rerun the entire property - based test suite. And the reason for that is because it helps Claude make sure that it didn't introduce new bugs.

When you instead are asking Claude to sort of test using, for example, like an example - based test and fix an example - based test bug, when you rerun, you only verify that just that that one example's fixed. You haven't accidentally introduced some new problem in your code. These are some of the methods and techniques that we're finding working with agents that are really effective. So, you feel free to take a screenshot of this or a photo. These are QR codes that will take you to both of the projects. As I said, they're very They're open source. They're available right now, and you can start using them. And we're so excited to get feedback from how that experience is going. All right, but we're not done.

There comes a moment in every software engineer's life when they decide they need to write a test. And I don't know about you, but I've definitely thought about this as being it'd be really nice if tests could just write themselves. And Moss agrees. I think you know where we're going with this. Luckily, we have Zippy. And Zippy really is excited about writing tests and helping out. Zippy installed the Antithesis skills earlier to demonstrate the research skill to all of you. But now we're going to talk about the other skills. Antithesis skills is a suite of skills that collectively teach agents how to think in properties, how to write really effective tests, and how to use Antithesis really, really well. We take all of that knowledge we've learned over the years of using Antithesis with our customers, and we distill it into these high - quality skills that try to teach agents the best practices that we've learned.

Just to speak a little bit on Antithesis. If we think about property - based testing as being sort of a random walk through your state space, Antithesis is like a supercharged version of that. We take those property - based tests, those properties that you've you've added to your system, we take workloads that help drive the system through the interesting portions of the state space and find all those nooks and crannies, and then we supercharge it with fault injection, thread fuzzing or thread order fuzzing, time faulting, like all sorts of really interesting and fascinating behaviors. Um we layer on top of that determinism so you can go and replay exact exact uh states from this huge state space of potential failure. And so this is tests are really powerful.

And whenever I try to explain them, I find the best way to explain them is that they're basically just a box of pain for your code. You take your code, you put it into Antithe sis, and out comes bugs. With Antithe sis skills, we envision this loop that you can hook up to your agents. This loop is composed of sort of two sides. Something that is called pass, something that is called failure. The thing about passing in Antithe sis and any property - based test is that it only tells us that we haven't yet found a counterexample. We don't know for sure that this is perfect, but we know that with pretty high degree of confidence, we haven't found any counterexamples. And you can sort of measure that.

But the nice thing about this is that it serves as a good foundation to then increase the amount of state space that you walk. It add more workload, add more properties, define more invariants so that your system can get even farther in terms of coverage, or it can find issues faster. So on this side of the uh the pass side of the loop, we envision this we we have the skill called workload. Uh workload it teaches the agents how to do all those things. It teaches the agents how to add properties and make the workload more effective at finding bugs and exploring your state space. Now, Antithe sis is very good at finding bugs. So eventually, that will probably happen. When that happens, Antithe sis is going to output a report that says we found a failure.

We found a property that was violated, and here's all the information we have about that that violation. When you take that, there's quite a lot of sort of steps you have to take to understand what that property failure means to your code. You have to look at the logs, you have to understand what actually happened to go into that failure, you have to understand the properties, maybe someone else on your team wrote the property. You have to understand the code. There's quite a lot of steps. So, we decided to squeeze all of the best practices that we've learned about doing really efficient bug triage into a skill called triage. The Antithesis triage skill teaches agents how to do this really effectively. And like many things, it's let's it's most interesting if we watch it in action.

So, in this demo I'm about to show you, I have Claude running in the background loop. And it's basically just sitting there waiting for reports or something interesting to happen. When something interesting happens, Claude immediately jumps into action and and runs the triage skill. The triage skill teaches Claude how to grab the latest run, look at what's failed, understand exactly what's happened, read the logs, correlate it with the code, and ultimately fix the bug. All this happens very quickly and in this case it was just a few minutes. And finally, we have Claude loop back and send a run to Antithesis to verify that the fix looks good. This This loop collectively forms sort of an Antithesis testing loop with agents. And we're really excited to see people deploy this and see how it works in their systems.

But we're also thinking really hard about composability cuz it's not enough for us to just give you this loop that just works in this one way. It's actually much more powerful if we make these compose with your agentic software loop. However you're thinking about using agents in your environment or our customers' environments, we want to be able to work with them. And that's why we made the research skill generic. It can work with any property - based system. All this together leads us to sort of our goal. As Will was saying, we're having a lot of people come into our community and suddenly care about software verification. Property - based testing used to be for experts. Like we all know that. It's hard to think in these ways.

But with these things and with agents and with skills and with Bombadil and Hegel, we want to make it as easy as possible to adopt the best testing we can. So, thank you very much. Property - based testing used to be for research experts. Now, it's for everyone.