Agentic Development, From the Ground Up

Transcript

00:00We'll just kick the show off with introductions.

00:02All right.

00:02So let's get this started.

00:03I'm Jason Weimann.

00:05I've been working in game development and software for 20 something years, spent a

00:09lot of time on MMOs and done a lot of YouTube stuff, teaching people how to make

00:13games, have a lot of fun writing code, but AI has completely changed the way that I

00:18think about code and what I'm doing nowadays.

00:20So decided to start this show up with my buddy, Chris Lucian, who's an AI genius

00:26and has a whole ton of AI experience.

00:28You want to introduce yourself?

00:29Uh, yeah, I'm Chris Lucian.

00:30Uh, so I've done a lot of, I guess, corporate style software and, or like

00:36environmental software and, and military and irrigation and all kinds of things,

00:40IOT, uh, and, um, and so, uh, you know, I've been doing that, uh, for a long time.

00:46I also do, uh, a lot of just small indie games in my free time and stuff like that.

00:51So I do, I do have a passion for, uh, gaming and just like you, Jason, we use a

00:56lot of, uh, of AI, it's completely changing the way that we're working as well.

01:01And in that, um, you know, I also have a background in that.

01:06So, you know, I've, uh, my master's is in, uh, machine learning,

01:10focused on computer science.

01:12And so I have maybe some deeper knowledge on some of that stuff.

01:16Uh, and my, uh, my master's thesis was around, uh, you know, using evolutionary

01:21algorithms to improve computer vision models.

01:23So, um, you know, that was all kind of before the modern day transformer models

01:28and LLMs, uh, but, uh, it all kind of stems from the same, uh, neural networks

01:35and things along those lines.

01:36So, uh, I have quite a bit of background in that area and, uh, and I love, uh, I

01:42love the world of agentic AI.

01:44It's really, it's become a passion for me.

01:47Uh, and so yeah, here we are.

01:50Yeah, it has for me too.

01:51Like I said, it's completely changed just the way I think about software.

01:54And I think you're sounding way too humble on, um, your actual experience.

01:58Chris leads like in a crazy team.

01:59I actually went to go watch his team work multiple times just to learn from the way

02:04that they work, cause they were doing such amazing stuff.

02:06And I'm kind of curious to see how AI is integrated into that too.

02:10Um, it's before we dive into that stuff, real clear, it's like, you've

02:14been doing this stuff forever.

02:15I mean, we used to talk, you used to talk about image recognition and AI stuff.

02:19Long, long ago.

02:20Did you kind of see this coming or, you know, code was eventually just going to

02:25be mostly written by AI?

02:29Uh, well, uh, that's funny because there was a, uh, a podcast I did, um, maybe,

02:35uh, two or three years before, uh, 2023.

02:39And I had seen what was happening in, with the tool called Tabnine for

02:44kind of AI driven auto complete.

02:47And, uh, there's, you know, there's a recording of me on that podcast,

02:53basically saying like, we're going to be just chatting to our system

02:58to build what we want.

02:59And so, uh, it was just a really funny, uh, thing.

03:03Cause then somebody left a comment after 2023 and it was like, wow,

03:06Chris really called this one.

03:08Um, but yeah, I've, I've always been interested in, um, in AI.

03:13I kind of all started with, uh, with game development and games too, because,

03:18uh, when I was doing my comp sci degree, you know, way back, uh, the, you know,

03:24big piece of that was I was taking a lot of classes, every elective I got.

03:29I put it into artificial intelligence or machine learning.

03:32And so, um, you know, and there's the whole, uh, the AI, uh, scenario of the

03:39Wumpus World simulation, where there's like a cave and, uh, monsters that can

03:44chase you and move away and things like that.

03:46And so, um, you know, ever since then I've been super into it and all the way

03:51through my masters, I took every single elective possible that was either AI or

03:55machine learning and made my whole schedule around it.

03:58So, um, so yeah, I, I did see, uh, see quite a bit of this coming along.

04:05Um, you know, I didn't, I didn't predict, uh, LLMs or anything along those lines,

04:10but there was still a lot of, uh, of text prediction and things.

04:15Um, one of the professors in my master's program was doing a lot of prediction

04:20of Japanese language, uh, sentences and things along those lines.

04:24So I, I saw, uh, maybe where the technology was heading.

04:29Um, and, uh, a big, a big topic, I think, you know, another one of my professors

04:33was a really big on this idea of the AI winter, uh, where, um, the XOR problem

04:40was not solved solvable by neural networks.

04:42And so everybody reduced funding and now we're in an age of unprecedented

04:46funding for AI initiatives, uh, and everything's accelerated tremendously.

04:52So, yeah, just, just a tiny bit.

04:55Just a tiny bit.

04:56Yeah.

04:57Yeah.

04:57It went from basically no funding to, um, you know, I really think that, uh, when

05:02Google released TensorFlow as open source, um, and they basically called out

05:08all of the papers out there, the academic papers saying like, you guys haven't

05:12been trying hard enough sort of thing.

05:14Um, that's, you know, and I think that was like right around 2014, 2015.

05:18Um, I think that's when things really kicked up because the hardware was

05:22there, uh, the motivation was there.

05:24Um, you know, I think it was stuck when the single layer perceptron was popular.

05:31Uh, and then when the multi-layer perceptron released, then I think

05:35companies like Google started like doing more, um, but I don't think it became

05:39really a public focus until after, um, after TensorFlow, uh, came out.

05:46Um, because you could do a lot of stuff with like scikit-learn and things like

05:49that, and then, uh, and then it like really exploded, um, in 2023, uh, obviously

05:55because of ChatGPT, but, um, if you go read, uh, there's a great book called

06:00the alignment problem.

06:01It talks about saliency.

06:02And so it talks about early days at Google where they were dealing with

06:05these trans transformer models.

06:07And, um, you know, they would say you could, you could basically add words as

06:12vectors, and so they would say things like, um, you know, Apple plus yellow

06:17equals banana and things like that.

06:20And then, and so that was all fine.

06:21But then, uh, the alignment problem was that you would say like doctor plus

06:25woman equals nurse instead of doctor.

06:28And so that was like a huge issue.

06:30Right.

06:31And so Google didn't, um, didn't want to like promote that and things like that.

06:35And even though ChatGPT had the same problems, they basically said, you know,

06:39uh, um, because they were small and it had like wide scale adoption and things,

06:44uh, it, it opened up everything.

06:47And I think that was really good for Google because they didn't have to take

06:50the flack for that sort of stuff.

06:52And then now they release Gemini where they can, uh, leverage the technologies

06:57that are inherently bias.

06:58Uh, but they can say, well, ChatGPT did it so we can do it too.

07:03Um, so in a way, um, Google needed to take a step back anyway.

07:10And, uh, and so, um, you know, I don't, I don't know if there was any strategy

07:15behind how it played out, but it certainly did play out that way.

07:18Um, so yeah, I would totally read the, uh, The Alignment Problem or Weapons

07:22of math destruction, if you're like interested in AI bias.

07:25Um, I'm going to start with the alignment problem and just read it.

07:28And I think anybody watching, just go ahead and read it and maybe we'll talk

07:32about it in the next one too.

07:33Yeah.

07:34Reading exercise for me.

07:36I need to do a lot more AI reading.

07:38I've been doing a lot of AI using and hitting limits constantly, but I probably

07:44could go for a lot more reading and a little bit more learning.

07:48Yeah.

07:49Well, there's, I mean, it's, it's really interesting because we're all using these

07:52large language models and they're neural networks with billions of parameters.

07:55But it's like, what does it look like to use a neural network with five parameters?

07:59And, you know, and so, uh, a lot of the time people don't understand, like, you

08:03know, there's all these deep learning models that we're using and they're all

08:06leveraging all these things that we learned from shallow learning, but like

08:09people talk about deep learning all the time, but deep learning implies that

08:13there's inherently this thing called shallow learning that, you know, that

08:16people don't really understand or know about, um, and, you know, supervised

08:21learning versus unsupervised learning and stuff.

08:23And so, uh, learning that stuff helps with interacting with these models.

08:28Um, because you, you understand where and why things are happening,

08:34you know, reasons behind it.

08:35So, um, you can certainly feel your way through experimentation

08:41in adopting these technologies, but there's kind of this next level, I think of like

08:45the, uh, the academic background of these systems and where they're going.

08:50And Chad GBT can, or, or GPT and LLMs in general can be, I think, easily replaced

08:56by some other big finding in the learning space.

09:00There's all this focus on LLMs and transformer models specifically.

09:05It doesn't mean that the next breakthrough isn't going to come from a

09:09completely different type of model.

09:11Um, and that's, that's a big thing about machine learning is that, you know, you

09:15want the machine to learn and so, uh, so the model is just our best approximation

09:21of how to handle that problem, but, um, it's all ends up being statistics in the end anyway.

09:27Okay.

09:28I'm lost.

09:30So how much do you think people really need to understand, um, of the, the low

09:37level, like, like how everything works and what do you, I guess, what do you think?

09:40Kind of like the, if there are things that are really, really important to understand,

09:44like what are those things?

09:46I mean, and how much do you think they need to know to kind of be productive news?

09:51Well, um, so part of it is, uh, like if you look at WordPress or, or Drupal or

09:56something along those lines, like a CMS, right.

09:58There are the people that built WordPress and then there are the people that use

10:02WordPress and the people that use WordPress don't necessarily know how to build WordPress.

10:06But then they get like super interested and they're like, I want to build a WordPress

10:11plugin.

10:11Now they have to learn a whole lot more than a typical WordPress user.

10:16Um, I think it's very much the same thing here.

10:18Uh, you know, I think a lot of people, um, they're very surprised.

10:23Like if somebody is, is, uh, doing agentic coding with me and they, and they see me

10:30type slash debug into the agent and they're like, I had no idea that was even there.

10:35And then it's like, oh yeah.

10:36And there's like a whole tree view.

10:37And I want to see like why it made this choice here after it ran this command.

10:42And it was like, oh, the output had all these other stipulations under the covers.

10:45And so, um, that's why I bring up the alignment problem even is that there's a

10:50lot of this question about AI saliency where you like know why it made a decision

10:55and chose to do the thing that it meant to do.

10:57And if you didn't, uh, you know, like right now, if you run into a problem with

11:03an AI agent, you're like, must be the model.

11:05I guess I'll go try a different one.

11:07Um, and, uh, and you know, you just give up, right.

11:12Um, and so, uh, the, um, yeah, the side effect of, of that is, uh, well, you

11:21know, should we have, uh, have tried harder, right.

11:25Or, and do we even know what to try or why?

11:28And so, um, I've never seen an AI agent suggest that I debug its reasoning.

11:35Um, and so when, when I'm using, you know, I, I'm a fan of like all of the UI that,

11:42uh, that GitHub Copilot has, um, and, uh, for like, you know, the, the day to day

11:48work and, uh, I, I also have been using Roo Code with local models.

11:53Um, and, uh, and when I'm using GitHub Copilot, turning on the debugger and

11:59actually looking at why the model is making those decisions.

12:02So you could be using, you know, uh, Claude Fable or, you know, kind of like

12:07at the time of this recording, uh, probably the best publicly available model.

12:12Um, and you could be using it and it could make decisions and then you, you

12:16would be like, oh, uh, that answer sucked.

12:19So I'm out of here.

12:20Um, and, uh, and so, you know, but was it really the model or was it the harness?

12:27And so when you, um, when you go into, like, you dive into things like harness

12:32debugging, then it's more and more valuable to have a depth of knowledge in these areas.

12:36Um, and it's, I think a very equivalent to using WordPress versus building WordPress.

12:41It's it's understanding, like, how did it get built so that you can make a

12:45WordPress plugin because really what we're talking about here is, you know,

12:48I'm here, I'm prompting agents and I'm doing some really cool stuff with agents.

12:52Uh, but then, um, you know, but then when you're stuck or when it's not doing the

12:57thing you want it to do, uh, then are you, are you just stuck and you're going to

13:02wait for an update so that you can do it?

13:06Or are you, are you capable of diving in deeper and understanding

13:11how you should change things?

13:13Um, and so I think that that ends up being like a big question and then, and

13:18that's where depth of knowledge comes from.

13:19Right.

13:19So, uh, if you, if you have a very shallow knowledge, then, then you

13:26will use the tool in a shallow way.

13:28And that might get you really, really far.

13:31Right.

13:31Um, uh, Chad, GBT, uh, in the web UI with no agentic tools, uh, still makes it so

13:39that you can, you know, update a resume.

13:43Right.

13:44Um, and, uh, but if you're going to do more and more advanced things, then you

13:48need deeper and deeper knowledge.

13:50Um, and so, uh, so I think the answer to that question is it varies depending

13:55on what you're trying to do.

13:57It does seem like it's pretty useful to kind of dig in and grab new tips and stuff.

14:02And knowing to debug things, kind of like you said, knowing that that's even an

14:07option or knowing that the harnesses matter, which I think is some people don't

14:13realize, I had a lot of misconceptions about AI stuff from people who are just

14:16kind of starting to use it.

14:17So, so a lot of, um, a lot of things people aren't realizing.

14:22Um, I think that I'd kind of like to just move on to what we're actually doing in AI

14:27stuff, like those talk about a couple of the interesting things and I think it may

14:30be in some future episodes, kind of go deep into how to use some of these debugging

14:36tools, how to, um, some of the differences between the agents and harnesses and

14:40obviously updates on like what's new and what's coming out in Fable 5, obviously

14:44the latest for this week and then who knows what it'll be next week, right?

14:49Yeah, for sure.

14:50Constantly changing.

14:51I did try, um, what was the open source one?

14:54G I've lost the name of it.

14:56Oh, GLM or Qwen.

14:59The GLM one.

14:59Yeah.

15:00It was okay, but it's still just not close.

15:04Um, my son was pushing it heavily on me and I've just got to try it.

15:10And that's kind of right.

15:11Was it a full GLM instance or was it quantized and like, do you know any of the

15:15specifics of what was going on in the background?

15:17I tried two different hosted ones and I couldn't get details on either of them.

15:21Yeah.

15:21What they were doing.

15:23But, and so, and the harness can matter as well there too.

15:27So if you used a different harness, then it also can impact.

15:31Um, but yeah, I think, uh, when I look online, I'm like, you know, and I search

15:37for, uh, alternatives to Fable or Opus then GLM 5.2 is what comes up and, and,

15:46uh, yeah, but it can be challenging to know how good it is because the harness

15:50could be bad, um, and, uh, so, so that's also a part, a piece of it, but, um, but

15:58yeah, I'm, I'm interested, like, I haven't used GLM, but I've been doing a lot of

16:01Qwen and, and Gemma even, uh, and, um, and doing a lot of these, I do exercises

16:09where I massively constrain myself so that I can, and that pushed me to learn

16:14the debugger or creating custom system prompts for the models and things like

16:17that.

16:18Um, so like in the, in the idea of doing a Cotta, I've been doing that, but like,

16:22what else have you been doing?

16:23So for me, it's been a lot of game development and a lot of experimentation

16:29so far, and it's maybe been, uh, four weeks or so they've been kind of going full

16:35on at it, just, um, prototyping out seven different games so far.

16:41I was doing one a day for a while and then dug in a little bit more on a couple of them,

16:46trying to figure out what I want to build.

16:47That's fun.

16:48It's a lot of the ideas of things that I wanted to build, but I was like, I'll

16:52never have the time I need to win the lottery and hire all my friends to be

16:56able to go make this now I'm able to get a working playable prototype, you know,

17:01in three, four hours and then realize that my idea sucked and I want to try

17:06a different one.

17:07So right now I'm going through a lot of different game ideas, trying to figure

17:12out which one I want to build once I've locked one down, I've got one that

17:16pretty close, a VR one and a non VR one that I want to try to do.

17:20Um, as long as they stay being fun for the next week, I want to finish them in

17:25like two weeks and get them done and playable and get the kind of, at least

17:29the core loop of the game, all playable to a point where I decide if I want to

17:34go no go on doing any more development on, but I've also been using it for a

17:38lot of app development.

17:39So built a couple, um, quality of life apps for just my house, one to track my

17:47kids chores.

17:48I called it Chore Quest super simple to use runs on the phone.

17:52I can manage what they're doing.

17:54They can go clean their room, take a picture with their iPad, submit it, get

17:59points, request prizes, all the, all the stuff that I wanted to do multiple times

18:04and make a fun interface for, but also never finished.

18:07I also built a life counter for playing magic, play a magic lot of huge tables.

18:12And I wanted to play one where I could do a distributed life counter where you

18:15could have one in the middle and then control from your phone.

18:18So did that.

18:19And then just told it to add all of the options from all of the other life

18:23counters and just make it do everything that everything else does and the stuff

18:28that I wanted just did it.

18:30Um, I really kind of worry about the value of apps longterm.

18:35I think that like, there's not going to be a whole lot.

18:38I mean, and speaking of that, I don't know if you saw a Roblox announcement

18:41about, uh, adding AI app generation in game on mobile.

18:46So kids can, or, I don't know if it's out.

18:50Yeah.

18:50You're going to look, I just saw the announcement last night.

18:52Um, but yeah, kids can go in and create games with AI while they're playing

18:57robots, no longer even need to be a developer using Roblox studio.

19:02You just talk to your iPad and tell it to make the game.

19:05And I feel like that's where things are going.

19:08Things are getting to hollow deck level just without the visualization.

19:13You still got to strap a VR headset on, but, um, at least it's kind of getting

19:18to where I always kind of dreamed things would be, hopefully we don't get like

19:23an alien invasion soon and yeah, whatever else comes along with all the

19:27Star Trek stuff, but

19:29let's help Claude to solve the alien invasion.

19:32Well, I think the next thing is the AI wars, right?

19:35I've seen Star Trek and we build data or the, the AIs and then we'll get some

19:42mutants and then there's a big AI mutant war.

19:44And I, uh, I just read a book called, uh, If Anyone Builds It, Everyone Dies.

19:49And it talks about the, uh, the inevitability of someone building

19:55an LLM that tries to escape.

19:57Um, and, uh, it's a, it's a pretty good book and it talks about the realistic

20:02reasons why AI might, might actually cause the extinction of humanity.

20:06Um, which is also, I think an important topic, but you know, obviously

20:09I read a lot of this random stuff.

20:12The, so on the game idea I was working on last night, the AI recommended

20:18integrating an LLM into the game to run the AI for the game too.

20:23I was like, ah, it's a good idea actually.

20:27But it still freaked me out.

20:30Yeah.

20:31I mean, there's, uh, you know, I was looking a little bit at that too.

20:35And so some of the open source LLMs are, um, you know, have like things where

20:41they're like, oh, you can't repackage it or whatever in your game in some cases.

20:44But, but for the truly open ones, um, you know, it's like, if you have eight

20:48gigs of VRAM on the machine and then, and you have something over that, so it's

20:53still, it's not cheap to run it locally.

20:56But I mean, you can get, uh, interactive dialogue with, you know, fully

21:03agentic NPCs, um, on a local harness.

21:07And, uh, they'll still take a little while to respond, but like, you know, the,

21:11it's pretty, it's still a lot more snappy than it used to be.

21:15I want to dig into the, um, Unity ones.

21:18They, cause they would, they've been doing that for like five, six years now

21:20with the Sentis stuff and putting the, the ML stuff right in there.

21:25I would bet, uh, I haven't really checked on it, but I would bet you could

21:29probably do something in there.

21:30Oh, I bet Claude could do something cool with that and make an interesting AI in there.

21:37I'm going to play with that.

21:38Yeah.

21:39What Claude can do.

21:42Yeah, it's pretty, I mean, uh, it's interesting, but I mean, there's also,

21:47um, you know, a lot of laptops and desktops come with NPUs now, the neural

21:52processing units and as long as they run on that hardware.

21:55So, um, I think, you know, I kind of think to the days of the PC where they have

22:00like the dedicated sound cards on, on them, um, I think, uh, or the dedicated

22:06network card or whatever, uh, PCI.

22:08You don't have those anymore.

22:10You just have a regular old onboard card.

22:12Yeah.

22:13Yeah, exactly.

22:15Um, but I mean, I think that, you know, the whole like splitting off the, um, the AI

22:21piece, uh, and possibly even running it separately from the video piece can, can

22:27make, give more consistency in that area, but there's, there's certainly like a

22:30hardware evolution that is happening slash needs to happen.

22:34Um, and, and I think we'll, we'll definitely see more games with those

22:38features come out as well.

22:41Um, and there's, I think there's quite a few of them out there now, uh, but they're,

22:46you know, they take up all of your video, your VRAM with the model and then you,

22:51and then you can do stuff after.

22:53Uh, but yeah, I'm interested to see, uh, what's out there and like, you know, with

22:58my limited time to experiment, I'm like going through model after model harness

23:02after harness, just trying things out.

23:04Um, yesterday I was playing with Kilo, uh, using Gemma.

23:08Um, yeah.

23:10Well, I'll let you know how the Unity one works in the, uh, in the next episode.

23:14I'll be doing that this week.

23:15Yeah.

23:16Go ahead.

23:16It'll be, it'll be good for us to try different stuff and, uh, we can end up

23:20talking about it and comparing notes later.

23:23Totally agree.

23:24Speaking of what, uh, stuff have you been doing?

23:27Yeah.

23:27I mean, so, uh, I've got, I've got lots going on, um, in this space, uh, you

23:34know, both like professionally and like in my free time sort of stuff.

23:37Um, and so, uh, I also have been, you know, building out game ideas.

23:43I've done, I've done a couple of things with like, uh, um, you know, voxel based

23:47freelance style shooter, like totally vibe coded, but with all the guardrails

23:51that I would put into any other code.

23:54And then, um, kind of at like the, uh, professional level, um, you know, quite

24:00a lot of, uh, agentic workflows.

24:04And I've been talking a lot about this idea of kite flying and plate spinning.

24:08And so kite flying being getting the agent to operate on its own independently

24:13for longer and longer periods of time.

24:16And plate spinning is getting more agents to, uh, to work on different things

24:21simultaneously in different directions.

24:24Um, you got to dive into that in a minute.

24:28But, uh, yeah, so, um, I, uh, I'm, uh, I'm actually putting those together as

24:33conference talks and I want to go speak about that, these concepts.

24:36Um, and, and then that kind of all boils into this, uh, idea of, um, using

24:42domain languages to give the agentic AI context on how to build things quickly.

24:49Um, and having AI build the, the app generators, not build the app.

24:55And then that way, when the, you can have deterministic generation of code

25:01using, uh, a domain language.

25:04And then that way, um, your LLM ends up building the generator one time.

25:10And then it reproduces.

25:11So if you, if you make like an input box on a website right now, if you ask

25:16Claude to do it or any LLM to do it, uh, it will write it differently each time.

25:21It's not going to write the same code everywhere that you have it.

25:25Um, and so all the boilerplate might be slightly different in each space.

25:29Uh, so what I've been doing instead is doing a domain language for the business

25:35that then results in, uh, an indication that a input box should be built.

25:41And then now it's all deterministic.

25:42So every time you say I need an input box, it's always written the same

25:45way with the same boilerplate.

25:47Um, and so the agent then builds the thing that then writes the code.

25:53Isn't it kind of like just using components though, somewhat similar?

25:56Just build this component and then use this thing.

26:02Um, how do, how are you doing that?

26:04By the way, I, I do something similar with like Unity stuff and prefabs.

26:08I'll tell it to go build these UI prefabs and just use the same stuff.

26:12Cause otherwise it will try to just do everything and code dynamically at

26:15runtime, right?

26:16Right.

26:16Yeah.

26:17So what's your process for that?

26:19I assume you have a better one than just.

26:21Well, no.

26:22Yeah.

26:22So in, in Unity, like prefabs are sort of a domain language.

26:27So it's, it's very similar in a sense.

26:29Um, so, so in the, in the environment I'm speaking about, um, we have, uh,

26:34basically Python objects that get put together.

26:37Um, and, uh, and so we also have other, um, departments or groups that we depend

26:43on, and so we can actually take their documentation, read it into that, then

26:47generate a website off of it.

26:49Um, and, uh, a lot of that is saying, you know, uh, having a system for

26:54importing or reading documents.

26:56And then that builds out, uh, it used to be JSON and now they're like, uh, Python

27:00objects, but it builds out a bunch of texts that then represents all the things

27:05that should be built later, and then there are, um, generators.

27:09And so in Python, there's a library called Mako.

27:13Um, and, uh, and it basically allows you to generate a, uh, a lot of code based

27:19on different flags and things like that.

27:21So, um, yeah, so, so in that space, it's like, if it should be an abstraction,

27:28then yeah, make a component out of it.

27:31Uh, if it's something that's unique to the business, then use code generation

27:36where before I think it was something you, uh, unique to the business, you just

27:40write the code that did the thing.

27:42Um, but, but now what you can do is, uh, you know, so it uses the component that

27:48you built, um, and so now you can tell an LLM, like, here's the context of my

27:53business, here's a language in how you might build a site to do this thing.

27:58Um, now write the code in that language to then build the site to do that thing.

28:03And then that way it removes the variance that comes from the

28:08random randomness in the model.

28:09Um, so, uh, um, so yeah, like code generation historically hasn't been, uh,

28:17an option that you wanted to reach for, uh, unless absolutely necessary.

28:21But I think in this case, uh, having code generators that build off of a domain

28:26language will then, um, you know, it reads the domain language and then

28:30produces the code afterwards.

28:32And that thing that produces the code is no longer a human.

28:35It is a highly variable LLM.

28:38And so you can remove the variability of that LLM by having the highly variable

28:43LLM make a deterministic code generator.

28:47Yeah.

28:48No, that makes sense.

28:48It sounds, um, somewhat similar to what my buddy Steve did.

28:53I'll have to have him on some time and talk about, I think I sent you a link to his thing.

28:57Super cool.

28:58Yeah.

28:58I would check that, but it is something similar.

29:00You give it the app idea and it hosts everything in AWS.

29:05That's it all up and you just get a link and it just works.

29:07So it's, um, it's gotta be, I assume he's doing something somewhat similar, just,

29:12uh, a little bit more in there to make it public.

29:17Interesting.

29:17Um, yeah, I want to talk though about this, this whole kite thing for a minute.

29:20So before Fable 5, one of my favorite things to do was just create

29:28tickets and set a goal for my agent to just go complete those tickets.

29:34And it would just work through them 12 hours nonstop.

29:38Just keep going.

29:39I wake up or whatever, or get done working and go look at it and see that,

29:44Hey, it had finished most of what I wanted and it kept going and iterating

29:48until I told it to stop or I thought to keep going for eight hours or whatever.

29:53At the time, especially when it's about to hit a reset limit, right?

29:56I'm not going to keep going until the reset limit or you run out of time.

29:59You keep going.

30:00Um, I found with Fable 5 that to be a little bit less, um, of my go-to

30:06cause I burned through the credits really fast.

30:10Yeah.

30:11But so what do you, like, what are your general thoughts?

30:15So is it something that you think like, should I just be using a lower model

30:19and setting goals like that and letting it run constantly, or should I

30:23be doing something different?

30:25Right.

30:26Uh, well, um, so, so Fable, uh, five is very good at diffuse reasoning.

30:34Um, and Sonnet 5 is very good at a very focused reasoning.

30:40Um, and so, uh, I've actually started incorporating a mix and I, like, I

30:44barely use Opus anymore, which is kind of weird cause Opus was like the thing

30:47that I went to all the time before that.

30:50Um, and, uh, and so, um, with Fable, yeah, it's very expensive to just

30:58have it goal seek like that.

31:00Um, but it also depends on how many guard rails you have in place.

31:04Um, because, uh, you know, if the, if the only way for the agent to see that it

31:10did something wrong is to go and like run the app and test it, then it often.

31:15Could introduce, uh, you know, again, this is all about introducing variance.

31:19Right.

31:20So it might choose to test something different.

31:22You might get a regression somewhere else, and then it might not find the

31:24regression until later because you have a test at some, some other time.

31:27Um, so I really like unit tests for making this, uh, more consistent.

31:32And so as long as you have like a really good unit tests, we, on the project, then

31:36all of a sudden the variance goes away and you it's cheaper to use Fable because

31:42then it's going to find where it broke something without like a burning credits

31:46on exploratory testing, for example.

31:49Um, and, uh, you know, where w with, with Sonnet, you can do like lots of rapid hits.

31:55And so, uh, if I, if I am ever telling an agent to do any exploratory testing, I

32:00just say, um, here's access to the website.

32:02It's locally hosted, has the new code on it.

32:05Go ahead and try the feature that you just built.

32:07Um, it then goes and uses a Playwright MCP server to then like start

32:11interacting with the website and, uh, you don't want Fable doing that.

32:15It's like super expensive.

32:16There's a lot of waiting, a lot of tokens going back and forth.

32:18So Sonnet is super cheap to do this.

32:20Um, and so, uh, and it's very, very fast.

32:24And so, um, I guess, yeah, the idea of kite flying, um, is very much like in

32:31the, in, in the area of where you'd set off Sonnet to go do that.

32:36Um, but, uh, but usually when you're using Fable, there's a lot of talk about Fable

32:43and security and Mythos and like limiting it access and everything.

32:47And the point to that is that, um, you know, you give it a million tokens of stuff

32:52and it's very good at, at finding relationships between very disparate

32:58sources of information and that's why it's such a good hacker.

33:01Um, and so, uh, so, so when you're doing something, I basically put my tasks

33:08into two categories right now, and this is like, you know, a couple of

33:11weeks of experimentation, right?

33:13So, you know, since they've like made it publicly available to me.

33:17Um, and, uh, and so if it's, if it's security related, or if it's like something

33:23where it's going to be far reaching, like a major conversion from an old

33:27technology to a new one, I'll put Fable on it.

33:30If it's something where, uh, it's just like building something or, or, uh, or

33:39maybe debugging, you know, like a production issue or something, then

33:42I'll put Sonnet on it.

33:43Um, and so, uh, you know, like right now in an AWS environment, you can have

33:48read only production CloudWatch MCP servers, um, and you can just tell a

33:54Sonnet agent to go use those MCP servers and it will go and grab the CloudWatch

33:58logs, evaluate for the problem, find it in the source code, go update the source

34:02code and say, push this, you know, uh, to fix the problem.

34:06Um, and, uh, and that process is very fast using Sonnet, but it's

34:10really slow using Fable.

34:12So, so for me, it's like, is it security?

34:14Is it a diffuse reasoning task?

34:16Give it to Fable.

34:18If it's any sort of execution task or a very like goal oriented

34:22directive, then I give it to Sonnet.

34:24Um, okay.

34:26Yeah.

34:27Makes sense.

34:27And so I assume you're using a lot more Sonnet than most of the time.

34:32Yeah.

34:33I mean, well, so, so I tend to be like, I tend to have these like really big

34:38ideas around changing something.

34:40And so I'll be like, I want to experiment with changing how authentication happens

34:44through the entire application.

34:46So like, that's something that I would give the Fable.

34:47So, so, uh, it's kind of like 50, 50, like I'll have, and so this is the plate

34:52spinning piece of the kite flying and plate spinning.

34:54Um, I'll have Fable going on something big and exploratory and like game changing.

35:02And then I'll have like three or four instances of Sonnet, all building out

35:06different features.

35:08Um, and then, you know, what I tend to do is then I, uh, um, I title each of those

35:14chats with a leading emoji.

35:16So I remember what I was doing.

35:18And then, um, and so I'm like, oh, uh, you know, the, the envelope is for text

35:23messaging and, um, the, like hammer is for DevOps and the whatever.

35:29And so, so then I can just be like, oh, here, have, take a task from this, take a

35:33task from that, and then they all kind of spin off and then I review the code and

35:36merge it back in.

35:37Um, and so, uh, you know, as we're, as we're doing development, um, and this is

35:44often with like, uh, with, uh, mob programming.

35:47So we have, you know, four people there all building the prompts together.

35:52Um, and so what we'll do is we'll like make a, um, I like this like wisdom of the

35:58crowd idea where everybody writes a prompt independently about everything and tries to

36:04include everything they're worried about.

36:05Um, and then we all enter them into the chat and then send them all at the same

36:10time, then we take all of those prompts and then merge them together into one big

36:14prompt, um, and then we run that prompt because now has everybody's concerns in

36:19there.

36:20And so some, some people are more focused on like database performance and other

36:23people are more focused on quality control and things like that.

36:26Um, and then that gets into the process as well.

36:29So that's kind of like the, um, in my mind, the evolution of like the mob

36:32programming into this age.

36:34I was just wondering about that.

36:35Like, how does that work with multiple people prompt bag?

36:38It makes sense.

36:38You end up with a lot more detail on that.

36:42I'm curious.

36:42Do you prompt like only typing or do you ever talk to your phone like a weirdo

36:49like I do, which is like probably 50% of my prompts.

36:52I'm just sitting there for like two minutes, like hoping that nobody's looking

36:55at me while I'm sitting there talking to my phone.

36:57I'm out somewhere.

36:58Sometimes I'm just at the house and the kids I'm like, I'm talking to Claude.

37:03Yeah.

37:04Well, it's kind of funny because like we have, we have people at the keyboard and

37:08as we're rotating at the keyboard, a lot of the time, the person at the keyboard

37:11is writing the prompts, so you're verbalizing your prompt often.

37:14Um, yeah.

37:15So I, um, you know, I tend to, uh, I don't know very early on.

37:23I think, I think I think a lot through typing.

37:27So, uh, so for me, uh, I do tend to, um, to, to do that more, but I,

37:32I also absorb information better if I listen to it.

37:35So I actually, I go the opposite direction.

37:38Um, I'm not reading what my agent, uh, puts on the screen.

37:42I'm listening to it because I have, uh, I, you know, I've been

37:45maintaining this open source project for over a decade now.

37:48It's a speed reader.

37:50And, um, and so I've added an MCP server to that speed reader.

37:54Uh, and so, yeah.

37:56And so now, um, uh, the agent can reserve a voice for itself in,

38:02uh, from the available voices.

38:04And so now each of the agents speak with different voices so that I can hear

38:07like what it was recent that has recently done, or if it has a question or something.

38:11Oh, nice.

38:12So when you've got multiple agents running, you just hear them chime in

38:16and talk instead of a little beat.

38:18Yeah.

38:18So I'll be like doing something.

38:19I'll be going through my email or something and be like, oh, I've done this.

38:22Like, you know, and I need your input on something else.

38:24I'll go like go into all time to it and start typing back.

38:27I should talk back to it probably, but, uh, but yeah.

38:31Yeah.

38:31I wonder what your results would be like with that.

38:33I noticed for myself that when I started, I saw it on some video,

38:38I forget which one it was.

38:39Uh, when I started talking instead of typing, um, I gave way more info and context.

38:45I did find that I like the first couple of times I would say something I'd be wrong.

38:49And I'd stop and go restart it.

38:51And then I was like, am I doing that?

38:52I'll just tell it.

38:53Nevermind.

38:54I changed my mind about that thing and it just, it was like, it just works.

38:59Um, and I'm giving it so much more context.

39:02Like I'm talking to it for like a minute or two.

39:04It's way more than I'm going to probably type.

39:06Cause when I type, I try to think like a programmer and be very specific on

39:10exactly what I mean to say, but leave out a lot of details.

39:13So, um, I thought it'd be really useful.

39:15I'm curious.

39:16Anybody watching to like comment with a do to talk or type or do you

39:20listen to it like Chris, I'm going to try listening though.

39:23Yeah.

39:24Oh yeah.

39:25You can check out the, the speed reader and, uh, yeah, you can find it.

39:29Um, just like search for my name and GitHub.

39:33You'll find it.

39:33I still find it to talk twice as fast.

39:36Yeah.

39:36Yeah, exactly.

39:38Yeah.

39:38The default does read it at 500 words a minute.

39:41So I do listen to it very, very fast.

39:43Uh, but you know, there's, there's that aspect of it as well.

39:48Yeah.

39:48You speak.

39:49Oh my God.

39:50I don't know.

39:52I kind of wanted to make it just talk like Tupac too though.

39:54So, yeah, just send it over to ElevenLabs or something and get it

39:59like all voice changed.

40:00Get all done.

40:01Actually.

40:02Um, speaking of that, one of the things I'm doing this weekend is taking one of

40:07my buddies who died a couple of years ago, um, putting his voice into ElevenLabs.

40:12So I can put his, him in the game, um, as one of the characters.

40:18Hmm.

40:18Yes.

40:19And he's going to have lots of Tupac quotes cause I wanted to be realistic.

40:23Yeah.

40:24There you go.

40:24Literally the last video that I have is of him like that.

40:29Just written about West coast rappers.

40:32So I got to come up with, let AI make a game about a West coast and East coast

40:36rappers, I'm not sure what the game is or how it's fun, but, um, yeah, he'd like it.

40:43Exactly.

40:44The homage.

40:45Uh, I mean, we, we, we really are in the age of bespoke software, bespoke

40:51ads, like making your girlfriend a game for her birthday is no longer like, Hey,

40:56but that was a really cool thing I saw on Reddit.

40:58And I was like, why the fuck didn't you do it?

40:59Like you just make her a little, make her a romantic little game

41:04of the thing that she likes.

41:05And so I've got to remember that for next year.

41:09Just going to get my face in going forward.

41:12And my friend's 12 year old made a Roblox game and my kids played it.

41:17Like it's, uh, it's getting to the point where it's just, there's, there's really,

41:22really no excuse other than your ideas suck, which is the one that I'm running

41:26into right now, so I'm trying to come up with an idea that doesn't suck anybody

41:29that's got one and doesn't mind me stealing it also feel free to share that.

41:34Happy to steal a good idea.

41:36If it's fun to play.

41:37Nice.

41:38A good idea that somebody wants stolen.

41:40I should say.

41:42Right.

41:43Just to be clear.

41:45Oh yeah.

41:45It used to be that the, you know, the idea was the cheap part and then, you

41:49know, building it was the thing that made it so that stealing ideas was okay.

41:54Because if you stole an idea and the other person didn't build it, then like,

41:59really stealing the idea then.

42:01But, uh, but yeah,

42:02yeah, the idea and the design is it.

42:05Yeah.

42:06Yeah.

42:07That's it.

42:07It's interesting.

42:08Um, it still takes, you know, a considerable amount of money to build all

42:12this, all this stuff, um, because of the number of iterations that the LLM has

42:16to go through in the first place.

42:18Um, but it's just drastically cheaper.

42:21It's just like gone from many, many hundreds of millions to tens of millions

42:27to single digit millions, hundreds of thousands now, like, if you're dedicated

42:34and a decent designer with some time, you could do it relatively cheap.

42:38And I think the, the last thing that I wanted to kind of make sure to mention

42:42was the thing that really kicked me over the edge of this, cause I didn't

42:45think about AI like replacing programming as kind of the core, not like replacing

42:52I mean, like changing the way that we do programming so that it's not mostly just

42:56us writing code, just like, you know, engines and, you know, IDEs change the way

43:02you do it, but I've been thinking about that since I guess slightly before chat

43:06GPT, but what pushed it over the edge was going to GDC and seeing, um, basically

43:13a, a Castle Crashers clone that visually looked better.

43:18It was built by an artist and a sound guy with no programmer and they had a

43:25working game that they had built using Bezi, which is another, um, it's an AI

43:30agent harness that works pretty well.

43:32Um, I think that's the one that I used.

43:33Uh, one of the ones that I used the GLM on, I tried, I tried it on there.

43:39They have a variety of them, but when I saw that they had built a game and I

43:43talked to them and like, yeah, not programmers, I can kind of read a little

43:46bit of code, but couldn't like write, you know, a game or an application, like

43:50kind of like know what an if statement means, but, um, but they're like, yeah,

43:54we just, you know, got to focus on the cool parts, like making the visuals the

43:59way we wanted to get to make the art, the, the, the 2d art and make the sound

44:03and make the boss fights, um, but not have to worry about the programming.

44:06And they did it in a really fast timeline too, but that was, I was like,

44:10okay, there's no more excuses, right?

44:14There's no, there's no, like, I can't build this game because I don't

44:17have enough money or team.

44:19It's really just like, whether or not you have the spare time and the idea.

44:24You just need time and idea.

44:26And that's it.

44:26I really feel bad for my buddy, Brad McQuaid, like died a couple of years ago.

44:30It was building Pantheon and he would have gone completely crazy and loved this

44:35cause you've been able to build like the massive crazy stuff that he wants, wanted

44:40to do always, but just needed billions of dollars to do all within reach now.

44:47Yeah.

44:47Pretty, pretty wild.

44:49The difference, you know, and, and yeah, there's certainly like, uh, you know, I

44:54do think as well, like of people who have like passed fairly recently, like what,

44:59what would they have thought of all of this?

45:01Cause it's, yeah.

45:02Um, it is a sort of a inflection point for, for culture in general.

45:09Um, yeah.

45:11Feels, I mean, to me about the same as the internet, it's, it's, it feels

45:17like about the time when people started to get that cable internet and other

45:22people that weren't weirdos like started to use the internet too.

45:26Yeah.

45:27It wasn't just me on IRC trying to talk to girls or whatever, you know,

45:33and play video games.

45:35That was, uh, that was exactly, I mean, so, so before 2023, I'd be like, yeah,

45:41we should be doing AI and everybody would look at me like I'm crazy.

45:44And then after 2023, they're like, tell me more.

45:46And I'm like, okay, well you guys just think it's popular cause the ChatGPT.

45:51Yeah.

45:53So yeah, certainly, uh, that that's exactly what it feels like.

45:57Um, so it's going to be exciting to see where it goes and, uh, what happens.

46:01I think, um, anybody that's not paying attention is definitely completely missing

46:07out and going to be left behind and be just like the people that, you know,

46:11don't use the internet, which was my mom until it was a lot of people until I

46:20got forced onto them on an iPhone.

46:22I mean, there was a big chunk of people that still would not be online

46:26if not for the iPhone.

46:28And I guess AI is going to kind of push that way too.

46:31Uh, it's going to be all in the phones and unfortunately we'll all end up with

46:36a situation like that.

46:37Mrs. Davis show, which have you, have you seen that?

46:40No, I have.

46:42I'll read the book.

46:43You should watch, um, just watch one or two episodes.

46:45You won't want to not watch it.

46:47It's all right.

46:49I will not spoil it.

46:50It is the best AI show.

46:54It doesn't look like an AI show.

46:56It is the one with the nun on the front.

46:58Oh, okay.

47:00All right.

47:01It is, it is a completely, um, yeah, misleading picture on the front of what

47:08the actual show is, but yeah.

47:11All right.

47:12All right.

47:12Well, we've got our assignments.

47:13I'll read the book.

47:14What was the book again?

47:15We'll pop it up.

47:16Oh, uh, it is, uh, the alignment problem.

47:20The alignment problem.

47:21And yeah, go check out Mrs. Davis.

47:22All right.

47:23Anything else you want to say before we wrap it up Chris?

47:25Yeah, no, I'm just excited to, you know, have this going and we can talk about all

47:30things AI and yeah, we'll probably be looking for guests and things like that too.

47:35So just, yeah, we'll bring somebody interesting on next week and talk about

47:39something specific and then I think in the future we'll dive into some more

47:43technical stuff, details on how to do some things and if people have questions

47:47that we can obviously do some episodes answering questions about AI stuff and

47:52what people want to learn about or what things people think that we should learn

47:56about that we might not know about as well.

47:58I think it'd be a lot of fun.

47:59Really looking forward to it.

48:00Um, don't forget to like and subscribe and all that stuff and drop all this

48:05comments down below with the questions and let us know what kind of AI stuff

48:08you're working on, maybe, uh, we'll demo soon too, but at least I'll demo some

48:12stuff soon.

48:13Oh yeah, that'd be great.

48:14Yeah.

48:15All right.

48:15Well, thanks everybody.

48:16Thanks Chris.

48:18Yeah.

48:18Thanks Jason.

48:19And bye everyone.

48:20Bye everybody.

48:21Let's, uh, find that stop button.