Hacker News
How to engineer an AI harness for game development
andai
|next
[-]
> Any time I tried to use an LLM for a game-y thing, it would fall over.
You know it's funny. I did a bit of LLM-assisted game dev this year.
I spent a few weeks working on a simple Pong game, just polishing it, making small changes here and there. Frontier models couldn't change or add a feature without breaking something else. It was still more productive than working "solo" but quite annoying.
Meanwhile, I asked a model to make an entire game -- about 100x more complicated -- and it just one-shotted it perfectly, no issues.
I'm not sure how both of those things can be true at the same time. Something something "in-distribution" ?
YesBox
|next
|previous
[-]
kkukshtel: You should keep that info, with links, even if its links to your site. Dont burden the reader.
Edit: I read the first half of the article then skimmed the rest (I dont use LLMs outside of occasional searches).
I've been a gamedev for five years. I have a hunch LLMs will not take over game development because of the necessity to control many, many small details to shape a great experience for the player, and because some parts of development cannot be described with words.
Small details: My hunch is when it comes time to polish and fine tune all the moving parts, this process will remain much faster to do by hand. It will be faster to update a specific field or line of code and test/iterate than to prompt your way through that process (the last 20% of the work takes 80% of the time)
Describing without words: An artist with a vision can only bring that vision to life with body movements. It is not possible to communicate all the details that go into a great piece of art with words. The colors, lines, shapes, positioning, specific emotions, history... is beyond spoken language.
augment_me
|root
|parent
|next
[-]
You say "vision, colors, positioning, etc", yet you have Candy Crush being the most played game on mobile for 13 years which simply employs direct signals and ques from cognition and cognitive neuroscience to achieve this.
The games that you describe are a small subset of what the general public is playing or is interested in.
This can further be pushed into a monoculture by shaping the players expectations about what a game should be. Maybe Skyrim has too much walking and dead-time, it's better to optimize for fast combat and action, like in Fortnite, and mold the playerbase in a way which makes them susceptible to simpler games more tuned to neurophysiology.
montagg
|root
|parent
[-]
augment_me
|root
|parent
[-]
I am saying that a small subset of people care about this given what people choose to play in practice. It's currently much, much easier to not care about these things and focus on the cognitive aspects to capture player attentive, and the human cognitive incentives are not stacked in favor of OPs view.
gen220
|root
|parent
|next
|previous
[-]
They would still come in at the end of the day to add their brush strokes to pieces of the canvas that they thought needed extra attention. We still refer to those paintings as if they were made by the Great Master, and justly so because their artistic imprint is felt in the final work.
In the same way where if you and I were asked to write the same game with an LLM, they would come out looking pretty distinct from each other. And if we wrote them by hand, they would be distinct from each other and the LLM-assisted ones again. All of them are still art and games, though, it'd be silly to call them anything else.
I agree with you that polish and fine-tuning is faster to do by hand when you are well-appraised of the foundations. I think a lot of the challenge of developing Real Software with LLMs today is maintaining the appropriate proximity to the work, when we are no longer holding the paint brush ourselves, so to speak.
lukan
|root
|parent
|next
|previous
[-]
Depends if what you want to do is new, or has been done a thousand times before
"Describing without words: An artist with a vision can only bring that vision to life with body movements. It is not possible to communicate all the details that go into a great piece of art with words. The colors, lines, shapes, positioning, specific emotions, history... is beyond spoken language."
And you absolutely can use sketches, pictures, drawings as input to the AI.
So in general I agree, at least right now it needs a lot of fine tuning. About the future I make no prophecy, I did not see AI coming this way, so why would I be right about AI in the future now.
I am making a novel game right now with heavy help from Fable and it is awesome. Because even though my game is novel - parts of it are not, but standard and I reinvented the wheel way too often before and am glad don't have to do that anymore.
drusepth
|root
|parent
|next
|previous
[-]
I think you're mostly right here -- however, I think you've framed prompting as an all-or-nothing (or maybe the OP did, as I only skimmed the article too...). LLMs are great for some things (scaffolding, review, refactoring, etc) but way less efficient at others (tweaking and letting the artist eye shine).
I think the crux is this: you shouldn't be asking an LLM to make these tweaks for you; you should be asking an LLM to build the tools that let you make these tweaks by hand faster or more effectively.
For example, we have probably a hundred tools [0] for quickly making all kinds of changes and tweaks in one of our Steam games, that are of course all done by hand. The tools were built by LLMs based on what would speed up our existing by-hand workflows, and once they're built they're just tools that let our people shine.
1. Do a thing by hand
2. Have LLMs build a tool that makes it easier to do that thing by hand
3. ????
4. An artist with a vision can bring that vision to life faster
d3vnull
|root
|parent
|next
|previous
[-]
Not easy to make statements about things you don't use.
If you're developing software, LLMs will be ubiquitous eventually. They aren't going to tune the turn radius or how long to hold a freeze frame but they will bang out a UI to let you do that quickly with quick turn around times.
So many simple little things that are helpful but not really worth doing will now be worth doing because the implementation cost is a fraction of what it once was.
bob1029
|next
|previous
[-]
The author appears to be on track to learn an important lesson about LLMs and art. LLMs have completely solved the coding side of game development. But this was never the hard part. Not even close.
The fixed randomness and determinism thing sounds great on paper (all of this does), but in reality this is an extremely confining way to build a video game. Ensuring you maintain determinism across many platforms is where things get super nasty.
If you want a turnkey AI game development experience, why not double down on diffusion? This sidesteps virtually all of the tooling and integration concerns. There is no game engine to integrate with if you use a diffusion model.
apf6
|root
|parent
|next
[-]
This is revisionist history. Coding was definitely one of the hard parts!
w4yai
|root
|parent
[-]
dimgl
|root
|parent
|next
[-]
pton_xd
|root
|parent
|next
|previous
[-]
This is not quite true. Currently you still need to give them guidance on what type of architecture would best work for your game, or after many iterations on gameplay you'll end up with terrible performance and a complete mess of a codebase. I expect that will be solved sooner rather than later, though.
LogicFailsMe
|root
|parent
[-]
PS no, I am not interested in your alternative harness that will supposedly 100% trust me bro fix all these issues
icantevenhold
|root
|parent
[-]
Fwiw I tried the same thing ~a year ago and the result was horrible; like so much worse.
Sure it’s not AAA, the art looks like shit and most people would probably not enjoy it - but it was fun for me to play and I’ll work on it a bit more to see if I can bring it into a state where others can enjoy it too.
glimshe
|root
|parent
|next
|previous
[-]
But many top games have complex engines and online services that require technical mastery beyond the current AI capabilities. For those genres, engineering is a huge barrier of entry.
For a not so extreme example, you won't vibe code Cyberpunk 2077 while just taking care of art and gameplay ideas. Gosh, humans could barely pull that one off!
rrradical
|root
|parent
|next
|previous
[-]
Why 'nasty'? I agree it's a problem to solve, but you just use a fixed-point math library and it's fine. Determinism is freeing, not confining. It allows you to do things like rollback netcode, replays and easy debugging. You can't just say that 'reality' happens to be what you in particular are used to.
avaer
|root
|parent
[-]
Assuming the agent didn't put in a native nondeterministic trig function deep in the code instead of using the library one, it's an easy mistake to make and hard to debug in a big game. It won't even show unless you have a robust CI matrix. It can indeed be "nasty" to maintain.
Also determinism might work for simulation state, but shader/gfx differences across platforms can be really nasty and a library can't save you. Not even something like WebGPU. Also if you're doing any sort residency streaming dependent on GPU timings trying to make that deterministic will be a nightmare.
rrradical
|root
|parent
|next
[-]
Take a given start point; feed a set of inputs; compute a checksum on the resulting state. Check that the checksum matches on different platforms.
And if your language is good you can parameterize your logic on a particular numeric type, so that a particular fixed point implementation is consistently used
mypalmike
|root
|parent
|previous
[-]
I’ve worked on games with intentional determinism, and it’s a very valuable tool. A playtester can hand a dev a recording of the game inputs (including clock) for whatever bug they encountered, and the dev can fast forward to the exact point and repro it 100%. The amount of time saved chasing down difficult bugs is enormous.
wccrawford
|root
|parent
|next
|previous
[-]
simonw
|root
|parent
|next
|previous
[-]
I learned that one the other day - I've been building a little RTS game with a replay mechanism, and replays in Safari broke where they worked in Chrome. Opus 5.5 fixed it by replacing calls to things like Math.sin() and Math.cos() with its own deterministic functions: https://gist.github.com/simonw/7df96ed7296c69b70a9990102cae0...
theptip
|next
|previous
[-]
Meta level, just push your agent to make things more legible, it’ll figure it out.
One other observation, remember that your agent is infinitely patient. So prompts like “build an interactive design doc with videos of each game mechanic in the style of Factorio dev diaries” can make reviewing changes easier. The general idea, all the small QoL dev tools that you don’t have time for, you can just spawn a side agent to build; if they are orthogonal to your main thread you can just progress them in the background.
One thing I think game dev has not (AFAIK) invested as much in is observability. There’s a lot of perf instrumentation available in debug builds, but things like measuring DPS, health, etc as an exported metric time series I have not seen before. I think agents will benefit from this disproportionately more than humans, eg for inspecting progression curves and balance.
gcampos
|next
|previous
[-]
armcat
|next
|previous
[-]
Because I am so-and-so pixel artist, the biggest unlock for me has been in rapidly iterating with graphics assets and sprites, and it's actually helped me become a better artist and animator (I built my own little tool for this [2]). The rest Codex or Claude Code seem to handle incredibly well (plus I have 30 years of programming behind me).
mawadev
|next
|previous
[-]
roger23
|next
|previous
[-]
mycocola
|next
|previous
[-]
eamsen
|root
|parent
[-]
Is something beautiful worth less if it was made by a probabilistic machine?
mycocola
|root
|parent
[-]
dudeinhawaii
|next
|previous
[-]
With the advent of AI, I've been _trying_ to use AI to accelerate game development and have come to a few conclusions.
1. AI models want to make everything deterministic. This sounds like a truly sound way to write a verifiable program. Games are not verifiable simulators except in the niche of ...simulators. Even in those cases, the vast infrastructure devoted to this leads to a lot of churn, bugs (AI forcing things to align with the verifier), and rigidity.
2. Making a game != Fun. This one is obvious but needs to be stated. Anyone on earth right now can say "make me doom" or "make me a super mario clone" and Opus/Astra will dutifully comply and make something generic and bland. Once the novelty of "oh wow it actually built it" wears off, you're left with a mediocre clone, lacking depth, that no customer would want hoisted upon them.
3. Content and novelty are the hard parts. AI models always create the mediocre middle. There's real work (and interesting work) involved in both inventing novel gameplay systems which LLMs will never stumble upon despite infinite 'brainstorming prompts'. But, AI is 'OK' at taking your novel idea and proposing directions -- that you can then strike down viciously until something interesting emerges.
On the content side, this has always been a truism for an independent developer. AI art tools aren't quite there yet. There's image-to-3D and there's Mixamo for animation but we're not there yet. Nearly though! But again, will every game look like a bland generic AI generated image with bland AI generated models?
4. AI models are crap at building re-usable large-scale architectures. Even building with Opus 5.5 and Astra they will happy design themselves into a corner. Yes, they can _also_ design themselves out of that corner at speed - with a lot of pain and code churn and mess. I've found it vastly better to iterate slowly, using your experience (if you have it) to structure the core architecture and systems the way you want (AI writing the code) -- and then build upon that solid base -- taking special care to reject updates that butcher your architecture. I think this applies to most coding projects actually.
5. Game dev currently surfaces very clear differences in model capabilities with Opus 5.5 leading the pack (in this domain) and Fable 5.1 and Astra/GPT-6.1 Sol following a decent distance behind. Part of this is wrong usage of the tool. If you say "make a game" -- Opus will outperform the others by a large margin -- in making a generic "game" that was undefined (a paragraph of text is not a defined game). In these cases, the model has filled in all the gaps in undefined areas with average 'what probably looks good' content. You're left with something devoid of creativity. This is the problem with the myriad three.js tech demos.
If you instead have a clear understanding of what you're building, models (nearly all SOTA) will get you there cheaper by building what you've asked without speculation. I don't believe you should hand off the one human worthwhile trait, creativity, to the AI -- so "Opus build a game" is indeed the wrong way.
6. It seems hard (and pointless) to collaborate with other human developers when AI coding on smaller/indie projects. Each AI model is trying to "solve the whole thing". The speed also makes it hard to stop at arbitrary boundaries and promotes a rushed feeling or conversely a sluggish feeling (AI did all the stuff, now I'm waiting to integrate with a friend/collaborator). This is probably easily solvable but has thus far felt weird when I've collaborated.
7. The return of custom engines? While models do great with Godot and Unreal, this is the chance to build custom engines like the golden years of the 2000s. OK, this one is a bit of nostalgia but I miss the world of dozen of game engines each with their own unique look and feel. Otherwise you get XBOX 360 era where every "realistic" character looks like a Gears of War meathead. Every adventure game has the same mechanics, you're hooking something, you're scooting under something, you're scooting between walls or things, you're on collapsing things that require either button sequences or hitting your 'hook' (every adventuring has some hook of infinite rope of course). OK, this entire bullet is just a rant about samey-ness. LLMs should unlock our human creativity and allow experimentation outside of that cost imposed same-ness.
All things said, it's _clear_ that the acceleration is here if you know what you want. You can go from concept to prototype in a few hours. You then have to be disciplined enough to step back and rebuild that thing (with AI) correctly and towards a useful objective that you've thought up. I'm looking forward to went content creation catches up. Or perhaps that's a sad day when creativity vanishes.
avaer
|next
|previous
[-]
For example, the Thing approach is reasonable for a small game, but it's a dead end for a performance budget or layered subsystems. In a reasonably sized 3D game, iterating and copying 30 subsystems of 1000's of Things is impossible to do in frame time, the only fix is a total rewrite of literally everyThing. That's why there's ECS, SOA, etc. which needs a different kind of harness.
Also, art completely dominates the production cycle of modern games, which (even if AI generated) needs its own category of harness not mentioned here. And AI can be very good at optimizing, but samply won't diagnose your overdraw, and GPU needs its own ablation and attribution.
For the time being there's no panacea and we still need "game developers", though the bar for prototypes (and certain genres) is lowered.
nylonstrung
|root
|parent
|next
[-]
It's conceptually interesting but I don't know it is instructive beyond simple projects
kkukshtel
|root
|parent
|previous
[-]
nsagent
|root
|parent
[-]
That said, I'm glad you shared your thoughts and experience. It's helpful to see what people try.
wslh
|next
|previous
[-]
Kuyawa
|previous
[-]
Oh no they're not. I built dozens of small games for the web and five games in Swift and they're spectacular, getting better since day one. Not like Call of Duty but just fun and bug-free enough to play them every day