It seems it's because OP is generating the whole screen every frame / every move. Of course that will give inconsistent results.

I wonder if this approach would work better:

1. generate the whole screen once

2. on update, create a mask for all changed elements of the underlying data

3. do an inpainting pass with this mask, with regional prompting to specify which parts have changed how

4. when moving the camera, do outpainting

This might not be possible with cloud based solutions, but I can see it being possible locally.

> The "real-time" version looks awful, etc

Dang man it's just a guy showing off a neat thing he did for fun. This reaction seems excessive.

criticism is how we grow. And it seems we're more and more afraid for some reason to criticize when "AI" is involved.

Yeah, as interesting as the concept is, the lack of frame to frame consistency is a real problem. It also seems like the computing requirements would be immense—the article mentions burning through $10 in seconds.

You can do this at home on your own computer with a 40x0 consumer GPU at 1-2 fps. You have to choose a suitable diffusion model, there are models that provide sub-second generation of 1024x1024 images. The computing requirements and electricity costs are the same as when running a modern game.

I like the idea behind https://oasis-ai.org/ where you can actually try to take advantage of the 'dream logic' inconsistency of each frame being procedurally generated based on the last one. For example, instead of building a house, build the corner of a house, look at that, then look back up and check if it hallucinated the rest of your ephemeral house for you. Of course that uses AI as the entire gameplay loop and not just a graphics filter. It's also... not great, but an interesting concept that I could see producing a fun dream logic game in the future.

OP here. Thanks for the feedback. I agree that frame to frame consistency is quite bad currently. I did address that in the post, hinting at some of the techniques others have mentioned here, like in/out-painting and masking previous frames. For me, the exciting parts of this experiment was finding the opportunities and limits of realtime generation, and exploring ways of grounding generated content in a solid yet player controlled world layer.

Yeah, but I find this fascinating regardless.

This is getting into the direction of a kind of simulation where stuff is not determined by code but a kind of "real" physics.

Why does using a language/vision model feel more “real” to you than using equations which directly describe our understandings of physics?

Because our ability to simulate/render a realistic world in real time using direct equations is still very limited. We’re accustomed to these limitations and often feel “graphics are good enough”. But, we’ll always be decades behind “ILM in real time”.

The AI route has a good chance of moving us from decades behind ILM to merely “years behind ILM”.

>The AI route has a good chance of moving us from decades behind ILM to merely “years behind ILM”.

Firstly: we have very accurate models. But not at real time speeds. Games only have some 30,16, or even 11 ms to render a frame. The techniques we have are faking the real physical interactions that render farms can take minutes or hours to pump out per frame.

Secondly: Not at this performance rate. taking 100ms to render a frame is unaccatable in the concept of a game. Games are already so pressed for time budget; unless some hyper JIT happens we can't take all that budget querying an LLM.

You're comparing apples to oranges, holding up today's practical real-time rendering techniques against a hypothetical future neural method that runs many orders of magnitude faster than anything available today, and solves the issues of temporal stability, directability and overall robustness. If we grant "equation based" methods the same liberty then we should be looking ahead to real-time pathtracing research, which is much closer to anything practical than these pure ML experiments.

That's not to say ML doesn't have a place in the pipeline - pathtracers can pair very well with ML-driven heuristics for things like denoising, but in that case the underlying signal is still grounded in physics and the ML part is just papering over the gaps.

The question was "why does it feel more real", and I answered that - because the best AI generated images today feel more real than the best 3D renders, even when they take all the compute in the world to finish. So I can imagine that trend going forward into real-time rendering as well.

I did not claim that AI-based rendering will overcome traditional methods, and have even explicitly said that this is a heavy assumption, but explained why I see it as exciting.

I think we'll have to agree to disagree about well done 3D renders not feeling real. Movie studios still regularly underplay how much CGI they use for marketing purposes, and get away with it, because the CGI looks so utterly real that nobody even notices it until much later when the VFX vendors are allowed to give a peek behind the curtain.

e.g. Top Gun Mavericks much lauded "practical" jet shots, which were filmed on real planes, but then the pilots were isolated and composited into 100% CGI planes with the backdrops also being CGI in many cases, and huge swathes of viewers and press bought the marketing line that what they saw was all practical.

I find it odd that you're that bothered by uncanny valley effects from game rendering but apparently not by the same in image model outputs. They get little things wrong all the time and it puts me off the image almost instantly.

>Hardware requirements for games today keep growing, and yet every character still has that awful "plastic skin", among all the other issues

That's because the number of pixels to render onto keep growing. Instead of focusing on physically based animations and reactions, we chose to leap from 480p to 720p overnight, and then to 1080p in a few more years. Now we quadrupled that and want things with more fidelity with 4x the resolution of last generation.

> images created by image models today look fully realistic.

Because they aren't made in real time (I'll give the BOTD for now and say theya are "fully realistic". Even this sample here claims 100ms. Rendering at 6-7 seconds per frame isn't going to work for any consumer product at any point in gaming history.

>Even if realism is not what you're aiming for, I think it's easy to imagine how this might change the game.

not in real time rendering. I am interested to see if this can help with offline stuff, but we're already so strapped for performance withoout needing to query an "oracle" between frames.

as of now, I'm not even that convinced by the Nvidia 5X series of frame interpolation (which would only be doable by hardware manufactureres).

Todays game graphics can render in any style the artist chooses. Its actually AI that has the inability to produce a unique style.