We really need models that produce intermediate files. A .blend, a whole godot project, an actual krita/psd file.
Things like this product such cool things....all the way up until you need to move a specific tree half a meter to the left, and then you're either stuck in some prompt/seed hell or have to open an abomination of a file in an editor.
WorldGen supports exporting the generated scene as a vertices colored mesh in .ply format, which you can load into any 3D editor (like Blender). The output is also in metric scale, so you can easily align it with other assets
Neat! But not quite what I was expecting. It doesn't really make a 3D "scene". It makes a 3D panorama (building a mesh based off the depth of the panorama). Move off the center and all the things that were occluded are missing.
I guess, because we're in the time of generative content, that I was expecting it would recognize what's in the scene and fill in the missing parts. Maybe in version 2
From GitHub page:
"We also support background inpainting for better scene generation, but it's currently an experimental feature, which may not work for all scenes.
It can be enabled by setting WorldGen(inpaint_bg=True)"
I really hope there is like an "AI" flair on projects like these, i love what they aim to do but man the saturation levels mixed with "enough to make it work in theory" effort makes me want to filter out these posts.
I understand its all vibes and fun, but like at least some warnings that "this thing might lag your computer" and buttons that work.
Thanks for your interest! the AI space is moving fast, and there is always something new grabbing attention. I believe it is better to release an MVP early and keep improving it, rather than holding everything back and risking being outdated or scooped. Just like in startups, I believe fast iteration is key for AI projects too.
I like that there are so many different approaches to generate worlds. One, few, or eventually all of them will stick.
We have been playing around with something similar at Greybox too, where we ask Claude 3.7 to create Lua scripts to define the scenes with primitives (cubes, spheres, etc) that you can then move around. It's not perfect but did better than we expected!
We saw that Meta, Krea are working on "compose the scene from 3d model generations or imports as individual components", we'll give that a try too, soon.
An opensource Blockade Labs Skybox.ai sounds good and thats a lot of tooling in your pipeline so congratulations putting it all together I would 100% clone this on a HuggingFace space !!
Really nicely done, and I'm excited to see where this goes.
I'm the founder of 3d real-world scene generator https://ayvri.com - the tech was acquired a while back, but I still have people reaching out to me to ask me to bring it back, and I've told people that the way to do it today would be to generate the scenes with splats and AI.
for what it does, its cool but claiming it's useful for games/vr/xr is a bit of a stretch, in it's current state it's only good for fixed position vr (which is a pretty rough experience) as the illusion falls apart when you move your head slightly. For games you could use it for environment maps or maybe a non-interactive background in menus or visual novel style.
But kinda think claiming a panoramic generator as a "World Generator" is intentionally hoping people expect it to be more than it delivers.
We really need models that produce intermediate files. A .blend, a whole godot project, an actual krita/psd file.
Things like this product such cool things....all the way up until you need to move a specific tree half a meter to the left, and then you're either stuck in some prompt/seed hell or have to open an abomination of a file in an editor.
WorldGen supports exporting the generated scene as a vertices colored mesh in .ply format, which you can load into any 3D editor (like Blender). The output is also in metric scale, so you can easily align it with other assets
Neat! But not quite what I was expecting. It doesn't really make a 3D "scene". It makes a 3D panorama (building a mesh based off the depth of the panorama). Move off the center and all the things that were occluded are missing.
I guess, because we're in the time of generative content, that I was expecting it would recognize what's in the scene and fill in the missing parts. Maybe in version 2
From GitHub page: "We also support background inpainting for better scene generation, but it's currently an experimental feature, which may not work for all scenes. It can be enabled by setting WorldGen(inpaint_bg=True)"
Is there an example of this?
A skybox with depth is only marginally better than a skybox for any sort of 3d experience. Using the depth for occlusion would be kind of cool.
which for me doesnt make any sense as why to generate a 3D mesh that can only be seen from one point of view? why not map the panorama to an sphere?
I really hope there is like an "AI" flair on projects like these, i love what they aim to do but man the saturation levels mixed with "enough to make it work in theory" effort makes me want to filter out these posts.
I understand its all vibes and fun, but like at least some warnings that "this thing might lag your computer" and buttons that work.
Thanks for your interest! the AI space is moving fast, and there is always something new grabbing attention. I believe it is better to release an MVP early and keep improving it, rather than holding everything back and risking being outdated or scooped. Just like in startups, I believe fast iteration is key for AI projects too.
I like that there are so many different approaches to generate worlds. One, few, or eventually all of them will stick.
We have been playing around with something similar at Greybox too, where we ask Claude 3.7 to create Lua scripts to define the scenes with primitives (cubes, spheres, etc) that you can then move around. It's not perfect but did better than we expected!
https://greybox.app/blog/articles/introducing-greybox-ai-cre... has a video at the top that shows what it looks like at the moment.
We saw that Meta, Krea are working on "compose the scene from 3d model generations or imports as individual components", we'll give that a try too, soon.
An opensource Blockade Labs Skybox.ai sounds good and thats a lot of tooling in your pipeline so congratulations putting it all together I would 100% clone this on a HuggingFace space !!
Really nicely done, and I'm excited to see where this goes.
I'm the founder of 3d real-world scene generator https://ayvri.com - the tech was acquired a while back, but I still have people reaching out to me to ask me to bring it back, and I've told people that the way to do it today would be to generate the scenes with splats and AI.
for what it does, its cool but claiming it's useful for games/vr/xr is a bit of a stretch, in it's current state it's only good for fixed position vr (which is a pretty rough experience) as the illusion falls apart when you move your head slightly. For games you could use it for environment maps or maybe a non-interactive background in menus or visual novel style.
But kinda think claiming a panoramic generator as a "World Generator" is intentionally hoping people expect it to be more than it delivers.
The splat outputs remind me of Cyberpunk 2077's braindances
Optimized the webpage layout for mobile users.
Looks like only nvidia cards. Can anyone guess how much vram you need for this?
It is currently 26GB, but I will reduce it to under 24GB so that you can at least run it on a 4090.
Have you seen https://github.com/mit-han-lab/nunchaku for running flux on lower VRAM and thus faster performance?
I have put up a PR https://github.com/ZiYang-xie/WorldGen/pull/7
Is it available via API somewhere?
Not really proper 3D, more like a kind of 2.5D.
[dead]
[dead]