SceneX: Procedural Controllable Large-scale Scene Generation

Mengqi Zhou1,3,4,6*, Yuxi Wang2*, Jun Hou2, Shougao Zhang8, Yiwei Li1, Chuanchen Luo5, Junran Peng7‡, Zhaoxiang Zhang1,2,3,4,6,
1University of Chinese Academy of Sciences (UCAS), 2Centre for Artificial Intelligence and Robotics (CAIR), 3Institute of Automation, Chinese Academy of Sciences (CASIA), 4New Laboratory of Pattern Recognition (NLPR), 5Shandong University, 6State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), 7University of Science and Technology Beijing, 8China University of Geosciences Beijing,
*Equal contribution, ‡Corresponding author
Paper arXiv Code (Coming Soon)

Abstract

Developing comprehensive explicit world models is crucial for understanding and simulating real-world scenarios. Recently, Procedural Controllable Generation (PCG) has gained significant attention in large-scale scene generation by enabling the creation of scalable, high-quality assets. However, PCG faces challenges such as limited modular diversity, high expertise requirements, and challenges in managing the diverse elements and structures in complex scenes. In this paper, we introduce a large-scale scene generation framework, SceneX, which can automatically produce high-quality procedural models according to designers' textual descriptions. Specifically, the proposed method comprises two components, PCGHub and PCGPlanner. The former encompasses an extensive collection of accessible procedural assets and thousands of hand-craft API documents to perform as a standard protocol for PCG controller. The latter aims to generate executable actions for Blender to produce controllable and precise 3D assets guided by the user's instructions. Extensive experiments demonstrated the capability of our method in controllable large-scale scene generation, including nature scenes and unbounded cities, as well as scene editing such as asset placement and season translation.

Text-based scene generation and scene editing demo

Text-based personalized editing of 3D assets demo

Based on PCGHub and PCGPlanner, the proposed SceneX can create large-scale 3D natural scenes or unbounded cities automatically according to user instructions.

The proposed SceneX can create large-scale 3D natural scenes or unbounded cities automatically according to user instructions. The generated models are characterized by delicate geometric structures, realistic material textures, and natural lighting, allowing for seamless deployment in the industrial pipeline.

The generated models are characterized by delicate geometric structures, realistic material textures, and natural lighting, allowing for seamless deployment in the industrial pipeline.

The framework of SceneX

The proposed SceneX can create large-scale 3D natural scenes or unbounded cities automatically according to user instructions. The generated models are characterized by delicate geometric structures, realistic material textures, and natural lighting, allowing for seamless deployment in the industrial pipeline.

SceneX framework converting user text input into diverse 3D scenes through four stages: scene decomposition stage, terrain generation stage, objects generation \& retrieval stage and assets placement stage.

Citation

@article{,
  author = {Mengqi Zhou and Yuxi Wang and Jun Hou and Shougao Zhang and Yiwei Li and Chuanchen Luo and Junran Peng and Zhaoxiang Zhang},
  title  = {SceneX: Procedural Controllable Large-scale Scene Generation},
  journal = {arXiv preprint arXiv:2403.15698},
  year = {2024},
}