Constraint-Aware Large-Scale 3D Layout Optimization via Hybrid Deep Reinforcement Learning

Large-scale 3D layout optimization requires simultaneous satisfaction of multiple heterogeneous constraints while optimizing a multi-component objective over a mixed-variable design space. Existing metaheuristic approaches suffer from exponential search space growth with problem scale, while most deep reinforcement learning (DRL) methods adopt either purely continuous or discrete action spaces that fail to capture the mixed-variable structure of real-world 3D placement problems. We present LayoutRL, a constraint-aware DRL framework integrating mixed variables within a unified hybrid policy. However, as non-overlap constraints are inherently difficult to satisfy under sparse rewards in continuous action spaces, minimum translation vector (MTV)-based action correction deterministically mitigates non-overlap constraint violations at each step, where a Beta distribution ensures numerical stability for MTV-corrected actions near boundaries. To further improve solution quality, we introduce fine-grained optimization to refine the DRL-generated solution. Experiments across $N=30, 50, 70, 100$ object scales demonstrate that LayoutRL achieves zero constraint violations at all scales. Even at $N=100$, where the metaheuristic baseline fails to produce feasible solutions, LayoutRL consistently achieves superior results, demonstrating its scalability to large-scale optimization problems. Furthermore, consistent high-quality performance across 100 test instances confirms the generalization capability.

Korean Society of Mechanical Engineers (KSME 2026)