The game objects that are children of the game object with a Vertical/Horizontal Layout Group have Content Size Fitters in them, which is what's causing the layout issue. The Content Size Fitters have a warning on them mentioning that the parent has a layout group component and therefore it shouldn't be used here as layout should be driven by the layout group. Working as designed.