I understand that it's not ideal to be unable to easily create the layout you'd like in the Settings window. The Settings window, just like the Inspector, is designed to be a "scroll area" where content takes only the space it needs and if it needs more space than available, as ScrollView (that you don't control) makes sure your UI is still accessible. All Settings panes use this same ScrollView parent so that it's not necessary for each pane to implement its own (likely unique) scrolling UX.
What you're trying to do goes against the UX pattern established in the Settings pane, hence the blockers. If you _really_ want to create something unique (for the Settings window), you definitely can, but you'll need to jump through some hoops.
Those hoops require the creation of custom parent VisualElement that walks up the UI tree, finds the parent ScrollView, and registers a GeometryChangeEvent callback on it. With this, you can always get the current size of the ScrollView and size your contents so that they always take exactly that size (never invoking the scroll bars). You can then create your own ScrollView inside for the dynamic part of your UI.
You're not wrong that alternate layouts should be easier, and we might invest in this in the future, but it doesn't make the current setup a "bug".