This behavior is by design.
This allows to focus more precisely on the selected object.
Changing the active object and focusing on it (F key by default), or using FPS navigation in the scene view are solutions to go around that behavior.
Another solution is also to change the SceneView camera speed to move faster in the view :)
I hope this helps,
Thanks