After giving it a second thought, it is better to keep the system this way. The alternative being:
For opaque objects, if perfectly smooth, no clamping if perf or quality with one bounce
For opaque objects, if perfectly smooth, no clamping if it hits an unlit or the sky.
For both cases, it will introduce a big visual difference as soon as the surface is not perfectly smooth, which will be a big issue.