The problem with rotating a Collider (that belongs to a Rigidbody) every frame is that the mass distribution is constantly changing and the center of mass needs to be recalculated each time. The floating point imprecisions from these calculations stack over time and the whole thing slowly drifts away.
Workaround/solution: rotate the Rigidbody, not the Collider to avoid center of mass drift each tick.