This appears to be a native visionOS bug.
Have a native metal visionOS app and register a onSpatialEvent. If you monitor the rate at which data flows into that closure while dragging, it will fall off every time you background and foreground the app using the digital crown. It starts a 90 events/sec, then as you cycle it goes to 45 events/sec, then 30 events/sec, then ~22 events/sec. (Looks a lot like 90/n where n is the number of times you background/foreground).
I've filed an Apple Feedback report for official tracking: FB19358603