This works as intended.
Changes are not persisted during play mode. So changes happening to ScriptableObjects will only be applied to memory. But because the ScriptableObject is selected, the memory is kept around across entering/exiting playmode but never persisted to disk.