Performance loss when enabling and disabling ShadowCaster2D

Steps to reproduce:
1. Open the user’s project
2. Enter Play mode
3. Notice FPS in the top left corner
4. Press the “Test” button (this button enables and disables ShadowCaster2Ds 500 times)
5. Notice FPS

Expected result: FPS does not change
Actual result: FPS gets lower (from ~500 dropped to ~20)

Reproducible with: 2020.3.40f1, 2021.3.11f1, 2022.1.18f1, 2022.2.0b10, 2023.1.0a13

Reproduced on: macOS Monterey 12.5.1 (Intel)

Note: Also reproducible in the Player