Colliders with IgnoreCollision take significantly longer to be destroyed

How to reproduce:
1. Open the “IN-111471.zip“ project
2. Open the “SampleScene”
3. Enter Play Mode
4. Select the “PhysicsColliderCleanupTester“ GameObject
5. Enable and disable the “PhysicsColliderTester“ Component a few times to stabilize the performance
6. Observe the times logged in the Console
7. Enable/Disable the “Use Physics Ignore Collision“ checkbox
8. Enable and Disable the “PhysicsColliderTester“ Component once
9. Observe the Console

Expected result: The time logged with Ignore Collisions enabled is similar to the disabled.
Actual result: The time logged with Ignore Collisions enabled is significantly longer than the disabled.

Reproducible in: 2021.3.55f1, 6000.0.56f1, 6000.2.2f1, 6000.3.0a6

Reproduced on: Windows 11 Pro (24H2)
Not reproduced on: No other environment tested

Notes:

• The enable and disable times are quite inconsistent. However, the time difference between IgnoredCollision enabled and disabled is very visible
• Creating the colliders takes around the same time regardless of IgnoreCollision
• Time taken to destroy the colliders rises exponentially with the number of colliders