Custom PropertyDrawers are not being released

Add the following script to a project:

using System;
using UnityEditor;
using UnityEngine;
[Serializable]
public class MyClass
{
}
[CustomPropertyDrawer(typeof(MyClass))]
public class MyClassDrawer : PropertyDrawer
{
    public override void OnGUI(Rect position, SerializedProperty property, GUIContent label)
    {
        GUI.Label(position, label);
    }
    public MyClassDrawer() => Debug.Log("MyClassDrawer");
    ~MyClassDrawer() => Debug.Log("~MyClassDrawer");
}
public class Test : MonoBehaviour
{
    public MyClass myClass = new MyClass();
}

1. Add the Test component to a GameObject
2. Select the GameObject, notice the property drawer is created. 
3. Deselect and reselect. A new drawer is created every time but the finalizer/destructor is never called.
4. Enter playmode or change a script. All the finalizers are called.

It seems that something is holding onto the property drawer and preventing them from being garbage collected. This makes it hard to clean up after a drawer and results in issues such as this https://forum.unity.com/threads/application-shutdown-cleanupmono-never-finishes.1118140/page-2#post-9171365

From what I can see, the drawer is stored in Editor.propertyHandlerCache however when the Editor is released (OnDisable etc) the drawer is still not released, even if you force propertyHandlerCache to null. It appears that it is being referenced somewhere. 

Tested on 2019.4, 2021.3 and 2023.3. They all have the issue.

Calling UnloadUnsedAssets and GC.Collect does not help.