The GC Unity currently uses is not a moving GC, so fragmentation is possible. Types with finalizers can cause additional fragmentation, as those objects must live a bit longer - waiting for their finalizers to be run.
So unfortunately this is expected behavior. As we move to use the CoreCLR runtime, Unity will use a moving GC which will handle heap fragmentation much more effectively.