Recursive rendering refraction is very different compared to Rasterization/PathTracing

Reproduction:
1) Open scene RecursiveBug
2) Select the quad gameobject
3) Toggle on/off Recursive rendering
4) Toggle on/off Path tracing

Expected: Refraction at least loosely matches rasterization or path tracing
Result: The refraction when recursive rendering is different than raster/path

Tested on:
Unity 2022.2.a15 + trunk hdrp

Additionally adding a video of a real world project showcasing why is this an issue (car glass reflection).