In order to strip out graphics assets from server builds, you can use the Excluded baking system assemblies option in the Project settings window for server builds (Multiplayer/build section). Graphic assets are present in the build because they are referenced by bakers being in the Unity.Entities.Graphics assembly. If you exclude the Unity.Entities.Graphics assembly from baking, the build size is getting reduced by more than the half (from 500MB to 170 MB) for server builds.