The inverse map is created during finalization anyway, now we just save it along with the `sites` map.