add inverse of `sites` mapping to finalized builders

The inverse map is created during finalization anyway, now
we just save it along with the `sites` map.