Numerical instability in symmetry finder

The check on this line is prone to numerical instability: https://gitlab.kwant-project.org/qt/qsymm/-/blame/master/qsymm/symmetry_finder.py#L492 Because only the set of keys in the original and transformed model are compared, it can happen that new keys show up in Rmodel that have order machine precision small coefficients, and the model is falsely identified as having changed under the transformation. A fix should make sure that Rmodel is cleaned up, with keys having small coefficients removed.