Importing Kwant takes too much time and uses too much memory
Importing kwant is slow and uses a lot of resources:
cwg@neron:~/work/our/kwant% /usr/bin/time -f '%e s, %M kB' python3 -c 'import kwant'
1.09 s, 95176 kB
cwg@neron:~/work/our/kwant% /usr/bin/time -f '%e s, %M kB' python3 -c 'import scipy'
0.13 s, 27516 kB
cwg@neron:~/work/our/kwant% /usr/bin/time -f '%e s, %M kB' python3 -c 'import sklearn'
0.23 s, 42984 kB
cwg@neron:~/work/our/kwant% /usr/bin/time -f '%e s, %M kB' python3 -c 'pass'
0.03 s, 9592 kB
This means that the startup of any script that perhaps only wants to use a single function of Kwant is already slowed down considerably. There are many possible applications of short-lived Kwant processes, for example a website that performs some computation dynamically.
The memory usage is also a problem. On our 48 core cluster nodes, simply importing Kwant will use more than 4 GB of memory.
The high resource usage is explained by the libraries that we use:
/usr/bin/time -f '%e s, %M kB' python3 -c 'import sympy, scipy, matplotlib.pyplot'
0.66 s, 74336 kB
But not every use of Kwant requires all of the functionality.
I thought that lazily importing some submodules of Kwant could solve the problem, and indeed it does, but it also introduces other unacceptable and unsolvable problems: !192 (comment 10417)
It seems to me that the only way to really tackle this issue is to give up on the idea that a single import kwant
makes all of Kwant's functionality available. This might be inconvenient, but this is what all the other packages seem to do.
We can do such a reorganization without breaking backwards compatibility because we can use the lazy import machinery (!192 (closed)). As said, the lazy imports are not perfectly transparent, but for keeping 99% backwards compatibility they seem more than enough.
Unless there is broad opposition to this proposal, the question that remains is how far to go. IMHO we should continue to import by default widely-used subpackages that do not have expensive dependencies, like lattice
or builder
. It would be indeed cumbersome to have to import kwant.builder
before using kwant.builder.Builder
and kwant.lattice
before using kwant.lattice.TranslationalSymmetry
. But things like plotter
and continuum
could become import-before-use. I'm not sure what to do about small, optional subpackages like digest
.
What do you think?