Plotter test fails when executed more than once

This itself is not a problem, but (probably) the same problem shows up when running nosetests with more than one process. Also, this function should seed the RNG.

$ python -c 'import kwant.tests.test_plotter as tp; tp.test_mask_interpolate(); print "2nd"; tp.test_mask_interpolate()'
2nd
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "kwant/tests/test_plotter.py", line 169, in test_mask_interpolate
    assert len(w) == 1
AssertionError

Added priority::low label

Added ~7 label

Added type:bug label

Status changed to closed

Status changed to reopened

Added difficulty::medium label

Cannot reproduce on master, @cwg do you confirm?

I actually see this again when I use kwant.rmt in some tests in a completely different file. This appears to be related to the way that mask_interpolate works when it checks whether there are any sites within a certain distance of one another. It seems to take a random sample of sites and only checks inter-site distances between these sites.

Mentioned in issue #66 (closed)

Mentioned in commit c82d12c3

Mentioned in merge request !50 (merged)

Mentioned in commit f2ba3eb2

Added In Progress difficulty::easy and removed type:bug priority::low labels

Added type:bug label

Removed difficulty::medium label

closed by !50 (merged) (not sure why this was not auto-merged

Status changed to closed

Removed In Progress label

!50 (merged) does not fix this bug. Proof:

$ git describe
v1.1.2-11-g296ee05
$ python -c 'import kwant.tests.test_plotter as tp; tp.test_mask_interpolate(); print "2nd"; tp.test_mask_interpolate()'
2nd
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "kwant/tests/test_plotter.py", line 169, in test_mask_interpolate
    assert len(w) == 1
AssertionError

I think I know how to fix it now, though.

Status changed to reopened

Well, you are not looking at the correct commit; I have

$ git describe 
v1.1.2-12-gf2ba3eb

But It seems, indeed that this doesn't solve the problem.

As far as I can tell the problem is in these lines in mask_interpolate.

The reason we don't get the same answer each time is because numpy's RNG is in a different state each time the function is run.

To be honest I don't really understand this code anyway, as the sites to compare are selected using numpy.random.randint which will in general give duplicate entries. I am pretty sure that instead we want to select unique sites.

You are right that the "proof" I posted was incorrect, but I knew from reading the code that Anton's fix cannot solve this problem. And this is indeed the case.

The basic problem of this bug (I first noticed it when running nosetests in multi-thread mode) probably cannot be fixed. The warning.catch_warnings context manager is not thread safe! We have multiple tests (and also code in Kwant) that use this context manager, so we're out of luck.

There's another problem, which is perhaps even unrelated to the above one. When debugging the multi-threaded problem, I managed to trigger a problem by running the same test twice:

$ python -c 'import kwant.tests.test_plotter as tp; tp.test_mask_interpolate(); print "2nd"; tp.test_mask_interpolate()'
2nd
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "kwant/tests/test_plotter.py", line 166, in test_mask_interpolate
    assert len(w) == 1
AssertionError

The above can be observed with the current version of the code, but let's quickly fix another small problem (that is unrelated I believe) and rewrite the test in question so that it has no longer the side effect of executing warnings.simplefilter("ignore"):

def test_mask_interpolate():
    # A coordinate array with coordinates of two points almost coinciding.
    coords = np.array([[0, 0], [1e-7, 1e-7], [1, 1], [1, 0]])

    with warnings.catch_warnings(record=True) as w:
        warnings.simplefilter("always")
        plotter.mask_interpolate(coords, np.ones(len(coords)), a=1)
        assert len(w) == 1
        assert issubclass(w[-1].category, RuntimeWarning)
        assert "coinciding" in str(w[-1].message)

    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        assert_raises(ValueError, plotter.mask_interpolate,
                      coords, np.ones(len(coords)))
        assert_raises(ValueError, plotter.mask_interpolate,
                      coords, np.ones(2 * len(coords)))

To me, this looks good now, but still len(w) == 0 during the second invocation.

When the last paragraph (the one with the two assert_raises) is commented out, the problem disappers. I do not understand why. This paragraph should not have any effect on the preceding one. Any ideas?

This is related to the thing that I commented on above: mask_interpolate itself uses the numpy RNG, so every time you call mask_interpolate you are altering the state of the RNG.

That's true, but it's unrelated to the problem at hand. Try modifying mask_interpolate so that the warning gets raised always:

-    if min_dist < 1e-6 * np.linalg.norm(cmax - cmin):
-        warnings.warn("Some sites have nearly coinciding positions, "
-                      "interpolation may be confusing.",
-                      RuntimeWarning)
+    # if min_dist < 1e-6 * np.linalg.norm(cmax - cmin):
+    warnings.warn("Some sites have nearly coinciding positions, "
+                  "interpolation may be confusing.",
+                  RuntimeWarning)

The above problem (len(w) != 1) persists.

ok, so if I do

with warnings.catch_warnings(record=True) as w:
       warnings.simplefilter("always")
       plotter.mask_interpolate(coords, np.ones(len(coords)), a=1)
       plotter.mask_interpolate(coords, np.ones(len(coords)), a=1)
       assert len(w) == 1
       assert issubclass(w[-1].category, RuntimeWarning)
       assert "coinciding" in str(w[-1].message)

Then I immediately get AssertionError for the first assert. It appears that the always filter is not doing what we think it does...

Or the catch_warnings context manager is not doing what we expect

Your code calls mask_interpolate() twice, hence two warnings occur and len(w) == 2. So far everything behaves as it should.

Sorry for not checking it thoroughly enough. I'll investigate in more detail.

Mentioned in merge request !48 (closed)

Ok, I think I finally understand. The following script won't show any warning (with Python 2)

import warnings

def func():
    warnings.warn("Blabla", RuntimeWarning)

warnings.simplefilter("ignore")
func()
warnings.simplefilter("always")
func()

So, calling a function that produces a warning (and does otherwise nothing) with warnings set to ignore is not the same things as not calling that function. Somewhere inside CPython this particular warning is "ticked off" and it is no longer triggered.

Changing "ignore" to "always" above has the effect of not ticking off the warning and thus it appears twice. Once a warning is is ticked off, however, there is no way to trigger it again.

But wait, that's only half of the story! The described behavior occurs only with Python 2, in Python 3 the above script will show a single warning, as originally expected! (A quick search didn't unearth any documentation or changelog for this, however.) This also means that this "bug" does not exist in Kwant 1.2.

We could fix the behavior for Kwant 1.1, but since the problem is so obscure, I propose to try to forget about all this as quickly as we can!

Status changed to closed

It turns out that this issue was due to a bug in CPython that was fixed for Python 3.5.

Since this issue is only visible in very special circumstances and manifests itself only in a failed test, I'm for simply waiting until it resolves itself with time.

Meanwhile, there was an unrelated minor problem with the same test, that is now fixed in 12929415.