Improve materials on testing and test driven development
This is a summary of the discussion we (@anton-akhmerov @jbweston and me) had over lunch.
The goal is to teach the students the benefit of tests and how to use tests in their daily work. The ideas described below are heavily inspired by the improvements we did to the Git materials last year.
The idea is to create the following scripted events to let the students experience first the problem that is solved by tests and then how to use these properly.
- The student works in a damped harmonic oscillator lab and has access to a Git repository containing basic analysis code (data extraction, fitting, plotting).
- The student get's an angry email from Prof. Smith that they should not send him data containing fits that are obviously wrong. The email contains an attached image of a some data (amplitude vs time) of a damped oscillation with a fit in red that is obviously not matching together with a pointer to the analysis notebook.
- The problem is constructed to correspond to a hardcoded bug in the analysis code. This can be either in the guess function corresponding to the fit (hardcoded zero phase for the damped cosine) or in the model that is used, this is a detail that needs to be worked out. The most obvious solution to this problem is to fix this hardcoded feature and replace it with the value that is needed. We need to think a bit in how we can ensure that the student will indeed implement a solution that breaks the old code.
- At this point an angry email by senior PhD student Mike is send to the students (possibly projected on the beamer), in which the senior student complains about this "fix" breaking all his advanced analysis. It is now the job of the student to fix this. Senior PhD student Mike attaches a notebook containing his (dirty) code that is now borken.
- The student should first go back to the old version of the code to verify that the notebook of Mike is working (refer Git)
- The student will now need to investigate what is b0rk3n and how to fix it. At this point the student is introduced to the idea of writing a (passing) test for his own functionality and another (failing) test for the code of Mike. How to run tests using py.test (pytest) should also be explained at this point.
- After writing the tests the problem can be addressed and changes can be made in the knowledge that the old functionality is not broken.
- After completion it is possible to show (but not set up for this exercise as this is out of scope?) the possibility of running tests online and getting automated notifications when someone breaks tests.
This is a good point to have a general discussion on how to use tests in your lab/group and how to include this in collaborative work. Using tests in the advanced projects that follow should be encouraged.
@anton-akhmerov @jbweston Let me know if this corresponds to what we discussed during lunch and if you have any more ideas, comments questions and/or suggestions .