make a BalancingLearner strategy that compares the total loss rather than loss improvement

Closes #103 (closed)

added 2 commits

• 6bf8d946 - fix bug in balancinglearner (this was an edge case, again)
• a97c66a2 - implement balancing_learner_v2

Compare with previous version

I think it should work like this, but i haven't tested it out (there must be bugs for sure)

I will continue with this tomorrow as I have to go at the moment

In fact it does work as a drop-in replacement. It also handles #104 (closed) better than the current balancinglearner, although it won't solve the underlying "issue".

Further I am not entirely sure how I should measure performance. Maybe we could have like a graph: I can think of 2 factors that may be important: the number of cores available and the number of learners you pass into the balancinglearner. As I can't modify the number of cores, I think this can be simulated by just using multiple processes.

We could either time by just timing the total elapsed time, although this gives no indication on the accuracy of the measurement, I suppose it does give an estimate.

I see that you wrote that you fixed a bug in the BalancingLearner but I don't know which bug.

Would it be possible to add a test for that?

Yes, I will do that, Maybe we should even cherry pick it into a separate MR I suppose

BTW: my personal measurements indicate that:

BalancingLearnerV2 is somewhat faster that V1 in combination with Learner1D.

BalancingLearnerV2 is quite a bit slower than V1 in combination with Learner2D.

BalancingLearnerV2 is many times faster than V1 in combination with LearnerND. (this is of course due to the with restore(self) which is extremely slow.)

I will however check if I can increase the performance of BalancingLearnerV2 with the Learner2D. I know it can be done when we would accept a slight error, yet preferably we wouldn't of course.

Would it be possible to add a test for that?

added tests and fixed it in !113 (merged)

mentioned in issue #60 (closed)

added 42 commits

• a97c66a2...c816cf67 - 41 commits from branch master
• 5e4b8364 - make a BalancingLearner strategy that compares the total loss rather than loss improvement

Compare with previous version

unmarked as a Work In Progress

@Jorn I've changed your code a bit and implemented the V2 as a BalancingLearner strategy that can be set. What do you think?

added 1 commit

• 2faf8055 - make a BalancingLearner strategy that compares the total loss rather than loss improvement

Compare with previous version

added 1 commit

• 71e65bd3 - make a BalancingLearner strategy that compares the total loss rather than loss improvement

Compare with previous version

changed title from Resolve "(BalancingLearner) make new balancinglearner that looks at the total loss rather than loss improvement" to make a BalancingLearner strategy that compares the total loss rather than loss improvement

assigned to @jbweston

added 1 commit

• ade2c084 - make a BalancingLearner strategy that compares the total loss rather than loss improvement

Compare with previous version

added 8 commits

• ade2c084...953ff845 - 7 commits from branch master
• 3696d4c1 - make a BalancingLearner strategy that compares the total loss rather than loss improvement

Compare with previous version

merged

mentioned in issue #125 (closed)