While running an Excel boot camp, we were discussing ranking sales employees based on sales targets. One of our brilliant students brought up a supposedly well respected Canadian businessman who was known for doing just that, and then firing his bottom performing staff, every month.
I immediately thought of three things: Israeli fighter pilots, photon pairs at the Large Hadron Collider (LHC) in Switzerland and the search for loaded dice.
I promise all of these are very relevant and I hope that by then end of this I will have convinced you to perhaps respect that canadian businessman a little less.
The problem is this: we often misguidedly use events of random probability to confirm theories and justify practices.
Probability in Particle Physics
In 2010 CERN fired up the LHC and started smashing together hydrogen atoms, close to the speed of light, with the goal of finding new undiscovered particles. This creates a temperature similar to just after the big bang so we can see how high energy particles work. The high energy particles will then decay into smaller ones, which we can observe.
The problem facing the experiment was how to pull a signal out from the random noise that filled their data. Most hydrogen collisions are pretty boring and don’t create new particles, but these still show up in the data.
The Large Hadron Collider beam pipe 
The ATLAS detector 
The LHC is 27 Km around
The way particle physics works is based on conservation of energy. The important thing to know is that we know the new particle we are looking for should decay into 2 light particles (photons). So the detector creates a database which records the total energy every time that it sees 2 light particles at the same time.
There are of course light particles bouncing around all over the place, so we expect to see a smooth distribution of background events, but since the photons from a specific new particle will always have the exact same energy, we will see a peak in that distribution.
Simply put, we look for a little bump among the noise, then work out how likely it is that the noise made that bump. If it is likely that it’s noise, we ignore it, if not, we can assume it’s a particle.
The dotted curve shows the expected background.
The bottom section is what we see with the background removed.
The little bump shows an excess of what we expect if it was only background. This tells us that there is a new particle that doesn’t come from random background fluctuations.
In 2012 CERN measured “a signal with a significance of greater than 5 sigma”. This meant that there was a one in 1.74 milllion chance that the observed signal came from random statistical fluctuations. Given this is pretty unlikely, they concluded it wasn’t noise and was a new particle.
CERN announced the discovery of the Higgs boson or “God Particle”, much champagne was drunk and physicists the world over celebrated.
In 2015 the LHC wasn’t done searching. The Large Hadron Collider was still looking for a new particle. By collecting the particles that were spit out and adding up their energy, the team at CERN would be able to see a new particle as a peak in a graph.
The important thing to consider is the question “how likely is it that this signal came from random noise”.
CERN once again measured a signal from 2 particles of light, this time at an energy of 750 GeV (about 3 times the mass of a Uranium atom).
After measuring the statistical significance, it was found that the probability that this signal would be created by random statistical fluctuations was one in 15 787.
Theoretical physicists rejoiced and thousands of papers were published on how the newly discovered particle proved their theory, however, all was not well. This would later turn out to be a statistical fluctuation.
The problem comes from the difference between global and local significance.
There was a 1/15 787 chance of finding that signal where we found it, but if you look long enough for something rare, you will eventually find something rare. The people hailing the new particle had looked at the probability of finding this signal in the exact spot they found it, but they had not considered the probability of finding it “somewhere or anywhere”.
The chance of winning the euro lottery is one in ninety five million. Does this mean we can take every lottery winner and lock them up for cheating? It certainly seems more likely that they cheated than the 1/95000000 chance they won fairly.
Of course not!
There is a one in 95 million chance of YOU winning the lottery, this is not the same as the probability that SOMEONE wins the lottery.
The “local” probability of quadruplets is one in 570 000, but with 7 billion people on the planet, the “global” probability is quite high, just like the local probability of our new particle being noise was very low, but the global probability of finding any signal somewhere in the noise, was very high.
Israeli Fighter Pilots
During a training regime for pilots in the Israeli airforce, the trainers wanted to know the best method of improving pilot performance.
The trainers tried out two different training styles.
When a pilot underperformed, they were admonished for their screw ups, whereas, pilots who did particularly well, were given compliments for their success.
On the next training run, what the trainers found, is that the pilots who had been given praise were less likely to do as well, while the pilots who had been reprimanded improved.
They concluded that negative reinforcement works, while positive reinforcement, not only doesn’t work, but it has a negative affect.
Why might that be? This flies in the face of nearly every controlled psychological study, so what was going on? Is flying just different to every other kind of learning?
As a dungeons and dragons player, I thought maybe I could adopt their techniques.
I started with 100 dice.
After rolling them all 20 times and adding their scores, I took the 10 worst performing dice (naughty dice) and yelled at them, really just let them have it. I told them if they didn’t improve I would fire them and melt them down for spare plastic.
I then took the best performing dice (good dice), put them in a dice jacuzzi, offered them promotions and told them they were the best.
What do you know, it worked! The under-performing dice improved while the dice that did well initially, got worse. Apparently inanimate plastic dice share a fundamental psychology with fighter pilots!
Of course I’m being a little silly here but that’s the point.
If we look at a bell curve generated by random events, the edges of the bell curve are occupied by the best and worst performers, but random chance governs our lives more than we’d like to admit.
If we select the worst performers, then let them try again, chances are that they will approach the average, and thus, improve. The same is true for those that overachieve, they will approach the average and thus, they will seem to get worse.
This is called “Regression to the Mean”.
Would I have improved my scores if I fired my underperforming dice every 5 rolls? Of course not. The same is likely true for sales people (as it is for fighter pilots).
That’s not to say that we shouldn’t be concerned with underperformers, but we need to consider why we are looking at a certain sample.
If we already have concerns about a specific salesperson then a low sales score could indicate lacklustre performance, but if we select someone precisely because of a single low score then that must be the START of data taking. Otherwise you will likely be firing your best salespeople because, one month, random chance put them on the left hand side of the bell curve.
Firing your underperforming employees every month might make you look tough, but anyone with an understanding of statistics should realise that you are a very, very, silly business person who is likely blowing way too much money on HR and severance packages.