Selection Bias in Software Sales
January 4th, 2010The other day I heard someone in a particular niche of the software industry say, “whenever we run into our competitors we always win.” I about choked on my food I was eating at the time. I like the company and they make a good product, but what made this claim particularly astounding was that I’ve heard it from two of their competitors as well.
I’ve heard a similar thing from someone in sales in our company as well. While I believe Mule ESB, Tcat, etc are great products, I surely don’t believe that we just have a marketing problem. If only people would get in front of our product then we’d always make a sale! Yeah, right.
Back to the first example though, how in the world can this be true for all three companies? Well, it comes from what is called a selection bias. Surely you’ve heard of it? Selection bias is when you have a bias in your conclusion based on who you are sampling from. For instance, let’s say you were trying to determine President Obama’s approval rating. If you said, hmmm, I’ll just take samples from urban centers – there are a lot of people around and it’s really convenient – that would be a selection bias. You’d be missing out on large swaths of republican, red state people who are much less likely to approve of Obama.
Similarly I believe that the 3 companies are coming to a wrong conclusion based on who their sampling from. This is almost too obvious to state, but when a lead comes in to a company they have already gone through many different gates:
- Hearing about your product
- Learning what your product is
- Evaluating your product
- Contacting sales
- Buying your product
I’m guessing that these 3 companies actually are only basing their data on those who made it to the last two points. Folks may have given up though on any one of the first three. They may not be hearing about your product. Your product could be described wrong for their use case. Your product may not work out of the box correctly. In any of these cases, you’ll never ever hear about it and you’ll be selling yourself short in the market.
Further reading
There’s actually a rather interesting field of research on this as it relates to cosmology. Nick Bostrom in particular has written a great book on it called Anthropic Bias (first 5 chapters are free). He asks questions about how we can reason with our limited perspective – we can’t see the grand scheme of things in the universe afterall, we can only see from our planet earth. So how can we make judgments about the probability of life? Or the probability of our species surviving after we know how to produce nuclear bombs? I find it a rather interesting topic. He also has a bunch of other great articles like Global Catastrophic Risks and Where Are They? Why I hope that the search for extraterrestrial life finds nothing.