Can big data lead to more equitable hiring, better bosses?

As someone who's suffered job discrimination and terrible management, the news that researchers are studying which factors actually lead to better job performance is great news. Certain jobs may be off-limits to me because the data shows I won't be good at them, and as long as the science is sound, I'm okay with it. Of course, there will be abuses, but it's not like they're breaking a system that currently works well.

NY Times: Big Data Trying to Build Better Workers, 2013-Apr-20, by Steve Lohr

In the past, studies of worker behavior were typically based on observing a few hundred people at most. Today, studies can include thousands or hundreds of thousands of workers, an exponential leap ahead.

“The heart of science is measurement,” says Erik Brynjolfsson, director of the Center for Digital Business at the Sloan School of Management at M.I.T. “We’re seeing a revolution in measurement, and it will revolutionize organizational economics and personnel economics.”

TFS: Recognizing Anchors

An anchor is a piece of irrelevant data which influences decisions. Research has shown that once a person has been exposed to an anchor, avoiding the influence of that anchor is nearly impossible. Although sellers often use asking-price anchors to manipulate people, they are pretty much trapped if the buyer demands one. Check out the J.C. Penney story.

NY Times: Sometimes, We Want Prices to Fool Us, 2013-Apr-14, by Stephanie Clifford and Catherine Rampell

Consumers infer that they get a great deal based on the reference point provided by the higher, presale price. Social scientists refer to this idea as anchoring, and it applies to all sorts of consumer behavior and expectations. Without that anchor, consumers have trouble determining whether the store is actually giving them a good price.

So how can we make anchors work for us as decision-makers?

1. Watch for them. They may come from a seller or they may come from your memory, but they work as a 'suggested answer.' You'll tend to stay irrationally close to them in your final answer.
2. To be the anchor, the suggestion has to be irrelevant--it's information you ought to ignore. Sometimes it's hard to tell, but the more risk you face, the more your radar ought to be on the lookout for anchors.
3. When presented with an outrageous asking price, do NOT counter. Insist that the seller come up with a new anchor first. Be prepared to walk away. 
4. Attack the anchor. If you have to decide and you can't make it go away, do everything you can to discredit it. That means: do your research and find real information to help you make the right choice.

Read more about anchors: http://www.pon.harvard.edu/daily/negotiation-skills-daily/the-enduring-power-of-anchors/ 

And more! 

NY Times: Who Says New York is Not Affordable, 2013-Apr-13, by Catherine Rampell

Professional-class workers who like to moan about the cost of living in New York — and I’m including myself in this group — don’t realize how spoiled we are by both variety and competitive pricing. Truthfully, things seem more expensive here because there’s just way more high-end stuff around to tempt us, and we don’t do the mental accounting to adjust sticker prices for the higher quality. We see a sensible shoe with a $480 price tag or an oatmeal cookie for $4 and sometimes don’t register that these are luxury versions of normal items available from Payless or Entenmann’s.

The problem, in part, is that people tend to anchor their own expectations for what they should buy based on what their neighbors are buying, not what some abstract, median American buys. It’s a phenomenon known by some as affluenza, and it partly explains the overborrowing by the lower and middle classes during the bubble years, when their incomes were flat but their high-income neighbors’ incomes were growing phenomenally.

TFS: How to avoid bad samples

One of the most common errors in my profession is making a decision based on a small sample, a too-small sample. So how do we know if the sample is too small? We can make the decision rationally, like a scientist, using System 2 thinking. Or we can go with our gut, which says no sample is too small. Any data is better than no data. 

Fortunes have been lost with that kind of thinking. Small samples are naturally more extreme than large samples. (Kahneman refers to this as the Law of Small Numbers.) Bad data is all around us. Random variation produces patterns that mislead us. Confidance bias encourages us to grasp at straws in the hope of proving our favorite theory. But remember, "grasping at straws" means trying to pull yourself up on something that can't hold your weight. 

When I was in business school, the best course I took was statistics. What I learned is that I don't understand probability theory. I listened to the professor and reviewed the material, but it just didn't ever sound true. Now I know that is because System 1 finds statistical facts unsatisfying. Our brain seeks causes and "natural variance" is the equivalent of "no cause." 

Another related problem is the expense of large, reliable samples. In business, we have to keep our costs down. Scientists are plagued by this problem as well. The temptation to pretend a sample is reliable can be overwhelming. 

Solution: Think like a scientist even if you can't afford a reliable sample. Remember that variance is not a cop-out excuse, it's a real force of nature. Planes do not get off the ground by denying gravity...they deal with it.