It reminds me about Soviet central planning: «Economist Paul Craig Roberts recalled, “a famous Soviet cartoon depicted the manager of a nail factory being given the Order of Lenin for exceeding his tonnage. Two giant cranes were pictured holding up one giant nail.”
The grand "eBay strategy" for Google Ads was to advertise on "long tail" keywords. This in practice meant that eBay would place its adverts on Google searches for extremely specific keywords like "Charizard pokemon card". This ended up maximising metrics like Click-Through Rate and Conversion Rate, and eBay's advertising division was famous in the advertising industry for its success in doing this. It was literally known as the "eBay strategy" and was treated with some respect. It took one insistent individual to convince eBay to turn all their advertising off and see if it decreased profits; it increased profits*. The problem is that right below the advert (which eBay was paying a lot of money for) was a link to the same page which eBay didn't need to pay for. The same page was a legitimate Google search result, and so the Google Search advertising was losing eBay a bunch of money.
[edit]
* - Thanks to the commenter above for the correction.
Note: I've tried to be more careful in not implicating eBay's entire advertising division.
This is exactly what happens. On the very long tail there may be no one willing to bid against you so there is no reason to buy the top slot, but if someone is, then suddenly it makes sense.
I agree, and the article doesn’t really explain a difference, either. Two quotes:
> Campbell’s law states that the more important a metric is in social decision making, the more likely it is to be manipulated.
> Goodhart’s Law, states that “When a measure becomes a target, it ceases to be a good measure”.
These are basically paraphrases if you accept that “target” = “important in decision making”, and “likely to be manipulated” = “not a good measure”.
That’s fine, important discoveries are often made multiple times independently, but let’s not pretend there’s some important difference where there isn’t any.
Goodhart was an economist and Campbell was a social psychologist. The fact that they independently made similar observations in different fields is substantial. Though, maybe it's more efficient to remember the over-arching idea, it also makes sense to identify them independently.
I agree they are basically saying the same thing but there is a subtle difference. Goodhart is observing a fact about measures, Campbell is offering an explanation of Goodhart’s observation, that is, why the target ceases to be a good measure. Goodhart is making a very general observation which might be true for a number of reasons. One alternative to the Campbell version is that value of a measure ages so, by the time the measure has become a target, the world has moved on and the target isn’t any good.
This is discussed in the article, but my summary would be that Goodhart says "if it (a measure becoming a target) happens it's bad", and Campbell is saying "it will happen if it's important".
I infer an undercurrent of “politics is about manipulating people not representing them” from Campbell and “Wally is going to code himself a minivan” (Dilbert ref) from Goodhart. So Campbell’s might be Goodhart with a politics context but it has a slimier feel to me.
> I infer an undercurrent of “politics is about manipulating people not representing them”
There's a scene or two in The Wire about "Juking the stats" (1) - its about fooling the people in order to look good and meet campaign promises. Manipulating indeed. More like "Mayor Carcetti's going to juke himself a second term"
> “Wally is going to code himself a minivan” (Dilbert ref) from Goodhart
Not really, Goodhart was talking about UK monetary policy in 1975. It's political from the start. (2)
2) "Goodhart's law is an adage named after British economist Charles Goodhart, who advanced the idea in a 1975 article on monetary policy in the United Kingdom, Problems of Monetary Management: the U.K. Experience:
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
I’ve said on here before, an additional extremely negative incentive that comes from an overemphasis on measurable metrics is that a lot of (important, subtle, meaningful) metrics are very difficult to dumb down for non-technical audiences, or even technical audiences lacking domain knowledge.
As a result the metrics that tend to be tracked are not just the measurable ones, but the ones for which the importance of them and the meaning of their values are comprehended by sometimes a very low common denominator audience. Usually needing to be tied directly in some way to money. (INdirectly doesn’t cut it, that ends up being too confusing a lot of the time).
Thinking about it, in the instance of important but esoteric metrics, do you think that the most frequent case has to do with the [expected] ability to impact or exploit it meaningfully? I don't do a lot of work with numbers, but it seems to me that in terms of affectation, those dumb statistics are the low hanging fruit with low investment cost in terms of manipulation, thus yielding "better" results and isolating them to the use case while alienating the complex metrics. It's a lot easier to take a metric that isn't contextually affected than the obverse, and up/downregulate it.
I think my example would be the quasi-metric of the single-sample workup you'd get at a hospital. You get a single datapoint. Compare that to a longer term study where samples are taken at 4 hour intervals. The latter brings up a lot of subsequent questions and has a lot of interconnections with the world at large and the individual's biological characteristics. E.g. getting caught by a train 2/5 days, boosting cortisol and norepinephrine, or low receptor affinity for X, thus producing the illusion of high X levels. As you modify you also complicate with an increasingly large list of rules. Or you just write a script for statins and pat Tom on the ass, noting next visit that serum cholesterol is down.
I am in the process of preparing for Facebook PM interviews. All their product sense and product execution “tips” hyperfixate on goals, metrics, and tying to the North Star.
In one sense, it seems as if the interview process is to indoctrinate a cult from the onset. I’m willing to bet there are people who go through this process, get rejected, and think the Facebook framework is THE right framework.
As a non Facebook user, I’m conflicted between hating their products to the core… on the other, their TC is salivating.
Yeh honestly being an SWE at FB just killed my soul (in part) because the job became so fixated on quantitative measures and experimentation...what we cannot measure cannot be important, surely....! :/ I think the only PMs/ICs who survive at FB are those who are either numbed or not aware of data fallacies like Campbell's Law.
>>what we cannot measure cannot be important, surely
Which concisely explains why the algos have zero correlation with moral compass, societal value, common decency, or caring for the commons upon which FB earns its wealth.
None of that can be measured, so it is all cut down and falls at the feet of the engagement algorithms. The result is an org whose business model is poisoning the common well for profit.
> the interview process is to indoctrinate a cult from the onset
Yes, I would say this is true in general at all companies, though I prefer the phrase "company/team culture" to "cult" (the phrases aren't that different in what they describe, just in emotional tone).
Because it is not in itself a negative. every team has a culture, by definition of "culture".
If they can make this explicit in the interview, then you as the candidate get a good sense of what this culture is and if you want to be a part of the team.
If they can not make this explicit in the interview, then you as the candidate know that they don't have control over this variable, again giving you valuable information on if you want to be part of the team.
Some metrics are useful. For the most pert I hate metrics. But you must deliver to managers if said managers lost touch with operations. They think they know what's going on by checking their metrics.
> Campbell’s law states that the more important a metric is in social decision making, the more likely it is to be manipulated.
This also holds for the fiat monetary system; a metric by which all economic value is measured. It has been manipulated in completely unjust ways to serve the interests of capital holders (and corporations) and that corruption is hidden behind mind-numbing complexity.
Having worked both inside and outside of the finance sector. I sensed that as I got closer to the money printers in my career, people's logic seemed to break down and they seemed to understand less and less what's going on. You start to meet people with unbelievably huge blind-spots. It's weird. It's as if people were selected for these roles based on their tunnel vision and unwavering faith in authority.
I would have thought that this was called Goodheart's Law, and if so, a perfect current example of it is the obsession by numerous policymakers, health officials and many members of the public with justifying so many different authoritarian and in some cases paranoid measures for the sake of stopping the spread of COVID (an impossible task at this point). Regardless of it being non-lethal or only mildly harmful to the vaccinated, the fixation has in some cases gone towards a blind need to try to "stop the spread", and an obsession with case numbers without consideration for whether they cause actual harm and mortality in largely vaccinated populations.
At this point, my gripe with covid metrics is that we're past the point where they control long-term outcomes. Covid will be endemic, you will either get vaccinated, covid, or both, and masks and lockdowns are only tools for buying time to get more people vaccinated because once restrictions are lifted, it will spread again. The only metrics that matter are vaccination rate and hospitalization rate, and for vaccine-rich countries, the latter is just a question of now or later.
Of all the metrics we have mortality is the least likely to suffer from these problems: death is eminently measurable, and won’t be fudged by wishful thinking, expectations, or placebo effects. Outright manipulation is hard, because you can just count the fresh graves in the cemetery to get a good estimate, because it’s is such an important event, and also because there are laws and a bureaucracy for measuring it. And it gets awfully close to what you really want, living long & happy. While it doesn’t measure happiness, every study on happiness shows that it is relatively resilient to changes in circumstances. Therefore, avoiding death is less of a proxy and more of a real thing than anything Google Anaytics will ever tell you.
A bit sad today, so I wanted to open a random thread and wish you well. Remember to take a moment to appreciate the people around you.
One surprising thing is that a lot of people go through life with serious existential dread, and that you can't really empathize with what it's like until you experience it yourself. Sometimes I wonder what the point is –– we evolved that behavior for a reason, and I can't think of a single way it improves one's ability to produce grandchildren. (Apparently the best evolutionary traits are those that let your children have children, rather than you having children directly, which was sort of counterintuitive.) In my experience, the existential dread seems to do nothing but chip away at people and their relationships.
Life gets strange at 30+. If you're around 20, my advice would be to relax and enjoy yourself, and truly realize that almost nothing you do now will matter in the long run. The reason that pushing yourself seems like the path to success is because you don't hear about the 9 out of 10 cases of ruined lives as a result of unrealistic expectations. But it's very easy to get caught up in being driven, as if it's religion.
Just find something small that makes you happy. Even if it's a pointless number puzzle, it's still yours. And that's good enough.
I don't think it's on purpose. I think the problem is that in a modern society, you are so far removed from the habitats that our ancestors evolved in that a lot of people simply can't cope. We're designed to spend time grabbing food, building shelters, making and taking care of babies, those sorts of things, all in small social groups. We're designed for immediate needs and feedback.
Seems like Nietzsche already saw this coming with science dealing a death blow to the old myths that kept people going, and with the advent of mobile terminals and apps optimized to keep our attention, I think we've just gotten closer to that vision.
Thankfully not. Just in a pensive mood. But my programmer friend has been unable to use the computer for two solid months due to some kind of sickness -- possibly pneumonia. I'm worried sick about him, because he lives in some kind of war-torn country with no functional medical system. I didn't realize how much I'd miss him till he was gone for a couple months on the verge of dying.
But that kind of thing can happen at any time. You just don't really notice it till you get older, so you never slow down to appreciate the people around you while they're there. Or at least, I didn't, for a long time. Been trying to change that in recent years.
Anyway, I really don't care about seeming weird. The weirdness is life itself, being a descendent of a monkey on a spinning speck of dust in the cosmic soup, worried about an abstraction we call money just to climb a societal ladder that doesn't really matter much in the end.
Everyone has to walk their own path, but my goal was to get one of you to stop and think about how you want to spend the next few years of your life. I've seen so many people allow themselves to think that work = life, and most of them seem to regret it. So I try to consciously avoid that in my own. Maybe you will too.
… The singular focus on a metric imposed from the top-down obliterated any incentive to innovate or differentiate production according to what customers might actually want.» https://www.forbes.com/sites/emilychamleewright/2020/01/15/w...