“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
This is a variation of one common low in economics, known as “Goodhart’s Law.” A principle lies under it can be used to describe a fundamental low of market movement: When one man had found a pattern in market time series and started to use it, the patter disappears. It means that as soon as a trader start using the pattern, he becomes a part of the factors that affect the sequence. The result, the all previous dependences changes to a new one, that not necessary will follow the old pattern.
Here is an excellent visualization on how a “Goodhart’s Law” works to a market patterns breaks:
Every time when a trader finds a new pattern and start using it, the pattern disappears. It can be variable from one to another: some models exist longer than average. Mostly because they are hardly recognizable. It is where a large hedge funds makes their money. They try to find a new pattern using more complicated models than concurrent. It ends when a crowd sees a pattern and start using it for speculations. By a crowd weigh a pattern is be broken and all starts again. Over and over.
The similar effect I saw when my boss decided to use KPI(Key performance index) to evaluate the quality of work. It looks quite a safety until the primary measure of KPI correctly relevant to the real outcome of the job done. In my case, when KPY had started tracking the amount of “activity” inside corporative web portal and use it to evaluate the performance of the worker, lot of people start posting “flood” comments to increase those score. By the end of the story, the fraction of a useful information decreased dramatically. The recalculation of the KPY index only helped to solve the issue.
Back to the data science problem, It is essential to find and use data that are not influenceable by the end user. In case of employer KPI, it must be a hard manipulated value, or some really valuable measure, for example, amount of sales for sellers. Otherwise, it can be turned into targets and lead to opposite effects one want to create.
Time since that I have started look not only at a quality of data but assume how future events can affect already used information. It’s fear of financial sector in particular, where human behavior changes market models constantly.
If you are interesting more about “Goodhart’s low” you should check the following links: Wiki, DataSceptic podcast, and ribbonfarm.com
All best!
