To achieve the conclusion, the research team looked at thousands of A/B tests to help refine how it creates services across teams. Specifically, the company will use the study to inform how it addresses inequality for users. The coauthors offer an example of an app that is created that may run more slowly on an older device. This lack of performance on older hardware could impact members in demographics that wouldn’t show up in A/B testing. That’s because typical testing looks at the average user and not those with more unique circumstances. To address the situation, LinkedIn says it will start tracking inequality impacts across businesses and members on its network. Furthermore, the company has developed a new multidisciplinary team to evaluate experiments. In its blog post, LinkedIn explains how it categorizes inequality for the purposes of the study:
“To establish inequality baselines, we use the Atkinson inequality index, which can be applied to any metric, and captures how unequally it is distributed (if everyone has the same amount of that metric, inequality is 0; if some people have a large amount and others nothing, inequality is high). It is routinely applied to income or wealth by economists. Here, we are applying it to metrics that capture economic opportunity for our members on LinkedIn. To measure the impact of our experiments on inequality baselines, we use inequality impact, which is used to measure the effect an experiment has on baseline inequality in our metrics. For example, if job applications are very unequally distributed, and an intervention makes them more equally distributed (e.g., by helping people who normally apply to few jobs apply to more of them), we say that there is an inequality reduction impact on job applications.”
Moving Forward
In the study, LinkedIn found that the neutrals of A/B testing as denoted by the average user may not be neutral for everyone. The company points out new members value a rich onboarding experience and it usually leads to more engagement. Now the company says it will work across sectors to achieve better practices for tech experimentation: “Combining measures of inequality and A/B testing provides us two distinct advantages,” wrote LinkedIn in a blog post. “First, instead of only measuring inequality impact, we can also trace it back to its causes: a specific set of features and product decisions … Second, unlike classical algorithmic fairness approaches, it helps us identify features that increase inequality impact without having to rely only on explicitly protected categories … We hope that increased understanding of the underlying causes of inequality can lead to similar approaches to ethical product design across several different industries.” How companies such as LinkedIn evaluate the quality of their offering might differ substantially from what consumers or business clients think. For some products, checking independent review sites might be the only way to get all the needed information before a purchase decision is made.
