Today we have a guest post by Patrick Gallagher. Patrick is a postdoctoral fellow at the Waterford Institute of Technology.
Statistical profiling of the unemployed has gained traction in recent years as an efficient, cost-cutting means of categorising job seekers into two groups the frictionally unemployed and those at high risk of long-term unemployment. The regulatory institutions suggest that data gathered during the profiling process allows PES to tailor services to individual needs and assist jobseekers in surmounting employment barriers. The technology, however, remains opaque, shrouded in a black-box which only the data scientists seem well equipped to prise open.
Despite the powerful voices which lend legitimacy to statistical profiling, it has not yet reached incumbent status. Only one-third of OECD countries have adopted the technology, and with the notable exception of Australia, no nation has handed over decision making to the machines. Instead, they continue to combine the technology with a human fail-safe. The technology is not stable, rather it is referred to as messy, complex, and prone to teething problems. In this respect, several of the tools have been withdrawn due to concerns over accuracy and problems with the implementation and acceptance (on the part of PES workers) of the technology.
Recently the project carried out an extensive ethical review of statistical profiling across the OECD taking a deep dive into five individual case studies. The data we present today was gathered as a part of this review. The table above is adapted from work by the OECD which outlines the main characteristics of those case countries which employ statistical profiling of the unemployed. The accuracy rate of the technology is a central feature of this story and is displayed in the fourth column. Rates vary across countries from as low as 65 % to as high as 90%.
Our investigation into the accuracy of the technology revealed several vital points. Firstly, the approach to the reporting of accuracy within the literature is minimalist. Interpretations of this approach range from organisations being structurally opaque to downright misleading. There is a notable lack of methodological papers which if they existed would outline the testing of the technology or offer an in-depth analysis of how the accuracy rates were achieved or indeed what they mean for operational efficiency. Finally, this suggests that there is a ‘performance of accuracy’ which legitimises the discourse of efficiency and cost-cost cutting which conducted to obfuscate the true role of statistical profiling technology which is the maintenance of case worker to client ratios in the face of volatile market conditions.
The methodological data we did uncover suggests that accuracy may well be the canary in the coal mine of statistical profiling. As we shall see, what is reported as the overall accuracy rate is a measure called forecast accuracy (from data science), and this measure works in a counter-intuitive way which has a misleading effect, distorting the perception of how well the technology works. For example, if the OECD reports that the technology achieves a 70 % accuracy rate, then given that the technology aims to categorise people into two groups, one intuitively assumes that 70 % means that for every ten individuals profiled three are misclassified. However, based on a feasibility study carried out by the department of work and pensions in the UK at a forecast accuracy of 70 % their model misclassified four individuals for every individual it correctly classified meaning it had enormous deadweight cost. In plain english, that means for every individual that the tool correctly identifies as high risk, it misidentifies four individuals as high risk.
The operational implications of these findings are profound and call into question the ‘real function’ of statistical profiling of the unemployed? In this respect we argue that these tools allow European PES services to maintain steady client to workers ratios in the face of volatile market conditions. The OECD has set the best practice caseworker to client ration for PES operations at 1:150. In the last two decades we have seen these ratios come under severe pressure during periods of market uncertainty, firstly during the global financial crisis and more recently during the COVID-19 crisis. Market instability has meant that PES has cycled in and out of times of high and low pressure on client caseloads. Statistical profiling allows PES to maintain a steady ratio no matter what the market conditions the tools continue to feed a steady stream of individuals towards PES services. If our data is correct (and we believe it is) many of these individuals (in fact the majority 4:1) had no need to of PES services to regain access to the labour market. This means that the primary function of statistical profiling of the unemployed is to create stability within European employment services, to ensure a steady stream of unemployed continue to cycle through active labour market programs no matter what the prevailing market conditions. Given the importance of these findings in our next blog post we will take an I depth look at accuracy rates, how they are constructed, how they work in whole numbers and discuss the operational implications of the technology.