The FCA Consumer Duty: algorithmic bias and discrimination

The FCA Consumer Duty: algorithmic bias and discrimination

There is a growing regulatory interest in how the use of artificial intelligence, algorithmic decision making, and algorithmic pricing influences people’s financial lives. One area of focus is the potential for unintended discrimination due to algorithmic bias.

How the FCA Consumer Duty protects against discrimination

Discrimination based on protected characteristics in the provision and pricing of products and services is prohibited in the UK, EU and US through general legislation and sector specific regulations. Increasing prevalence of algorithmic decision making, however, presents a new type of challenge in this area as the models used are often opaque as to the factors that led to an approval or a pricing decision for any individual customer. Legislators and regulators are responding, for example:

• The EU is moving towards primary legislation through The AI Act¹, which could become a global standard similar the impact that EU’s General Data Protection Regulation (GDPR) had.
• The Federal Trade Commission (FTC)² in the US has published guidance on use of algorithmic decision making, in part re-emphasising the long-standing legislation and case law directing automated decision making, including in financial services.
• The UK government published an AI Regulation Policy Paper³ in July 2022, recognising the risk of algorithmic bias alongside the myriad of expected benefits.

The FCA Consumer Duty will play a key role in governing the use of algorithmic decision making in financial services in the UK, specifically in preventing or alleviating the effects of potential algorithmic bias. As we have discussed elsewhere, the cross-cutting requirements of the Consumer Duty “set out a clear expectation that firms address biases or practices that hinder consumers achieving good outcomes⁴. FCA guidance on the Price and Value Outcomes related to Consumer Duty is clear that:

Firms should be particularly careful where groups that share protected characteristics (as defined in the Equality Act 2010) may be disadvantaged. Firms should satisfy themselves, and be able to evidence to us, that any differential outcomes represent fair value, and are compatible with their obligations under the Equality Act⁵.

There is a clear imperative for financial services providers to ensure that the algorithms they use to support their credit and pricing decisions do not, inadvertently, lead to discriminatory outcomes.

Algorithmic bias persists in product approval and pricing

Some types of algorithmic bias can be obvious, at least after the fact. For example, a voice recognition algorithm trained mainly on people speaking with accents prevalent in white affluent areas will struggle to deliver the same quality of service to all potential users. Same will be true for many other applications using biometric data. Bias may also be due to design, such as in the case of UK Home Office visa streaming algorithm designed to discriminate on basis of nationality (which also included a feedback loop where the biased algorithm was generating further biased data for the algorithm to use)⁶. The potential for bias and its origins are not necessarily so obvious, however.

Algorithms increasingly drive credit acceptance and pricing decisions for financial services products. Due to the nature of modern algorithms (particularly machine learning models), their potential bias may become evident only after a detailed investigation of the outcomes they deliver.

A growing number of studies is laying bare the depth and breadth of the challenge with potentially discriminatory outcomes, in part due to algorithmic bias. For example:

• Research by the Citizens Advice on discrimination in the insurance market in the UK found that “customers living in areas with a high proportion of Black and South Asian people in the population, customers were quoted at least £280 more for car insurance, compared to areas where the population is mostly White.The research further found that in some areas the difference in price was more than 100%, which could not be explained by other factors such as differential crime rates. The authors estimate a total ethnicity penalty of £213 million per year in the car insurance market.⁷
• A study, Bartlett et al (2022), of consumer lending in the US found that lenders charge otherwise-equivalent Latinx/African-American borrowers 7.9 (3.6) basis points higher rates for purchase (refinance) mortgages, costing $765 million yearly.⁸ Significantly, this study was able to control for prevalent omitted variable problems to isolate the effect of apparent discrimination on base of ethnicity.

Algorithms as part of the solution

To be clear, the issue is not limited to algorithmic decision making, or specifically about FinTech. There is a long literature on the differences in the credit approval probabilities and prices paid between minority and non-minority borrowers.

In fact, the increasing use of algorithms is an opportunity to address effects of (any) inter-personal discrimination in approval and pricing decisions. For example, Bartlett et al (2022) finds that although FinTechs fail to eliminate impermissible discrimination, algorithmic lenders reduce rate disparities by more than third and show no discrimination in rejection rates in their sample.

Algorithms are easier to fix than other drivers of discrimination

The good news is that algorithmic bias is easier to identify and address than most other drivers of discrimination. The starting point is to understand how the biases may arise and how to detect them, which begins with understanding that algorithms do precisely what we design them to do (for now at least!), using data that we provide them with.

The main challenge with design question is that modern modelling approaches that are the best at predicting, for example, individuals likelihood of defaulting on their mortgage payments, are typically the worst at explaining the prediction was produced. The flexibility that the machine learning model is afforded to produce the best possible prediction, typically, comes at cost of opaqueness covering the decision-making process. It is often not possible to see into the black box to directly read out what difference any one factor made to the prediction. It is, however, possible to examine the recommendations or decisions made by the algorithm for evidence of discrimination, as well as to assess the design principles and data used in developing, training and testing the model.

There are many challenges with data and how it is used, including:

• Unrepresentative, incomplete and/or biased data: Any algorithm can only learn from historical data that is available to train and evaluate it. Without direct intervention to contrary, the models will learn what types of individuals have been deemed credit worthy, thereby embedding historical inequalities into future decision making. This can arise without an overt bias in e.g. the historical credit approvals – if for whatever reason individuals with a particular protected characteristic were less likely to apply for and therefore repay particular types of credit products, they may present to the model as higher risk. Keeping algorithms unaware of protected characteristics is not sufficient to prevent bias: even when the models are kept blind to protected characteristics, they may still discriminate through use of proxy data that may correlate with a protected characteristic.
• Use of proxy data: It is legitimate to differentiate between individuals based on factors that have a causal effect on e.g. credit worthiness – such as observed behaviours, or personal characteristics such as income or credit score from a rating agency⁹. Potential for discrimination through algorithmic bias arises with the use of proxy data that correlates with both credit worthiness and a protected characteristic, where the correlation with a protected characteristic remains even after controlling for fundamental drivers of credit worthiness. One example of this is use of postcode data, as found in the research by Citizens Advice. Another example would be to use historical healthcare expenditures as a proxy for healthcare needs in algorithms used to manage population health.¹⁰
• Classification rules: Developers may encode their own unconscious biases into the way they manipulate and categorise data used in the development and evaluation of the models.

Although the above list of risk areas may seem long and daunting, it is also inherently empirical and quantitative. Equipped with an understanding of how unintended bias may arise, it is possible to assess both the algorithms themselves and the outcomes they deliver to ascertain whether they are at risk or in fact do discriminate in unintended ways.

The available strategies include:

• Assessment of the model design, development and training process to identify potential risk factors, such as a lack of precise definition of fairness used in testing the model, and/or lack of in-depth knowledge of methods deployed by the teams responsible for the modelling (a particular risk with the use of commercially available third-party software facilitating implementation of machine learning models by non-specialists).
• Assessment of the datasets used in model training and testing process to identify potential risk factors, such as use of proxy data and/or potential issues in classification rules.
• Statistical analysis of past recommendations or decisions by the algorithm, to assess whether the outcomes in fact are biased against individuals with protected characteristics, when controlling for potential effects of observable drivers of e.g. credit risk.
• Controlled experiments of the algorithmic decision-making process, for example by inputting series of hypothetical customer applications where the applicants are carefully designed to vary only in status of a protected characteristics. This may be particularly effective for fintech companies and other settings where decision making is fully automated.

In short, the solution to potential algorithmic bias is to be careful and deliberate about the two aspects potentially leading to it – the design of the algorithm, including a conscious definition of fairness (lack of bias) in its development, and the data used by it.