CEO Message

There is no denying that AI is adept at making swift decisions. It can wade through thousands of applications, put a flag on a suspicious transaction, put forward product recommendations and even anticipate a customer’s next move.

Yet one has to ask the uncomfortable question: What if the AI has learned the wrong patterns?

An unfair outcome does not require an intentionally biased model. A system can be steered off course by bad training data, historical precedents, missing information or features that were not well chosen. This is where the value of AI bias auditing comes in.

AI Bias Auditing

What exactly is AI bias auditing, it it something like an IRS or accounting audit, or is it something more? In short, it is testing a machine learning model to see if it yields systematically different results for different groups. Is the AI ethical and fair?

Let’s use a company screening job applicants with AI as an example. On the surface the model looks fine; overall accuracy is high. But when developers break down the numbers they may find that one demographic is being turned away far more often than another. The aggregate accuracy conceals the issue, so a bias audit is needed to get a more granular view of the model’s behaviour.

Developers have open-source tools at their disposal to spot these kinds of problems before they turn into costly business matters. With frameworks like AI Fairness 360 (AIF360) and Fairlearn, identifying unfair outcomes is possible. The difficulty lies not in running the software but in interpreting what it says and making fairness a habit in the development process.

Why are such tools necessary? For one thing, manually inspecting every model is not feasible when an organization is retraining on a regular basis. Open-source options offer reusable ways to measure bias and test for mitigation.

Open-Source AI Bias Auditing Tools

IBM Research’s AIF360 is a case in point. It offers the metrics and algorithms to detect and mitigate algorithmic bias. Then there is Fairlearn, which lets you assess fairness, visualise disparities and compare how a model performs across groups. Do not look for a magic button that will tell you if something is fair or not. Context is everything. What works as a metric for a hiring tool is not necessarily right for fraud detection. It is up to the developer to decide which groups and metrics are relevant and what level of disparity warrants action.

IBM Research - AI Fairness 360 (AIF360)

The AIF360 framework can be applied to both datasets and models. An audit might start with the identification of protected attributes. From there, a developer can run the numbers to check for statistical parity or disparate impact. Say an AI is handling loan approvals and 70 per cent of one group is approved against 45 per cent of another. That calls for some investigation. It is not proof of discrimination on its face; the data may be legitimate or the model may be relying on poor historical features. AIF360 can also be put to work on mitigation, whether that is in the data prep phase or after predictions have been made.

Community Driven Fairlearn

Fairlearn was started in 2018 by Miro Dudik from Microsoft Research as a Python package to accompany the research paper, A Reductions Approach to Fair Classification. The project was greatly expanded in the second half of 2019 thanks to the involvement of many contributors from Azure ML and Microsoft Research. In 2021 Fairlearn adopted neutral governance and since then the project is completely community-driven.

With Fairlearn the emphasis is on how the model behaves from one group to the next. You can compare selection rates, false-positives and the like. A fraud detection model might boast 96 per cent accuracy, which is impressive enough until you note that its false-positive rate for a certain segment of customers is double that of others. The top-line figure doesn’t give the full picture. Fairlearn gets at that second layer, which is vital since aggregate numbers can mask issues with underrepresented populations.

Interpreting the results

It is easy to make the mistake of treating a fairness metric as you would a unit test – pass or fail. It is not always that straightforward. When an audit turns up a disparity, the model should not be written off as defective without looking at why. Perhaps the training data is thin on a particular group, or a neutral feature is acting as a proxy for something sensitive.

That is where the technical side must meet the business side. Data scientists will see the statistical variance, while product and compliance teams can speak to how the model is used and what the regulations say. The metric is only the beginning of the inquiry.

Making it part of CI/CD

The best course of action is to stop treating fairness as a once-a-year exercise and build it into the workflow. A typical ML pipeline could run from code change and data validation through to training and performance testing, with fairness checks included before deployment. Automated tests in the CI/CD pipeline can be set to predefined thresholds. If a retrained model shows better performance but the disparity has crept up too far, the system will flag it.

Whether that blocks deployment is a matter of context. A lending or hiring system of consequence will need human sign-off and hard limits. A low-stakes recommendation engine can be monitored with less rigour. Consistency is what counts; you cannot rely on an annual audit to catch what next week’s data will bring.

The Case for a Fairness Gate

Any workable fairness gate must be able to put some fundamental questions to rest:

Which demographic or protected groups are being evaluated?
Which fairness metrics apply to the model?
What level of disparity requires investigation?
Which results should stop deployment?
Who reviews exceptions?

The Fairness Gate rules need to address these concerns, and that need to be put in writing and version-controlled. Left to their own devices, teams have a way of redefining “acceptable” when a model gives them an inconvenient answer. It is also wise to track model versions and datasets in tandem with audit findings. Should a new model show up with a marked disparity, developers can then go back and compare it with an earlier iteration to see what has changed. In that way, the testing becomes far more valuable.

Where Bias Comes From

It is easy to think of bias as something that happens during model training, but it can infiltrate the AI lifecycle at several junctures.

Consider data collection. If the training set is not representative of the population the system is meant to serve, performance will suffer for some groups.

Or take feature engineering; a developer ought to check if variables such as location, education or employment history are acting as proxies for sensitive traits. Then there is the matter of the algorithm itself – different optimization strategies can yield different fairness outcomes.

And once in the wild, real-world data often bears little resemblance to what was used in testing. A model that looks fine on paper may behave quite differently after deployment, which is why auditing cannot end at production.

Putting Accuracy in Perspective

Some in the business side will tell you that you have to give up accuracy to gain fairness. While there are tradeoffs, it is not always so. With better data or a different model, one can sometimes have both. But even where a compromise is called for, the cost of turning a blind eye to unfairness should be factored in. A model that is marginally more accurate yet puts a certain group at a consistent disadvantage is a recipe for legal, financial and reputational trouble. The most accurate option is not invariably the soundest business choice.

Pitfalls in Auditing

Good tools are no use if they are misapplied:

Relying on a single metric is a mistake; none of them will account for every type of unfairness, so the choice of metric has to fit the application’s risks.
The same goes for looking only at the aggregate – overall figures can mask what is happening at the group level.
Nor should a statistical variance be taken as proof of intentional discrimination; it is merely a prompt for inquiry.
And do not make the error of thinking one audit is enough. Business conditions and models evolve, so the process must be repeatable.

Integrating Fairness into Engineering

Tools like AIF360 and Fairlearn are at their best when fairness is woven into the fabric of software development and not treated as a box to tick at the eleventh hour. There is no need for a sprawling governance apparatus. Start small. Have a team pick a key model, define the relevant groups and run an initial audit. When the process is proven out, automate it. Put the checks in the CI/CD pipeline and have human oversight step in when results cross a threshold. Gradually the organization can bring more high-risk applications under this umbrella.

Conclusion

In the end, responsible machine learning demands some form of bias auditing. The tools provide a means to measure disparities, but they are only part of the equation. Developers have to know the context and be prepared to dig into unexpected numbers. A good place to begin is to select an important model, document the assessment and build that check into the pipeline. After all, the fact that an AI’s decisions are encoded doesn’t mean they should be let off the hook.

For more on this important topic, you can read more on our article on How to Prevent Bias in AI Projects Before It Causes Real Harm

To have a deeper conversation about Bias Auditing and Ethical AI, and how to implement them in your digital development project, please CONTACT ScreamingBox.

Check out our Podcast on AI Cybercrime to see what AI with no ethics looks like.