Exploring Algorithmic Recourse
The Explainable AI and Ethics space covers a wide range of topics — I always try to mention you can find philosophy, history of technology, mathematical methods, etc. Before attending an AI ethics event, I decided to do a deep dive into ‘Algorithmic Recourse’ — finding the change needed for a subject of the model to change their outcome (ideally smaller, feasible changes). It might not be a widespread term, but it shows up in industry, research papers, an ICML workshop, and the actively-developed CARLA benchmark.
My initial thoughts were:
- How do these systems note which variables can or can’t change for an individual? If I remove variables such as age and race from recourse analysis, how do I investigate fairness issues?
- What type of cost function represents which variables are easier to change than others?
- How does this all work in the era of GPT-3, Stable Diffusion, DALL-E?
the CARLA paper and benchmark
I started with the CARLA paper on OpenReview. As expected, it’s important to set immutable features and a cost/distance function. The CARLA library includes eleven methods for developing counterfactuals (LIME being a familiar one), settings for hyperparameters, and TensorFlow / PyTorch classifier support. The researchers were skeptical of causal methods because they are based on assumption you have the true causal graph and correlations between variables (classic example being income and savings — people with higher income will find it easier to have higher savings). The team finds no one method that excels in all metrics and datasets.
After the paper’s release, the GitHub repo has a few additional methods and a new dataset: Home Equity Line of Credit (HELOC).
A few surprises — algorithmic resource tries to limit the number of variables that change, not just the closest change. It’s described as a human-in-the-loop method (because the human changes their input? because a human notes which columns can change?). For a Census income dataset, most of the methods expect categorial variables to have only two values, so these lose some information getting binarized. OpenReview comments were critical of the categorical variable issue, and also evaluating methods by execution time (highly variable).
One of the test datasets is the COMPAS recidivism risk dataset. This shows up in a lot of fairness papers but furthers the idea that an algorithm could justly keep a person in prison? My go-to paper on this would be Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead which gets similar results to the COMPAS model with a small ruleset:
Recourse does not grant equity
I watched a video on algorithmic recourse from December 2021.
It mentions papers from 2019 and 2020 which cover fairness of recourse — i.e. the recourse is not helping if some groups are given burdensome changes.
‘Equalizing Recourse’ uses multiple credit datasets and COMPAS (called propublica in the paper). The main focus was modifying the classifier for gender equality, looking at SVM and some black-box models.
‘On the Fairness’ uses causal methods on the Census Adult and some credit datasets. It’s also using probabilistic methods instead of any black-box / neural net type methods. So I wouldn’t be likely to use this.
What about generative models?
The bread and butter of the data science / modeling / ML commercial work is tabular data, non-neural methods, and ‘boring’ data (trading stocks, maximizing reach of advertisement budget, credit and insurance, etc.) Recourse methods on these models is important. But the headlines are large language models, image generators, translators, etc. etc. so let’s go.
For language models, we have a term for changing the input in a small way to change the classifier: adversarial attacks. If your video was demonetized or your Airbnb complaint got lower priority, and you could get a suggestion to subvert the model (‘pandemic’ to ‘panini’), that would defeat the purpose of trust/spam models. I think this contributes to a larger discussion of ‘what is a generative model failure?’, beyond someone prodding a model to say something offensive. A more conceptual failure might be, ‘when generating children’s stories, people with turbans or tattoos rarely appear and only as villains’. You could talk about model weights and a storybook corpus but I’m not sure how to convert it into a change for the user (maybe specifically prompting the story model with info about people in your family and culture?).
For image models like DALL-E, OpenAI returns multiple images for a query, and is experimenting with changing the prompts on some of those images for more diversity. This method got reverse-engineered because if a person-generating prompt ends with ‘a text sign that says’ then signs appear saying ‘female’ or ‘black’.
You were expecting to generate a variety images for ‘professor’ or something, and originally the images were from a subset of possible options, and OpenAI modified the input to get better results. But this, future generations of this prompt-editing, and generating on multiple fine-tuned models, are all probably best categorized as a form of prompt engineering.
I don’t think that ‘algorithmic recourse’ will get adopted in these large language model communities because recourse is so closely tied to the field of counterfactual examples, and LLMs are already so difficult to interpret.
Additional Reading
CARLA cites Wachter’s 2017 paper for popularizing counterfactual explanations for black box models:
Beyond individual changes, the video mentioned subpopulation-based recourse using a method called CATE, where I found some dense math but not a good explainer.
Some recent papers from the Twitter-verse:
