Uncovering biases that no one knew were there... really??

People in AI feel very righteous when they acknowledge social biases in machines. But are they?

I have used the following screenshot countless times in talks and in teaching. If I pitch it well on the day, I get quite a few giggles – because yes, it is so ridiculous. If I don’t, the point sometimes go unnoticed.

This screenshot was taken from the website of the MIT review in 2016. The article on that page features research by MIT researchers, who have purportedly shown that machine representations of meaning (‘vectors’ or ‘embeddings’) are sexist, with strong tendencies to associate women with certain conventional roles. The article’s teaser enthusiastically advertises the research with the line As neural networks tease apart the structure of language, they are finding a hidden gender bias that nobody knew was there.

Now, as much respect as I have for my colleagues, I’m afraid to say they weren’t actually the first to make this important discovery. They’d been scooped. And not by another research group a few months earlier, but by thousands of people for thousands of years.

Because believe it or not, anybody who’s ever been discriminated against, even slightly – and that’s quite a lot of people over the years – knows that they’re being talked about in very specific ways, in ways that are not particularly nice. But of course when those people talk about their experiences, if they do, they don’t usually say ‘Gosh, there’s a lot of data against me’, or ‘That use of the woman vector really upset me today’. So clever people who use words like ‘data’ and ‘vector’ or ‘embedding’ then feel entitled to say they’ve discovered something.

Just let me spell it out, in case it’s not clear. AI representations of meaning are built from naturally-occurring language data. They just learn what’s in the data. So if the data is sexist, racist, ableist, or if the data thinks kittens are really cute, then the representations will similarly be prejudiced and kitten-loving. Anyone who’s taken a 101 course in distributional semantics should know that. It doesn’t mean it shouldn’t be studied: it can be useful to look at vector representations to verify, understand, and visualise prejudicial trends in specific types of discourse (see e.g. one of our attempts here). But we are never uncovering anything. Real people have said it before. The humanities have said it before. My 6-year old neighbour knows that.

Now, this kind of ‘discovery’ keeps popping up everywhere and especially loved by social media and pop science. The more recent example is possibly Google’s sentiment analyser revealing its homophobic and anti-semitic tendencies:

I am absolutely sure that every time such an article gets posted somewhere, people who are actual subjects of discrimination go to their special head-banging wall and pop a few additional cracks in the already crumbling plaster. Meanwhile, companies and AI researchers can feel really good about themselves for acknowledging the deficiencies of their technology and saying they’re ‘working on fixing it’.

So okay, there is perhaps a slight misuse of the concept of ‘news’ there, and we knew it all along, and we shouldn’t be surprised, but still cool that people are working on those biased vectors. We don’t want biased AIs, right? Right.

The trouble is, the cure follows the same path as the discovery. Some people somewhere, who somehow hadn’t noticed before that the world was a jungle and have now realised by looking at a string of numbers on a screen, will think that if the vector is biased, the vector should be fixed. What does that mean exactly? Well, as we’ve seen a vector is a representation of the data. The obvious way to change the vector is to change the data… but changing the data, which is data that real people have produced out there, would basically mean changing the world. And for sure, that sounds rather complicated. So what to do instead? Forget about the data and forget about the world (again). We’ll change the vector!

What does it mean to change the vector? Let’s take a metaphor. Let’s suppose that we’ve drawn a map of the night sky, with the position of all the stars and planets we could observe. Let’s call each star or planet a vector on the map. Now, we don’t like the position of that particular star. Gosh, it looks like it might die any minute, and it’s not so terribly far from Earth. So what could we do? Well, we’ll move the star on the map. We’ll move it really far away from our planet so that the drawing looks right, and then we can relax. You relaxed? I’m relaxed… This is what changing a vector means.

Before you think it is completely absurd, let me qualify what I’ve just said. It actually makes some sense to try and fix AI meaning representations. Why? Because they are latent in everything we do with technology. And if they’re biased because society is biased, they will just reinforce the existing prejudices. We know, for instance, that search engines have been caught serving terribly biased representations of minorities, both in text and image results. So actually, changing the vectors in the technologies we use most might indeed have an effect in the world. But wait, how should we change them? How do we know where to put that dying star?

Here is the way related in the MIT article above. You go and find some humans and you ask them questions about your vectors, trying to elicit bias.