The UK has stopped using an AI system to judge immigration applications after determining that it suffers from PIPO.

PIPO, a term I just coined, stands for "prejudice in, prejudice out." This failure is common in AI systems trained on the actual behavior of people in a system where there is systematic prejudice.

In an ironic way, though, converting the prejudiced habits of officials into a program made the existence of the prejudice easy to demonstrate. Will that pave the way to eliminating the prejudice, or will the ministry only drive it underground?