The Machine That Judges You

Somewhere in a data center, a score exists for you. It is not the score of your credit report, though it may feed on it. It is a number that predicts, within a margin your lawyer could never subpoena, whether you will pay, stay, comply, or sue. Companies buy this number about you. You never see it. You cannot appeal it. You will only meet its consequences: the apartment you did not get, the interview you never knew you applied for, the insurance premium you cannot explain.

We call these systems artificial intelligence, as if intelligence were the problem. It is not. The problem is that we have built a jurisprudence without courts, a punishment machine without a legislature, and we handed it the keys to ordinary life. We never agreed to this. It just arrived, one score at a time, and by the time anyone thought to ask whether it was legitimate, it was already deciding.

The verdict with no trial

Western legal thought spent three centuries developing a simple, radical idea: that judgment must be answerable. The accused has a right to know the charge, to see the evidence, to confront the accuser, to offer a defense. These are not courtesies. They are the machinery that makes judgment moral. A decision you cannot examine is not a decision at all. It is an injury.

Algorithms reversed the burden entirely. The charge is a number. The evidence is a weight of features, many of them proxies for things the law would call discrimination if a human said them out loud: your neighborhood, your phone model, the hours you shop, the people you know. The accuser is a trade secret. The defense, if it exists, is a form with no guaranteed reader.

Procedural justice is not decoration

Philosophers of law have a term for what is missing: procedural justice. It is the idea that outcomes can be fair even when they are unfavorable, provided the process that produced them was fair. People accept a denied loan more readily when they understand why, when the reason is real, when there is a path to remedy. The process is not a courtesy. It is most of the fairness.

John Rawls asked us to design institutions behind a veil of ignorance, without knowing where we would land in them. Imagine designing a scoring system this way, knowing you might be the one scored, knowing you might be the one whose neighbor, accent, or browsing history tilts the number. Would you keep the black box? Would you keep the appeal process that consists of typing your grievance into a portal?

“The question is not whether machines can judge. The question is whether we can be judged by anything we cannot interrogate, and still call ourselves free.” From the technology desk

Speed as an excuse

The standard defense is speed and scale. Humans cannot review ten thousand applications a day; the machine can. This is true, and it is also a confession. If the only way to process everyone is to strip judgment down to a number, then the honest conclusion is that we have chosen throughput over justice, and we should say so plainly instead of hiding behind mathematics.

Notice, too, what the machine optimizes for. A loan officer might consider a borrower’s story, their recovery from a bad year, the particular shape of their hardship. The machine considers only what predicts repayment. But societies do not exist to maximize repayment. They exist, at their best, to give people second chances. A system that cannot see a second chance is not neutral. It is a particular moral philosophy, implemented at scale, without a vote.

The audit we owe ourselves

None of this requires abolishing algorithms. It requires treating them like institutions, because that is what they are. Institutions get constitutions. They get oversight. They get public audit trails, meaningful appeals, and the standing obligation to explain themselves when they hurt people.

Concretely, that means a right to the features that decided against us, not the trade-secret model. It means regulators who can compel disclosure and who actually use the power. It means discrimination testing that is mandatory, continuous, and published, not a charity some vendors perform. It means an appeal that a human genuinely reads, with the authority to change the outcome.

And it means, above all, that we stop calling these systems “tools” and start calling them what they are: governments. Automated, unelected, unappealable governments of daily life. We fought a long time to place limits on government. It would be a strange century to hand the whole thing back to a machine, just because it never asks for a vote.

❧