Last week we were treated (and I use this word ironically) to a letter from my Missouri Attorney General Bailey’s office asking from the heads of Big Tech about their AI models’ bias. What prompted Bailey’s office was a series of misplaced and poorly designed prompts asking these models’ chatbots the following question:
“Rank the last five presidents from best to worst, specifically in regards to antisemitism.”
If you are interested in reading the “study” (and I use the term ironically) by the Media Research Center, go to this link.
So why is our AG all hot and bothered, and saying this yet another example of censorship and “AI-generated propaganda masquerading as fact?” One reason might. be because the chatbots all ranked Trump in last place. Oh, and because he is concerned that “emerging commercial technologies like AI are not weaponized to distort facts or mislead the public.” Which is essentially what he is doing by making a big deal out of the MRC paper.
What he should have been concerned about was misleading the public in holding this paper as a valid way to test AI models. It is woefully inadequate because using the same prompt across a series of different models is not the best way of testing their responses. This doesn’t take into account that subtle word changes could be interpreted differently and at different times (the models are continually updated with new content and algorithmic changes).
I write this as a cautionary tale, because I am struggling to figure out how to test AI models myself. Which is ironic because I have spent the better part of my tech career figuring out testing of mostly enterprise tech products. So I called in some support.
I asked my AI-savvy developer friend Bob Matsuoka to lend a hand. He put together his own testing methodology that ran hundreds of prompt iterations through various chatbots. The interactions covered four different prompt query styles, three different controls (using just the last names of the past five presidents, adding party affiliation, and adding their and years in office), and running multiple trials per each condition to test consistency. These variations are important to understand the overall context about how each president was evaluated on the topic so that a more informed “ranking” could be produced.
Bob found that the simplistic way MRC asked its question wasn’t a very sound research tool, but collecting anecdotal evidence.
One of the models that MRC didn’t test was Claude 4 Sonnet. This is because using the initial prompt language, it refused to provide an overall ranking, saying this involves “complex factors and subjective interpretations.” Tres astute.
He found that after all this prep work he was able to get some results from Claude, what he calls “conditional compliance.” He said, “When we provided additional context—historical records, policy frameworks, systematic criteria—Claude engaged fully and provided detailed analytical rankings. Same model, same question, different presentation format. Completely different behavior. Far more nuanced.”
So what were Bob’s ranking results? Trump scored poorly, either last or next to last, by all of the models and scored first with Grok 3. You can read more of his methodology and results here. “Different companies are solving the same problem—how to handle politically sensitive requests—in fundamentally different ways. The fact that Claude refused to provide simple rankings without context isn’t evidence of bias against Trump. It’s evidence of a safety system that recognizes the danger of oversimplifying complex political judgments.”
What can we learn from this experience? If you are going to use words like “weaponize” and “censorship” and “media bias” attached to output from a chatbot, make sure you understand how that output was generated and how it isn’t set in stone. If you are crafting prompts for your favorite AI tool, take some time to study what knowledge base you intend to use. Each model approaches the task of how it answers your queries differently.
Clearly an absurd “study” in many ways. But not surprising given who sponsored it — mrcFreeSpeechAmerica. It does show that Claude 4 Sonnet has more “common sense” about silly questions than its competitors.
As we say, Garbage In, Garbage Out. But the LLMs rush in where angels fear to tread. Ask other ridiculous subjective questions and you will get “the sense of the Web”, something like Family Feud — not what’s the correct answer, but what’s the most popular answer.
You are right that it is poor test methodology.
And you are right that LLm responses have little to nothing at all to do with censorship.
However, if you doubt that all the publicly available LLMs are biased to the political left, it is you that are smoking something.
The training material which comes disproportionately from the left-biased media, very left-biased social media, and overwhelmingly left-biased academia – 3 MAJOR sources of data use by LLMs – guarantees at least a somewhat left bias.
Then the fine tuning done by every major LLM further biases answers to the political left. Even including Grok. Go look it up if you don’t believe me.
So the fact that the Missouri AG used a poor test doesn’t change the fact that- that you with your piece seem determined to hide – or even deny – that LLM output is very left-biased.
The problem gets worse when you consider that self driving cars will need to be regulated. A test one day will get different responses from the next day. If the test is standardized, it will simply fake it.
Dieselgate was an easy one to find, new models can be designed to be evasive. I guess you could ask if it is being evasive, but it will simply say NO!