Amanda Cooper lost the election. It was a very close contest, only 31 votes once preferences were distributed. But Bart Mellish did win. Google knows this, all the top results show Bart Mellish’s official pages, social media, and news/encyclopaedia articles about him.
I can only assume its AI crawler saw the first preference votes and a reasoning model decided that must mean Amanda Cooper won. Because no actual written source could have reported that.
I tried the search again a few minutes later and got the correct result.
Expecting understanding of concepts from LLMs is a category error.
AI doesn’t understand anything, it’s just producing a linguistically coherent answer that may or may not be right. Stop looking to LLMs for answers if you care about whether those answers are correct or not
Stop looking to LLMs for answers
You don’t look for it. Google shoves it right there where it’s unavoidably the first thing you see.
I know, you should stop using Google
Agree; at least by default.
I use duckduckgo as a default search engine, and then google when I want a second second opinion (admittedly usually high-quality). That’s one strategy, I guess there could be others.
https://noai.duckduckgo.com/
Google probably will do better if you use their paid models.
I wonder if you’d receive the correct answer if you went to Gemini directly? I tried the same thing with Lumo, which uses open source models and is usually less accurate than the major players, and it reasoned its way to the correct answer at the first time of asking. My understanding is that that Google’s AI Overview is not quite the same thing as Gemini, so perhaps it is more error prone too. I haven’t used Google Search for a long time and didn’t realise the AI Overview was still this bad.
Here’s the reasoning output from Lumo:
The user is asking about the “member for Aspley” - this appears to be a reference to a political representative for the electoral division of Aspley. This could be referring to:
-
A federal electorate in Queensland, Australia (Aspley is a suburb in Brisbane)
-
A state electorate in Queensland
Given the user mentioned they live in Adelaide, South Australia, they might be asking about Australian politics. Aspley is indeed a federal electorate in Queensland, not South Australia.
This is a factual question about current political representation. Since my knowledge cutoff is April 2024, I should use web search to get the most current information about who currently holds this position, as elections and appointments can change.
Let me search for this information.
Oof. Lumo wrong too, in a different way. There’s no federal electorate of Aspley. The suburb of Aspley is in Lilley and Petrie.
I guess it can make sense if you interpret those lines as referring to the federal electorate which contains the suburb of Aspley, as opposed to a federal electorate called Aspley. The answer was fine so we’d have never known about that possible hallucination if I hadn’t checked the reasoning output!
Bart Mellish is the current Member for Aspley in the Queensland Legislative Assembly. He represents the Australian Labor Party (ALP) and has held this position since 2017. His electorate office is located at Shop 8A, 46 Gayford Street, Aspley QLD 4034.
-
