The great Google Gemini deceit

On a week like this, it’s hard not to feel like you’re living in dueling realities.

On one side, you’ve got the futuristic vision Google is telling you about at its annual developer event, Google I/O.

The company waxed on for nearly two hours about how its Gemini generative AI assistant will help provide even more complex answers from the web, make purchases and complete bookings on your behalf, and generally just do all the Gemini stuff it does now faster and better. (And it isn’t alone: Just a day earlier, Microsoft told us at its Build event how Copilot will soon act as an “enterprise brain” and “suggest ideas” as you type — even, conceivably, offering to create entire legal agreements on your behalf.)

If you believe what these companies are saying (along with the same sorts of surreal-seeming realities being laid out by OpenAI and — well, practically every other tech player out there these days), we’re living in an era where artificial intelligence is always on the brink of a life-changing breakthrough.

What’s especially wild is that none of this is even that big of a leap from what these same sorts of systems have already been promising and the way they’ve been framed for months — for casual individual use, sure, and also for serious company business.

And that, my friend, is an ever-increasing liability for anyone who still cares about getting things right.

[Get level-headed knowledge in your inbox with my free Android Intelligence newsletter — three things to know and try every Friday, straight from me to you.]

Behind the generative AI curtains

Let’s back up for a second and talk about what all of these generative AI tools actually are — and aren’t.

The real issue here isn’t that this type of technology isn’t in any way valid or useful — far from it. It’s just that it isn’t designed to do or currently even capable of doing what nearly every tech company out there is breathlessly telling us it can handle in our personal and professional lives.

At their core, Gemini, ChatGPT, and other such systems are powered by a type of technology known as a large language model — or LLM, for short. In the simplest possible terms, an LLM looks at massive amounts of real-world language data and then uses that to learn patterns and predict the most likely next word over and over, in response to any prompt it’s given.

In other words, these engines don’t truly understand context or “think” about the answers they’re giving you in any human-like sense. They merely predict words, based on patterns observed in sprawling sets of human-created data, and then string those words together to form sentences and, eventually, entire paragraphs and documents whenever they’re summoned and given a task.

Somehow, that’s translated into tech companies plastering them into every possible place and presenting them as the end-all answers for every possible purpose — everything from replacements for search to replacements for writing in Gmail, Google Docs, and other such places. (The situation is even more extreme in other non-Google arenas, too — like with Microsoft’s legal-document-creating disaster-waiting-to-happen.)

And we don’t need to rely on theoretical examples to see just how dangerous of a situation this can create.

Artificial intelligence, genuine jeopardy

The reality of these systems’ limitations has been showing itself time and time again for quite a while now, in scenarios that are all too real and should serve as a serious wake-up call for any company — or even individual — buying into the hype.

Plain and simple, these things don’t know what they’re saying. They string together characters that sometimes make sense but are also extremely likely to include errors and often even flat-out fabricated nonsense. When they do get something right, it’s mostly just by chance. And you, as the user, have no way of immediately detecting the difference.

For instance, across the broader genAI landscape — with the exact same foundational limitations affecting Gemini and all these other similar systems:

Just last week, Anthropic — the company behind the business-popular genAI chatbot Claude — had to apologize in court after learning its system completely made up a legal citation the company’s lawyer used as part of an ongoing copyright case. (I’ll take irony for $500, Alex!)

Days earlier, a California judge discovered “numerous false, inaccurate, and misleading legal citations and quotations” in a submitted brief apparently created with the aid of AI.

Last month, a company relying on an AI “support agent” was forced to apologize when it learned said agent was making up nonexistent policies while interacting with customers.

In an experiment, Carnegie Mellon University created a simulation in which AI agents were tasked with handling low-level chores at a realistically structured software company — performing the same sorts of feats at which these systems are promised to be proficient. They failed miserably.

In the oft-cited area of genAI coding expertise, researchers are finding case after case where the systems invent package names that don’t exist and lead to all sorts of time- and money-wasting errors — not to mention troubling security vulnerabilities.

The Columbia Journalism Review tested eight different genAI search engines and found they got all sorts of information wildly wrong — offering incorrect and fabricated info and even nonexistent citations — and, worse yet, they served up those inaccuracies with an astonishing amount of confidence.

Lest you think this is an artifact of long-dated early versions of these engines, the instances above are all from the past few months. (They’re also just a surface-level sampling of the many, many examples of generative AI failure that pop up practically every day at this point.) Worse, as reported in The New York Times earlier this month, the act of “AI hallucination” — a fancy euphemism for “their tendency to serve up lies and inaccuracies” — only seems to be getting worse as the systems get more powerful.

Yet somehow, hardly anyone seems to be taking notice — or letting that reality get in the way of the much more enticing vision the tech industry is desperate to sell us. Just this week, a new report revealed that a whopping half of tech execs expect these very same sorts of error-prone AI agents to function autonomously in their companies within the next two years — which, translated out of corporate geek-speak, essentially means they’ll be replacing human workers and operating with little to no supervision.

Crikey. How long will it take for everyone to wake up?

Time for a generative AI reset

Google and all the other companies behind these tools — along with the corporate number-crunchers, clout-chasing LinkedIn bros, and mainstream media outlets blindly buying into the hype — like to pretend that all the stuff we just went over somehow isn’t a serious and immediately disqualifying issue for these systems. But no matter what sorts of impressive demos and over-the-top marketing materials they sling at us, the reality here in the real world is that these large-language model chatbots simply aren’t reliable when it comes to providing accurate answers and information.

So, sure: 

They’re incredibly handy as “legal advisors” — until you realize how the technology actually works and how likely it is to (a) get information wildly wrong and (b) flat-out make up facts along the way.

They’re fantastic as “search engines” — until it dawns on you that 20% of what they tell you is likely to be inaccurate.

And they’re wonderfully useful as “coding assistants” and “customer service agents” — if you conveniently look past the constant instances of them screwing stuff up and costing you business.

What’s especially troubling is the justification that these generative AI helpers are at least getting better and growing less likely to get stuff wrong. Even if you set aside the aforementioned data challenging that notion, a system getting something wrong 5%, 10%, or even 20% of the time is arguably worse than one that gets stuff wrong half the time (or more).

Think about it: If something’s wrong constantly, people are at least likely to notice and realize that it’s useless as an info-providing tool. But when it’s wrong only once or twice out of every 10 or 20 uses, it’s especially dangerous — as users will be lulled into a false sense of security and less likely to be watching for those errors.

Now, again, all of this isn’t to say these genAI systems aren’t at all useful or worth using. They can be quite helpful — if you think of them in the right way.

The problem is mostly that that reality doesn’t align with the much broader vision that tech companies are trying to peddle. But if you think of these systems as narrowly limited starting points for certain types of specific tasks, they can actually save you time and make your life easier.

With that in mind: No, Gemini and other such generative AI services aren’t instant answer engines, nor are they digital lawyers or even coders. But they can be useful note-takers and info-organizers. They can be quite helpful as image analyzers and manipulators. They can work wonders when it comes to creating polished presentations without all the usual effort, too — or creating calendar events without the typical clunkiness.

They can even be valuable brainstorming partners, in a sense, or deep-dive research assistants. But, critically, it’s up to you to think about how you’re using ’em and to treat their offerings as simple starting points — ways to save you the early steps of seeking out sources and stumbling onto ideas as opposed to a single-step replacement for critical human thinking.

At the end of the day, one lucky instance doesn’t discount the very real and completely unpredictable risk of random fabrications and inaccuracies. And, clearly, it’s entirely on us to use these tools wisely and take ’em for what they are: word prediction engines that can be helpful in certain limited, specific scenarios — not the all-purpose magic answer machines some companies desperately want them to be.

Ready for more no-nonsense Googley insights? Join my free Android Intelligence newsletter to cut through the hype and get plain-English perspective on the news that matters.