HomeTechnologyReview | I tried...

Review | I tried the new Google. Its answers are worse.


Have you heard about the new Google? They “supercharged” it with artificial intelligence. Somehow, that also made it dumber.

With the regular old Google, I can ask, “What’s Mark Zuckerberg’s net worth?” and a reasonable answer pops up: “169.8 billion USD.”

Now let’s ask the same question with the “experimental” new version of Google search. Its AI responds: Zuckerberg’s net worth is “$46.24 per hour, or $96,169 per year. This is equivalent to $8,014 per month, $1,849 per week, and $230.6 million per day.”

Um, none of those numbers add up.

Google acting dumb matters because its AI is headed to your searches sooner or later. The company has already been testing this new Google — dubbed Search Generative Experience, or SGE — with volunteers for nearly 11 months, and recently started showing AI answers in the main Google results even for people who have not opted in to the test.

The new Google can do some useful things. But as you’ll see, it sometimes also makes up facts, misinterprets questions, delivers out-of-date information and just generally blathers on. Even worse, researchers are finding the AI often elevates lower-quality sites as reliable sources of information.

Normally, I wouldn’t review a product that isn’t finished. But this test of Google’s future has been going on for nearly a year, and the choices being made now will influence how billions of people get information. At stake is also a core idea behind the current AI frenzy: that the tech can replace the need to research things ourselves by just giving us answers. If a company with the money and computing power of Google can’t make it work, who can?

SGE merges the search engine you know with the capabilities of a chatbot. On top of traditional results, SGE writes out direct answers to queries, interspersed with links to dig deeper.

SGE is a response to the reality that some people, including me, are starting to turn to AI like ChatGPT for more complex questions or when we don’t feel like reading a bunch of different sites. Onely, a search optimization firm, estimates that using SGE can make a user’s overall research journey 10 to 20 times shorter by assembling pros and cons, prices and other information into one place.

An all-knowing answer bot sounds useful given our shrinking attention spans. But Google has a lot to work out. We expect searches to be fast, yet Google’s AI answers take a painful second or two to generate. Google has to balance the already-fragile economy of the web, where its AI answers can steal traffic from publishers who do the expensive and hard work of actually researching things.

And most of all, the new Google has to deliver on the promise that it can consistently and correctly answer our questions. That’s where I focused my testing — and kept finding examples where the AI-supercharged Google did worse than its predecessor.

Putting Google’s AI answers to the test

Often when you’re Googling, what you really want is a short bit of information or a link. On a day-to-day basis, the new Google is often annoying because its AI is so darned chatty.

A goofy example: “What do Transformers eat?”

The AI answer told me that fictional robots don’t really need to eat or drink, though they need some kind of fuel. Meanwhile, old Google had the one-word answer I was looking for: Energon. (It’s a kind of magical fuel.) You got that answer from new Google only by scrolling down the page.

This doesn’t just happen with alien robots. When SE Ranking, a firm dedicated to search engine optimization, tested SGE with 100,000 keyword queries, it found the average answer it generated was 3,485 characters — or roughly a third as long as this column. One of Google’s challenges is figuring out when its AI is better off just keeping quiet; sometimes, SGE asks you to press a “generate” button before it will write out an answer.

Most of all, when we search, we expect correct information. Google claims SGE has a leg up on ChatGPT because its knowledge is up-to-date.

Yet I found the new Google still struggled with recent affairs. Three days after the most recent Academy Awards, I searched for “Oscars 2024.” It told me the Oscars were still to come and listed some nominees.

And nothing undermined my trust in Google’s AI answers more than watching it confidently make stuff up.

That includes facts about yours truly. I asked it about an award-winning series I wrote for The Washington Post, and it attributed it to some stranger — and then gave a link to some other website.

Then there was the time SGE all too happily made up information about something that doesn’t even exist. I asked about a San Francisco restaurant called Danny’s Dan Dan Noodles, and it told me it has “crazy wait times” and described its food.

The problem is that this is an imaginary shop I named after my favorite Chinese dish. Google’s AI had no problem inventing information about it.

So-called hallucinations about real and fake topics are a known problem with current AI. A disclaimer above SGE results says, “Generative AI is experimental,” but that doesn’t solve the problem. Google needs to figure out how to say “I don’t know” when it isn’t confident.

To give us answers to everything, Google’s AI has to decide which sources are reliable. I’m not very confident about its judgment.

Remember our bonkers result on Zuckerberg’s net worth? A professional researcher — and also regular old Google — might suggest checking the billionaires list from Forbes. Google’s AI answer relied on a very weird ZipRecruiter page for “Mark Zuckerberg Jobs,” a thing that does not exist.

In my tests, suspect sources were a pattern. At the suggestion of Onely, I asked the new Google which was more reliable: Apple iPhones or Samsung phones. As a longtime reviewer, I could tell you lots of good sources of information on this, including professional journalists and repair organizations like iFixit.

Instead, the AI cites random views of people pulled from social media. Beyond the limited usefulness of a single Reddit user’s experience, how does Google know that it wasn’t a fake review posted by the phone maker?

“Google SGE plays by a different set of rules compared to the traditional search engine we know today,” said Tomek Rudzki, Onely’s head of research and development.

SEO firms have been trying to do quantitative studies of SGE’s values, though they’re limited by Google’s requirements on test accounts. But they’ve found a similar pattern in the disconnect between the sitesthat the old and new Google link to. SEO software company Authoritas tested searches with a thousand shopping terms in late March, and found that 77 percent of the time, the domain of the No. 1 traditional search result showed up nowhere in the AI-written answer.

And in its study of 100,000 keyword searches, SE Ranking found that question-and-answer service Quora is the most-linked source by SGE; LinkedIn and Reddit were fifth and sixth. How often would those sources be acceptable on an eighth-grade term paper?

On searches about tech topics — including lots of “how to” questions — SE Ranking found the most-linked domain was simplilearn.com. I’d never heard of it before; the site describes itself as an “online boot camp.”

“This trend not only diminishes the quality of search results but also reduces traffic and revenue for many small businesses, including affiliate websites,” says SE Ranking’s head of SEO, Anastasia Kotsiubynska.

Google says SGE is an opt-in experiment. But Google already blew past its expected end last December, and it hasn’t offered any update on when it will come to search for everyone. It’s possible that Google doesn’t think SGE is accurate or fast or profitable enough and that it will end up changing it dramatically.

They are wise to go slow, even if it makes Google look as though it’s behind in the AI race. Rival search engine Bing from Microsoft made a similar AI overhaul in February 2023, but its AI is still best known for going off the rails.

In an interview, Elizabeth Reid, a Google vice president leading SGE, characterized it as a work in progress.

“We’re really focused on ensuring we get the experience really right. There are a lot of different factors on this — things like latency, accuracy, helpfulness,” Reid said. “What we’ve been finding as we’re iterating and learning is that it’s pretty nuanced.” In other words, there are times the AI is helpful and other times it’s not — and Google is still trying to figure out where to draw the line.

When I shared the examples in this column, Reid told me that SGE’s hallucination rates are “very low” and have decreased “meaningfully” since SGE’s May launch, though she declined to be specific.

“I don’t want to minimize it — it is a challenge with the technology” and something “we’re really working on,” Reid said. Putting links right next to the AI answers, she added, is important to enable people to check the facts for themselves.

Here’s a proposal: Because Google acknowledges correct facts are a problem, it ought to disclose its own data on accuracy before it brings SGE to a broader audience. With billions of searches daily, even 0.001 percent can add up to a lot of wrong information.

Another area of Google’s focus is “trying to help ensure that we get to the core of the question as quickly as possible, and then give additional elaboration,” Reid said.

As for citing low-quality sources, Google disputed the outside research on SGE, saying it is based on searches that are more limited than what Google sees in practice. But it declined to share data of its own.

Reid said SGE doesn’t have a different standard than old Google. “We do see more diversity of sources that are coming forth. But the aim is really to continue to put high quality content at the top,” she said.

Choosing who to believe is hard enough for humans. What makes Google think its current AI tech, known as LLMs, or large language models, is up to the task?

“They’re not perfect,” Reid said. “We want to take this thoughtful approach because the brand of trust that people have with Google is really important.”

The future of our information depends on it.



Source link

Most Popular

LEAVE A REPLY

Please enter your comment!
Please enter your name here

More from Author

Read Now

Fever select Caitlin Clark with No. 1 pick in WNBA Draft

BROOKLYN, N.Y. — Eight days after completing one of the greatest collegiate basketball careers ever recorded, Caitlin Clark was selected with the No. 1 pick in Monday’s WNBA Draft by the Indiana Fever.Clark is poised to not only help the Fever return to the postseason for...

In 1998, Bollywood star killed two blackbucks. A gang still seeks revenge.

Khan has been in the crosshairs of an armed gang since 1998 for shooting two blackbucks on a recreational hunting trip.Two members of a notorious criminal gang have been arrested for firing at the home of Bollywood actor Salman Khan in apparent retaliation for the superstar’s...

14 More Games Land on PS Plus Extra, Premium Today

As always, we ran a Poll over the weekend to gather your thoughts on the line-up, and (almost) as always, one-third of you responded by saying "it's a crap selection". 26 per cent of voters were at least somewhat happy with the selection, with Dave the...

Where Biden’s billions in chipmaking money have gone — and what could be next

The White House announcement this week of up to $6.4 billion in chipmaking cash for Samsung (005930.KS) marks a move to the next phase of the Biden administration's effort to bring semiconductor manufacturing back to the US.The focus so far has largely been on leading-edge logic...

WNBA expansion headed for 16 teams. Portland, Toronto and Philly are options

NEW YORK — The WNBA is looking to continue growing in the next few years. The league intends to reach 16 teams by 2028, commissioner Cathy Engelbert said before the draft, confirming The Athletic report from earlier Monday.The league announced earlier this year that the Bay...

Microsoft Makes High-Stakes Play in Tech Cold War With Emirati A.I. Deal

Microsoft on Tuesday said it would make a $1.5 billion investment in G42, an artificial intelligence giant in the United Arab Emirates, in a deal largely orchestrated by the Biden administration to box out China as Washington and Beijing battle over who will exercise technological influence...

A hockey puck hurtled toward her son’s head. A fan jumped to block it.

Just over a minute was left on the clock during a hockey game in Cleveland Thursday when a player struck the puck, sending it barreling into the stands and toward Asia Davis’s 4-year-old son. That’s when another fan jumped and extended his right arm to block...

Thailand’s Finance Ministry to lead 17 agencies in considering House report on legalizing casinos

Thailand’s Ministry of Finance has been appointed lead agency to consider a House committee study recommending the legalization of casino gaming but will work with 16 other agencies in doing so, local media reports. According to English-language outlet The Nation, the appointment was ordered by way of...

WWE RAW Results Coverage, Reactions and Highlights for April 15, 2024

It’s Monday night, and you know what that means! WWE RAW will go down tonight, and Ringside News has got you covered with live play-by-play results coverage for the April 15, 2024 episode. The start time for WWE RAW is 8:00 PM EST. Match-by-match highlights...

Coachella Weekend 1 Recap, J Balvin’s Out Of This World Coachella Set, Metro Boomin & Future Top Hot 100 | Billboard News

It’s Monday April 15th, Billboard counts down the top 10 on the Hot 100, we’re running down all the headliners and special guest performers and some notable audience members that went down at the first weekend of Coachella, J Balvin gives us a behind the scene...