Microsoft’s public demo of new AI-powered Bing has bugs

Microsoft’s public demonstration last week of an AI-powered revamp of Bing appears to have included several factual errors, highlighting the risk the company and its rivals face in incorporating this new technology into search engines.

At the Bing demo at Microsoft headquarters, the company showed how integrating artificial intelligence capabilities from the company behind ChatGPT would empower the search engine to deliver more conversational and complex search results.

The demo included a list of pros and cons for products such as vacuum cleaners; an itinerary for a trip to Mexico City; and the ability to quickly compare corporate earnings results.

But it apparently failed to differentiate between types of vacuums and even made up information about certain products, according to an analysis of this week’s demo by independent AI researcher Dmitri Brereton. He also missed relevant details (or forged certain information) for the referenced bars in Mexico City, according to Brereton.

Furthermore, Brereton found that the engine incorrectly stated the operating margin for retailer Gap and compared it to a set of Lululemon results that were not factually correct.

“We are aware of this report and are reviewing its findings in our efforts to improve this experience,” Microsoft said in a statement.

“We recognize that there is still work to be done and we expect the system to make mistakes during this preview period, so feedback is critical so that we can learn and help models improve.”

The company also said that thousands of users have interacted with the new Bing since the preview was released last week and shared their feedback, allowing the model to “learn and already make many improvements”.

The discovery of Bing’s apparent errors comes just days after Google was called out over an error made in its public demonstration last week of a similar AI-powered tool.

Google shares lost $100 billion in value after the error was reported. (Microsoft stock was basically flat on Tuesday.)

After the viral success of ChatGPT, an AI chatbot that can generate surprisingly convincing trials and responses to user requests, a growing number of tech companies are racing to deploy similar technology in their products. But that comes with risks, especially for search engines, whose goal is to surface accurate results.

Generative AI systems, which are algorithms trained on massive amounts of online data to create new content, are notoriously unreliable, experts say.

Laura Edelson, a computer scientist and disinformation researcher at New York University, previously told CNN : “there is a big difference between an AI appearing authoritative and actually producing accurate results”.

A CNN also conducted a series of tests this week that showed that Bing sometimes struggles with accuracy.

When asked, “What were Meta’s fourth quarter results?” the Bing AI feature gave a response that said “according to the press release” and then listed the dots that appear to indicate the Meta results.

But the markers were incorrect. Bing said, for example, that Meta generated $34.12 billion in revenue, when the actual figure was $32.17 billion, and said that revenue was up year-over-year when it had actually dropped .

In a separate survey, the CNN asked Bing, “What are the pros and cons of the best cribs for babies?” In its response, the resource listed several cribs and their pros and cons, which were cited extensively in a similar Healthline article.

But Bing declared information that appeared to be attributed to the article that, in fact, was not there. For example, the engine said that a crib had a “water resistant mattress”, but that information was not listed anywhere in the article.

Executives at Microsoft and Google have already acknowledged some of the potential problems with new AI tools.

“We know we won’t be able to answer every question every time,” said Yusuf Mehdi, vice president and director of consumer marketing at Microsoft, last week.

“We also know that we’re going to make our share of mistakes, so we’ve added a quick feedback button at the top of every survey so you can give us feedback and we’ll learn.”