Social media is filled with examples of Google's new AI Overview product generating bizarre responses, such as suggesting users put glue on their pizza or eat rocks. Due to the messy rollout, Google is rushing to manually disable AI Overviews for specific searches as various memes are being posted. This explains why users are witnessing many of these disappear shortly after being shared on social networks.
This situation is quite unusual considering that Google has been testing AI Overviews for a year now. The feature was launched in beta in May 2023 as the Search Generative Experience, and CEO Sundar Pichai mentioned that the company served over a billion queries during that time.
Pichai also stated that Google has managed to reduce the cost of delivering AI answers by 80 percent over the same period, thanks to advancements in hardware, engineering, and technology. It seems that this optimization may have occurred too early, before the technology was fully prepared.
"Once known for delivering cutting-edge and high-quality products, the company is now recognized for producing low-quality output that has become the subject of memes," commented an anonymous AI founder to 24 News.
Google insists that its AI Overview product consistently provides "high quality information" to users. According to Google spokesperson Meghann Farnsworth, many of the examples they have observed are unique queries, and there have been instances of manipulated or unreproducible examples. Farnsworth also stated that the company is promptly removing AI Overviews for specific queries that violate their content policies, and using these instances to make broader improvements to their systems, some of which are already being implemented.
Gary Marcus, a renowned AI expert and emeritus professor of neural science at New York University, expressed to The Verge that many AI companies are promoting unrealistic expectations, claiming that their technology will progress from 80 percent accuracy to 100 percent. According to Marcus, achieving the initial 80 percent is relatively straightforward as it involves processing a vast amount of human data. However, he believes that reaching the final 20 percent is an extremely challenging task, and in his opinion, it might be the most difficult aspect.
Marcus explained that to achieve the last 20 percent, AI systems need to engage in reasoning processes, such as evaluating the plausibility of information and determining the legitimacy of sources. These tasks resemble the work of a human fact checker and may require artificial general intelligence. Both Marcus and Meta's AI chief Yann LeCun concur that the current large language models powering AI systems, such as Google's Gemini and OpenAI's GPT-4, will not lead to the development of AGI.
Google is facing a challenging situation. Bing has made significant strides in AI, with Satya Nadella's famous "we made them dance" quote, and OpenAI is reportedly developing its own search engine. A new AI search startup is already valued at $1 billion, and a younger generation of users is gravitating towards TikTok for the best experience. The company is clearly under pressure to compete, which can lead to messy AI releases. In 2022, Meta had to take down its AI system called Galactica shortly after its launch, as it made inappropriate suggestions, similar to what Google is experiencing now.
Google has big plans for AI Overviews, with the current feature being just a small part of what was announced last week. The company aims to implement multistep reasoning for complex queries, create an AI-organized results page, and introduce video search in Google Lens. While there is a lot of ambition behind these plans, Google's reputation currently depends on getting the basics right, which is proving to be a challenge.
According to Marcus, the models being used are unable to perform sanity checking on their own work, which has become a significant issue for the industry.