Listen to this article
Estimated 5 minutes
The audio version of this article is generated by text-to-speech, a technology based on artificial intelligence.
Chatbots such as ChatGPT can provide direct answers to your questions that seem complete and comprehensive. But a new study shows that, compared to a regular Google search, the amount of knowledge gained is less.
We have become accustomed to looking up information on Google when working on projects or just having casual conversations and want to verify an argument. Google usually responds by referring you to a number of websites that deal with the topic, which you have to go through to pick out the information you are looking for.
On the other hand, chatbots, which have access to huge data sets and use large language models, can sift through all that material for you, and spit out an answer in a very conversational, almost personal way. Because the chatbot is programmed to sound like a human and give you a thorough result, the tendency is to accept it at face value.
A study at the University of Pennsylvania tested more than 10,000 students over seven different experiments by asking them to learn about a topic — such as how to plant a vegetable garden — then write advice on that topic to a friend. Half of the group used a regular Google search, the other used the latest version of ChatGPT.
The results were then analyzed for how much the participants had actually learned, and how likely they were to follow their own advice. Researchers found that across all seven experiments, those who used the regular Google search provided longer, more thoughtful presentations using richer, more varied language than those using ChatGPT. And the people who used ChatGPT were overall less informative in the advice they gave, and less likely to have confidence in the advice they gave.
The scientists concluded that because a Google search presents a number of links on the topic, it requires more brainpower to understand the subject before writing it down. Different sites might provide alternative views or suggestions, so the users had to check the sources of that information and come to their own conclusions.
Even though one of the experiments involved using a version of ChatGPT with plenty of links and references, only a quarter of the participants actually clicked a single link. The rest just accepted whatever answers were given.
This study is just one of many similar projects looking at the cognitive costs of chatbots. A group at MIT used EEGs to peer into the brains of subjects as they wrote SAT essays. In a pre-print study, they found that those using chatbots all wrote similar essays, and had the lowest level of neural activity through the process. They also discovered that the results were cumulative, so the engagement in the brain reduced more with each subsequent essay.
Meanwhile, the participants who used no technology at all to write their essays showed the highest levels of neural activity, and were overall more creative, curious, engaged, and satisfied with the end results.
This demonstrates the difference between information and knowledge. We are surrounded by an ocean of information on our computers and devices, but much of it can easily pass through one ear and out the other. If it doesn’t sink in, you haven’t learned anything.
Whereas if you have to actually do some thinking, and assemble information through research to gather it into a body of knowledge, there is a much better chance it will stick. You have really learned something.
Clearly, AI programs are here to stay. They are proving to be powerful research tools, gathering details or seeing patterns in data that humans miss. But these results suggest that critical thinking, fact checking, and source verification are still important parts of the process.
I tried this myself on the topic of how to stay healthy after the age of 70. ChatGPT gave me a simple list of daily exercise and eating habits to stay fit. Google gave me links to pages from medical institutions with far more in-depth reading material that explained the process of aging, how the body changes as we grow older, and how to adapt exercise and diet appropriately for a senior.
The end results were the same, but the Google results came from a much richer database that I cruised through myself to see which method works best. And I felt more informed after the Google search than I did using the chatbot.
Whatever search engine you use, remember to take the time to think about the results you’re getting, and you just may really learn something.
