Fact-Checking the Top 9 Chat GPT Health Answers With Doctors
Together with doctors, we analyzed the AI-generated information for the most popular health queries related to flu, diabetes, blood pressure, RSV, cholesterol, ADHD, and others.
Many people go online for health information. It saves time, money, and effort. However, web sources can sometimes be vague, too promotional, or simply inaccurate.
Instead of searching online, more people are turning to artificial intelligence (AI) tools. For example, a million users joined Chat GPT in the first five days.1 By January 2023, it grew to 100 million, becoming the fastest-growing app in the world.2
By scraping the entire internet, AI provides brief answers to concrete medical questions. Although it warns it doesn’t have credibility, users rely on it.
As health professionals, we want our patients to get accurate and reliable medical information. Together with doctors, we analyzed the Chat GPT answers to the most frequently searched health-related questions on Google and figured out what to trust.
What Are the Most Popular Health Questions?
In April 2023, we used the Ahrefs program to analyze more than 100 Google health search terms.
In the following table, search volume represents the number of searches over the last 12 months for a keyword. Keywords are words or phrases users type into search engines to find information. Our study excluded COVID-19-related keywords to avoid monotony.
We classified the most common health-related keywords into 3 main categories:
- “Symptoms”: flu symptoms (618K searches), diabetes symptoms (331K), ADHD symptoms (310K).
- “How to”: how to lower blood pressure (156K), how to get rid of hiccups (136K), how to lower cholesterol (107K).
- “Others”: what is rsv (180K), is pneumonia contagious (164K), what causes high blood pressure (105K).
Experiment Methodology
After identifying the most popular health-related Google searches, we sent them to Chat GPT. The first AI-generated message, following the query, was accepted as an answer. No further questions were asked.
Then, together with medical experts, we evaluated the accuracy of the information provided and considered whether the answers could be used for first-hand knowledge. We also noted potential pitfalls or areas where additional information may be necessary.
#1 Flu Symptoms
The Chat GPT answer:
“Influenza, commonly known as the flu, is a respiratory illness caused by the influenza virus. The symptoms of the flu can vary from person to person, but common symptoms include:
- Fever or feeling feverish/chills
- Cough
- Sore throat
- Runny or stuffy nose
- Muscle or body aches
- Headaches
- Fatigue (tiredness)
- Some people may also experience vomiting and diarrhea, although this is more common in children than adults.
It is important to note that not everyone with the flu will experience all of these symptoms, and some people may have mild symptoms, while others may experience more severe symptoms. If you are experiencing flu-like symptoms, it is important to rest, stay hydrated, and seek medical attention if necessary.”
Expert opinion
The AI-generated text here on flu symptoms is accurate in the information relayed and reasonably complete.
However, when readers look up an article on flu symptoms, they will likely be interested in what distinguishes them, for example, from those of COVID-19 (for which there is substantial overlap) and the common cold (for which there are significant differences).
What really stands out about flu symptoms is the constellation of achiness – especially muscle aches and headaches – coupled with fever and pervasive fatigue.
Flu really “knocks you out” much more than a cold, mainly because of the muscle soreness, weakness, and uncomfortable chills accompanying many fevers. In presenting the symptoms, I’d give these top billing, with a quick heads-up that this helps to differentiate the flu from a cold.
While it’s true that flu episodes generally also entail bouts of a persistent cough, sore throat, and runny nose, these are also seen far more frequently with colds.
Flu symptoms strongly parallel COVID, except for shortness of breath and loss of taste/smell. The severity of related COVID symptoms is significantly higher than that of the flu.
Overall, the AI-generated text can be great for a quick, easy, and generally accurate topic read. But you shouldn’t expect much depth or comparative nuance to be relayed.
#2 Diabetes Symptoms
The Chat GPT answer:
“Diabetes is a chronic medical condition characterized by high levels of sugar (glucose) in the blood. The symptoms of diabetes can vary depending on the type of diabetes and the severity of the condition. Some common symptoms of diabetes include:
- Increased thirst and hunger
- Frequent urination
- Fatigue (tiredness)
- Blurry vision
- Slow healing of cuts or wounds
- Numbness or tingling in the hands or feet
- Dry, itchy skin
- Unexplained weight loss (in type 1 diabetes)
- Recurring infections, such as skin or urinary tract infections
It is important to note that some people with diabetes may not experience any symptoms, especially in the early stages of the condition. Therefore, it is important to have regular check-ups with a healthcare professional to monitor blood sugar levels and to detect diabetes early. If you are experiencing any of these symptoms, it is important to seek medical attention promptly.”
Expert opinion
AI’s description of diabetes is textbook-accurate and, for the most part, acceptable. Because Chat GPT only possesses a certain amount of information right now, I don’t think that definition is 100% correct. Type 2 diabetes is usually reversible, and it’s almost too easy to show this. This was just studied recently, so the Chat GPT database might not have that knowledge yet.
Symptoms may vary depending on the type and severity of diabetes. The answer also does not provide detailed information on the potential complications of diabetes, which can be severe and life-threatening if left untreated. Also, some patients with diabetes may not experience symptoms, so they need regular check-ups and monitoring of blood sugar levels.
#3 ADHD Symptoms
The Chat GPT answer:
“ADHD, or attention-deficit/hyperactivity disorder, is a neurodevelopmental disorder that affects both children and adults. The symptoms of ADHD can vary depending on the individual and can be classified into three main categories: inattention, hyperactivity, and impulsivity. Some common symptoms of ADHD include:
Inattention:
- Difficulty paying attention to details or making careless mistakes
- Difficulty staying focused on tasks or activities
- Difficulty organizing tasks and activities
- Forgetfulness and losing things
- Easily distracted by noises and other stimuli
Hyperactivity:
- Fidgeting and squirming
- Difficulty sitting still or remaining seated
- Excessive talking and interrupting others
- Difficulty engaging in quiet activities
- Always being ‘on the go’ and difficulty relaxing
Impulsivity:
- Acting without thinking
- Difficulty waiting for a turn
- Interrupting or intruding on others’ conversations or activities
- Engaging in risky behavior without considering the consequences
It is important to note that not everyone with ADHD will experience all of these symptoms, and some individuals may experience symptoms primarily in one category over others. Additionally, the severity of symptoms can vary from mild to severe. If you or a loved one is experiencing symptoms of ADHD, it is important to seek evaluation and treatment from a healthcare professional.”
Expert opinion
This answer is medically and psychiatrically accurate, and it is a helpful snapshot of the different signs and symptoms of ADHD. It also considers that not everyone presents similarly, and individuals can have varied symptoms.
This information serves best as a primer and introductory explainer. ADHD, like any disorder, is complex with many nuances.
When it comes to AI-generated answers, they are useful because they distill a large amount of info into digestible chunks and exclude a lot of jargon.
It’s similar to looking at a Wikipedia site and having the main facts before you. When you’re not a sophisticated information consumer, you are left thinking there’s nothing there, or its impact is minimized.
For in-depth analysis and information gathering, you’ll need more than AI-generated information – for example, a consultation with a doctor.
#4 How to Lower Blood Pressure
The Chat GPT answer:
“High blood pressure, also known as hypertension, is a common medical condition that can increase the risk of heart disease and stroke. Here are some ways to lower blood pressure:
- Exercise regularly: Regular physical activity can help lower blood pressure. Aim for at least 30 minutes of moderate-intensity exercise most days of the week.
- Eat a healthy diet: A diet rich in fruits, vegetables, whole grains, lean proteins, and low-fat dairy products can help lower blood pressure. Limit salt intake and avoid high-sodium processed foods.
- Maintain a healthy weight: Losing weight can help lower blood pressure, especially if you are overweight or obese.
- Limit alcohol consumption: Drinking too much alcohol can raise blood pressure, so it is recommended to limit alcohol intake to no more than one drink per day for women and two drinks per day for men.
- Quit smoking: Nicotine in cigarettes can raise blood pressure, so quitting smoking can help lower blood pressure.
- Manage stress: Chronic stress can contribute to high blood pressure, so finding ways to manage stress, such as meditation or deep breathing exercises, can help lower blood pressure.
- Take medication as prescribed: If lifestyle changes alone are not enough to lower blood pressure, your doctor may prescribe medication to help control it.
It is important to note that blood pressure management is a long-term process that requires commitment and lifestyle changes. It is important to work with your healthcare provider to determine the best approach for you to manage your blood pressure.”
Expert opinion
The answer is medically correct and provides a comprehensive overview of ways to lower blood pressure through lifestyle changes. Also, it’s good that it didn’t list specific antihypertensive medications so patients won’t self-medicate.
However, it doesn’t specify the best blood pressure range or how often, how long, and how intense to exercise, depending on the individual’s age, sex, health status, and other factors. Some people with underlying medical conditions need additional or alternative treatments to manage blood pressure effectively.
#5 How to Get Rid of Hiccups
The Chat GPT answer:
“Hiccups are involuntary contractions of the diaphragm muscle, which can cause a sudden intake of air and a characteristic ‘hic’ sound. Although hiccups are usually harmless and self-limiting, they can be annoying and disruptive. Here are some ways to get rid of hiccups:
- Hold your breath: Take a deep breath and hold it for as long as you can, or until you feel a slight discomfort. Exhale slowly and repeat as necessary.
- Breathe into a paper bag: Breathing into a paper bag can help regulate your breathing and reduce hiccups.
- Drink water: Drinking a glass of water quickly and continuously without stopping can help stimulate the vagus nerve and stop hiccups.
- Swallow sugar: Swallowing a teaspoon of sugar can help stimulate the vagus nerve and stop hiccups.
- Gargle with ice water: Gargling with ice water can help stimulate the vagus nerve and stop hiccups.
- Pull your knees to your chest: Pulling your knees to your chest and holding them there for a few minutes can help relax your diaphragm muscle and stop hiccups.
- Have someone startle you: Having someone startle you can interrupt the hiccup reflex and stop hiccups, but this method is not always effective or recommended.
It is important to note that hiccups usually resolve on their own within a few minutes to hours, so it is not necessary to seek medical attention for hiccups unless they persist for an extended period of time or are accompanied by other symptoms.”
Expert opinion
This answer provides a medically correct overview of hiccups and several ways to stop them. The methods listed are generally safe and commonly used. Additionally, some of the tips listed, such as having someone startle you, are not always effective or recommended because they can cause harm or injury.
Another option mentioned is to hold your breath, which increases the amount of carbon dioxide in the blood and reduces the urge to hiccups. This method is based on the fact that a disturbance in the typical breathing pattern often triggers them. However, this method may only work for some, and hiccups usually resolve on their own within a few minutes to an hour.
#6 How to Lower Cholesterol
The Chat GPT answer:
“High cholesterol is a common medical condition that can increase the risk of heart disease and stroke. Here are some ways to lower cholesterol:
- Eat a heart-healthy diet: A diet rich in fruits, vegetables, whole grains, lean proteins, and low-fat dairy products can help lower cholesterol. Avoid high-fat and high-cholesterol foods, such as fried foods, fatty meats, and full-fat dairy products.
- Exercise regularly: Regular physical activity can help lower cholesterol. Aim for at least 30 minutes of moderate-intensity exercise most days of the week.
- Maintain a healthy weight: Losing weight can help lower cholesterol, especially if you are overweight or obese.
- Quit smoking: Smoking can lower ‘good’ HDL cholesterol levels, so quitting smoking can help increase HDL levels and lower overall cholesterol.
- Limit alcohol consumption: Drinking too much alcohol can increase cholesterol levels, so it is recommended to limit alcohol intake to no more than one drink per day for women and two drinks per day for men.
- Take medication as prescribed: If lifestyle changes alone are not enough to lower cholesterol, your doctor may prescribe medication to help control it.
It is important to note that cholesterol management is a long-term process that requires commitment and lifestyle changes. It is important to work with your healthcare provider to determine the best approach for you to manage your cholesterol.”
Expert opinion
In medicine, we first do blood tests, appraise lifestyle, and only then provide specific recommendations. Also, the AI-generated answer doesn’t contain such information:
- Chronic stress can lead to high cholesterol levels. In such people, we need to implement effective stress management strategies.
- Genetic factors, such as familial hypercholesterolemia, can cause high cholesterol. Statin therapy is required for them.
In general, it is a good response on how to lower cholesterol in most cases.
#7 What Is RSV?
The Chat GPT answer:
“RSV stands for Respiratory Syncytial Virus, which is a common respiratory virus that can cause mild to severe respiratory illness, particularly in young children and older adults. It can infect the lungs and breathing passages, leading to symptoms such as cough, runny nose, fever, and wheezing.
In most cases, RSV infections cause mild cold-like symptoms that can be treated at home, but in some cases, particularly in young infants and older adults with weakened immune systems, RSV infections can lead to severe complications such as pneumonia or bronchiolitis. RSV is highly contagious and spreads through respiratory secretions, such as coughing or sneezing, or by touching contaminated surfaces.
There is currently no specific treatment for RSV, but supportive care such as rest, hydration, and fever management can help alleviate symptoms. In severe cases, hospitalization and supplemental oxygen may be necessary. Vaccines and antiviral medications are available for high-risk populations.”
Expert opinion
The AI-generated response about RSV, the symptoms, in-home care, and contagiousness is medically correct. However, it’s not accurate to say that a vaccine to prevent RSV infection or antiviral medications to treat RSV are available because scientists are still actively developing them. However, a preventive treatment called a monoclonal antibody can help protect at-risk populations from the virus.
One of the pitfalls I see for AI-generated responses as a pharmacist is some inaccuracies in treatments and prevention for conditions. A critical missing item is prevention strategies like frequent handwashing, wearing a mask, and cleaning high-touch surface areas.
To provide a comprehensive and informative answer, it may be beneficial for the AI-generated response to include additional information about central apnea, the limited efficacy of nebulized medications, the relationship between RSV and asthma, indications for specific therapies for high-risk populations, and the potential risks of death and hospitalization.
#8 Is Pneumonia Contagious?
The Chat GPT answer:
“Pneumonia can be contagious, depending on the cause of the infection. Pneumonia can be caused by viruses, bacteria, fungi, or other microorganisms, and the contagiousness can vary depending on the specific pathogen causing the infection.
Bacterial pneumonia caused by Streptococcus pneumoniae, Haemophilus influenzae, and Legionella pneumophila, among others, can be contagious and can spread from person to person through respiratory secretions, such as coughing or sneezing.
Viral pneumonia, caused by viruses such as influenza, respiratory syncytial virus (RSV), or adenovirus, can also be contagious and spread through respiratory secretions.
Fungal pneumonia and non-infectious types of pneumonia, such as aspiration pneumonia caused by inhaling food, liquid, or vomit, are not typically contagious.
To prevent the spread of contagious pneumonia, it is important to practice good hygiene, such as washing hands frequently, covering the mouth and nose when coughing or sneezing, and avoiding close contact with people who are sick. It is also important to seek medical attention promptly if you have symptoms of pneumonia, as early treatment can help reduce the risk of complications and reduce the risk of spreading the infection to others.”
Expert opinion
This information is medically correct and can be first-hand knowledge. It explains the various causes of pneumonia and which types are contagious. Also, it offers practical advice on preventing the spread of infectious pneumonia.
However, the answer doesn’t mention asymptomatic carriers. People with pneumonia who don’t show symptoms can still spread it to others.
Additionally, AI could have provided more information on the transmission modes of pneumonia types. For example, bacterial pneumonia is primarily transmitted through respiratory droplets, while Legionnaires’ disease can be contracted by inhaling contaminated water.
#9 What Causes High Blood Pressure?
The Chat GPT answer:
“There are many factors that can contribute to high blood pressure, also known as hypertension. Some common causes include:
- Unhealthy lifestyle habits: Unhealthy lifestyle habits such as a diet high in salt and saturated fats, lack of physical activity, excessive alcohol consumption, and smoking can contribute to high blood pressure.
- Genetics: High blood pressure can run in families, suggesting that genetics may play a role in its development.
- Age: Blood pressure tends to increase with age, as the arteries become less elastic and more narrow, and the body’s natural systems for regulating blood pressure become less effective.
- Obesity: Being overweight or obese can increase the risk of high blood pressure.
- Chronic conditions: Certain chronic conditions such as kidney disease, diabetes, and sleep apnea can contribute to high blood pressure.
- Stress: Chronic stress can contribute to high blood pressure.
- Certain medications: Certain medications such as birth control pills, some cold and allergy medications, and certain prescription drugs can cause a temporary increase in blood pressure.
It is important to identify and address the underlying causes of high blood pressure in order to effectively manage the condition and reduce the risk of complications. Treatment may involve lifestyle changes, medication, or a combination of both.”
Expert opinion
This answer is medically correct. One potential flaw is that the answer may not provide a complete list of all possible causes of high blood pressure, as there may be fewer common contributing factors – kidney disease, sleep apnea, adrenal gland tumors, or pregnancy.
Can You Trust Chat GPT’s Health Answers? Our Final Verdict
For a conclusive evaluation, we consulted with Dr. J. Wes Ulm – a US-based physician, medical researcher, and bioinformatics expert with MD and PhD degrees from Harvard Medical School and MIT.
As someone who has worked in medical text generation, Dr. Wes says it exemplifies many of the pros and cons of artificial intelligence.
So what are the cons? Experts name the same issues that hobble AI text generators everywhere.
Even the best AIs, such as Chat GPT, aren’t great at describing and communicating the quality and nuance of their knowledge. And they don’t have the intuition to tailor articles to the questions readers may have.
“This can be seen by assessment of the ‘5 c’s’ of good composition, the kind of thing your university writing professor might have delved into on Day 1.
The AI-generated text is correct, concise, and complete in what it conveys. Still, there’s a lack of context and connection to what’s most relevant to know,” says Dr. Wes.
He adds that AI is just the result of learned patterns in corpora (the plural form of canon, a body of text). They are fed and fine-tuned into software algorithms designed for a specific task, producing a short capsule summary of health topics.
“In layman’s terms, it explains a well-known and well-traveled medical topic in consumer health with a controlled vocabulary. For this, it scans tons of solid, succinct articles online. So the AI wouldn’t even need to train on a large corpus. It could just recapitulate a few of the best pieces already out there,” concludes Dr. Wes.
In other words, AI-generated text gives you a quick, effortless, and mostly accurate understanding of your health questions. However, don’t expect it to be comprehensive or detailed.
Remember that AI answers should not replace medical advice from healthcare providers. For any health-related concerns, consult your doctor instead of self-diagnosing online.
This way, you can protect yourself from risks and receive accurate and timely advice.
Sources
- Greg Brockman, Open AI Co-Founder:
https://twitter.com/gdb/status/1599683104142430208?lang=en - Chat GPT becomes fastest-growing app in the world, records 100M users in 2 month — Mint:
https://www.livemint.com/news/chatgpt-becomes-fastest-growing-app-in-the-world-records-100mn-users-in-2-month-11675484444142.html