Background : A recent assessment of ChatGPT on a variety of obstetric and gynecologic topics was very encouraging. However, its ability to respond to commonly asked pregnancy questions is unknown. Reference verification needs to be examined as well. Purpose: To evaluate ChatGPT as a source of information for commonly asked pregnancy questions and to verify the references it provides. Methods: Qualitative analysis of ChatGPT was performed. We queried ChatGPT Version 3.5 on 12 commonly asked pregnancy questions and asked for its references. Query responses were graded as “ acceptable” or “ not acceptable” based on correctness and completeness in comparison to American College of Obstetricians and Gynecologists (ACOG) publications, PubMed-indexed evidence, and clinical experience. References were classified as “verified” , “broken”, “irrelevant”, “non-existent” or “no references” . Review and grading of responses and references were performed by the co-authors individually and then as a group to formulate a consensus. Results: In our assessment, a grade of acceptable was given to 50% of responses (6 out of 12 questions). A grade of not acceptable was assigned to the remaining 50% of responses (5 were incomplete and 1 was incorrect). In regard to references, 58% (7 out of 12) had deficiencies (5 had no references, 1 had a broken reference, and 1 non-existent reference was provided). Conclusion: Our evaluation of ChatGPT confirms prior concerns regarding both content and references. While AI has enormous potential, it must be carefully evaluated before being accepted as accurate and reliable for this purpose.
Artificial intelligence (AI) has emerged as a transformative force and expanded into everyday life. Large-language models (LLM), a subset of AI, have rapidly become popular due to their near-human ability to write coherently across an endless range of topics. Chat Generative Pre-Training Transformer (ChatGPT) Version 3.5 (also referred to as GPT-3.5 by its creator, OpenAI) is the fastest growing internet application to reach 100 million users, achieving this feat within 2 months of its release in November 2022 [
ChatGPT has shown tremendous promise in clinical application. It demonstrated competence in clinical reasoning and general medical knowledge by passing USMLE Steps 1, 2, and 3 without formal medical training [
ChatGPT’s matter-of-fact responses are not without flaw. It may hallucinate output or “make up facts”. AI-generated essays submitted by college students have been described as “really well-written [but] wrong” [
We know that pregnant women frequently use the internet for information regarding pregnancy topics. A survey of postpartum women with internet access estimated that more than 80% use it at least weekly to answer their pregnancy-related questions [
We acknowledge that a caveat emptor approach makes sense to a panel of reviewers with postgraduate degrees; however, use by individuals without this advanced training may be problematic. Popp (2023) suggests that “users need to double check absolutely everything” [
AI technology is expected to infiltrate most industries within the next decade prompting many to voice concerns regarding human replacement by these 21st century machines. While the interest and study of chatbots continues to surge rapidly, the issue of user confidence and acceptance of this technology remains unclear [
The purpose of this study is to evaluate ChatGPT as a source of acceptable answers for commonly asked pregnancy questions and to verify the references it provides. Proof that ChatGPT is a valid and trustworthy source of pregnancy information based on reliable references is essential prior to its endorsement for this purpose.
For a glossary of terminology relating to artificial intelligence, please see the Appendix section.
We queried ChatGPT-3.5 with 12 commonly asked pregnancy questions based on current literature and clinical experience. We chose the basic format “Can I [action] in pregnancy?” as we anticipate a pregnant woman might do when seeking information regarding a specific activity as it relates to her pregnancy. We excluded questions that were deemed too broad, such as “What medications are safe in pregnancy?” “Can I exercise during pregnancy?” and “Can I go to the dentist while pregnant?” Each queried question was followed up by the prompt “Tell me your references for this question” to record the LLM’s source for its response. A new chat was created each time with the prior data unsaved to avoid the model’s learning mechanism. All queries were conducted and recorded from May 14-20, 2023. The query questions are listed in
| 1) Can I fly on a plane while pregnant? 2) Can I drink alcohol while pregnant? 3) Can I drink coffee while pregnant? 4) Can I eat deli meats while pregnant? 5) Can I dye my hair while pregnant? 6) Can I do laser hair removal while pregnant? 7) Is marijuana use safe during pregnancy? 8) Can I eat sushi while pregnant? 9) Can I vape while pregnant? 10) Can I do yoga while pregnant? 11) Can I go in a hot tub while pregnant? 12) What sleep position is best during pregnancy? |
|---|
Responses were recorded, reviewed and graded by each co-author individually and then as a group to formulate a consensus. The review team consisted of two board-certified OBGYN physicians (JDB, RAR), two OBGYN resident physicians (KK, NF), and two second year medical students (CW, AC) of which three identified as female, three as male with an age range from 24 to 56 years old. We chose American College of Obstetricians and Gynecologists (ACOG) Practice Bulletins and Committee Opinions as the default expert opinion (whenever available) against which ChatGPT responses were compared. When ACOG guidance was not available, we used best available PubMed-indexed evidence in addition to clinical experience. Each response was evaluated and assigned a grade as defined in
ChatGPT was queried on 12 commonly asked pregnancy questions and was asked for references. Its verbatim response in italics, consensus grade, and grade explanation are listed in that order. Summaries of ChatGPT Response Grading and ChatGPT Reference Evaluation are presented in
| Grade | Interpretation |
|---|---|
| Acceptable | Response is free from error and contains all necessary information. |
| Not Acceptable | Response is missing necessary information (Incomplete). |
| Response contains factual errors (Incorrect). |
| Reference Issue | Explanation |
|---|---|
| Broken Reference | ChatGPT provided a website address or an article DOI number but the website’s page led to an error message and/or the DOI number was misleading. Digital Object Identifier, DOI, is a string of numbers, letters, or symbols used to uniquely identify an article or document on the internet. |
| Irrelevant Reference | ChatGPT provided a list of references for a query that had little or nothing to do with the topic of interest. |
| Non-Existent Reference | ChatGPT listed references that do not exist. |
| No References | ChatGPT could not provide a list of references. |
| Verified Reference | ChatGPT provided verified reference(s). |
| Question | Response Grade | Explanation |
|---|---|---|
| Air travel | Not Acceptable (Incomplete) | ACOG states, “Most commercial airlines allow pregnant women to fly up to 36 weeks of gestation [ |
| Alcohol | Acceptable | Both ACOG and ChatGPT strongly recommend against the use of alcohol [ |
| Caffeine | Acceptable | Consistent with ACOG and provides a dose-limited recommendation along with various sources of caffeine [ |
| Deli Meat | Acceptable | Consistent with ACOG recommendations on Listeria and proper food safety guidelines [ |
| Hair Dye | Acceptable | Consistent with ACOG expert opinion on dying hair [ |
| Laser Hair Removal | Not Acceptable (Incomplete) | ChatGPT fails to consider the following evidence from a systematic review from 2019. “The available evidence, limited to low evidence-level case reports and series, indicates cutaneous laser treatment during pregnancy is safe for both mother and fetus [ |
| Marijuana Use | Acceptable | ACOG and ChatGPT both recommend against marijuana use, while acknowledging potential risks and uncertainty of current evidence [ |
| Sushi | Not Acceptable (Incorrect) | ChatGPT incorrectly states that women can eat sushi during pregnancy. According to ACOG, “although not mentioned in the revised 2017 advice, it is important that pregnant women avoid all raw and undercooked seafood, eggs, and meat [ |
| Vaping | Not Acceptable (Incomplete) | ChatGPT fails to mention the CDC’s advisory statement regarding vaping in pregnancy. “Recently, the Centers for Disease Control and Prevention (CDC) issued an advisory notice investigating a multistate outbreak of noninfectious severe pulmonary disease associated with e-cigarette and vaping product use. With the recent CDC advisory and the effects of e-cigarette and vaping product use on offspring health, immediate discontinuation of e-cigarette and vaping products should be advised among all pregnant and postpartum women [ |
| Yoga | Not Acceptable (Incomplete) | ChatGPT does not include prospective data from a study published in 2015. Twenty-six yoga positions in healthy pregnant women in their third trimester were evaluated. Maternal and fetal status were evaluated during the yoga session. There were no adverse changes in maternal and fetal well-being during the session [ |
| Hot Tub | Acceptable | Consistent with ACOG and AAP, ChatGPT cautions regarding hot tub exposure and ways to reduce risk of harmful fetal effects [ |
| Sleep Position | Not Acceptable (Incomplete) | ChatGPT does not include information from a prospective study on maternal sleeping position published in 2019 which concluded “supine or non-left sided sleep through 30 weeks gestation was not associated with adverse pregnancy outcomes linked to decreased placental blood flow in a large prospective cohort [ |
| Question | Reference Issue |
|---|---|
| Air Travel | Broken Reference Irrelevant Reference |
| Alcohol | No Reference |
| Caffeine | Verified Reference |
| Deli Meat | Verified Reference |
| Hair Dye | No Reference |
| Laser | Verified Reference |
| Marijuana | Verified Reference |
| Sushi | Non-Existent Reference |
| Vaping | No Reference |
| Yoga | No Reference |
| Hot Tub | No Reference |
| Sleep Position | Verified Reference |
In general, flying on a plane while pregnant is considered safe for most women, but it is recommended that you speak with your healthcare provider before making any travel plans. Your doctor can assess your individual pregnancy and any potential risks or complications that may arise from air travel.
Some airlines have specific policies related to pregnant passengers, so it’s important to check with your airline to see if they have any restrictions or requirements. Additionally, it’s a good idea to plan for extra comfort during your flight by booking an aisle seat, bringing plenty of water, and wearing loose, comfortable clothing.
It’s also important to keep in mind that the risk of developing blood clots during long flights may be increased for pregnant women. To reduce this risk, you can try to move around regularly during the flight, drink plenty of water, and wear compression stockings.
In summary, it is generally safe for most pregnant women to fly on a plane, but it is important to check with your doctor and the airline before making any travel plans.
GRADE: Not Acceptable (Incomplete)
We found ChatGPT’s air travel response to be not acceptable because it is incomplete. Its omission of the statement that “most commercial airlines allow pregnant women to fly up to 36 weeks of gestation” could influence the purchase of non-refundable tickets or plainly lead to unsafe air travel after 36 weeks. In addition, the response lacks recommendations against air travel for women with medical or obstetric conditions that could be exacerbated by flight or that could require emergency care [
As an AI language model, I must prioritize your health and the well-being of your baby. It is strongly recommended to avoid drinking alcohol during pregnancy. Consuming alcohol during pregnancy can pose significant risks to the developing fetus, leading to a condition known as fetal alcohol spectrum disorders (FASDs).
FASDs encompass a range of physical, behavioral, and cognitive disabilities that can occur in babies exposed to alcohol in the womb. These effects can be permanent and have lifelong consequences. There is no known safe level of alcohol consumption during pregnancy, so it’s best to err on the side of caution and abstain from drinking alcohol entirely while pregnant.
If you have concerns or questions about alcohol consumption during pregnancy, it’s crucial to consult with your healthcare provider, who can provide personalized advice and guidance based on your specific circumstances.
GRADE: Acceptable
The consumption of coffee during pregnancy is a topic of debate among healthcare professionals. While moderate caffeine intake is generally considered safe during pregnancy, excessive consumption of caffeine has been associated with an increased risk of certain adverse pregnancy outcomes, such as miscarriage, preterm birth, and low birth weight.
The American College of Obstetricians and Gynecologists recommends that pregnant women limit their caffeine intake to no more than 200 milligrams per day, which is roughly equivalent to one 12-ounce cup of coffee. It’s important to note that caffeine is not just found in coffee, but also in other beverages such as tea, soda, and energy drinks, as well as in some foods and medications.
It’s always a good idea to discuss your caffeine intake with your healthcare provider to determine what level of consumption is safe and appropriate for you during pregnancy. If you choose to drink coffee, you may want to consider switching to decaf or limiting your consumption to one small cup per day to help minimize any potential risks.
GRADE: Acceptable
While it is generally safe to eat deli meats during pregnancy, there are some precautions that you should take to reduce the risk of getting an infection called listeriosis. Listeriosis is caused by the bacterium Listeria monocytogenes, which can be found in some types of deli meats.
To reduce the risk of listeriosis, you should follow these guidelines:
1) Heat deli meats until they are steaming hot before eating them. This will kill any Listeria bacteria that may be present.
2) Avoid eating cold deli meats or meat spreads from the deli counter, unless they have been heated to steaming hot just before serving.
3) Avoid eating refrigerated pâté or meat spreads, as they can contain Listeria.
4) Always wash your hands before and after handling deli meats, and clean any surfaces that come into contact with deli meats.
By taking these precautions, you can safely enjoy deli meats during pregnancy. However, if you are unsure about the safety of any food, it is always best to consult with your healthcare provider.
GRADE: Acceptable
Dyeing your hair while pregnant is a topic that has been a subject of debate among medical professionals. The concern revolves around the potential risks associated with the chemicals present in hair dyes. While limited research is available on the topic, it is generally considered safe to dye your hair during pregnancy, especially after the first trimester. However, it’s always recommended to consult with your healthcare provider to get personalized advice based on your specific situation.
If you decide to dye your hair while pregnant, consider the following precautions:
1) Opt for safer alternatives: Look for hair dyes that are ammonia-free and contain fewer harsh chemicals. Additionally, semi-permanent or vegetable-based dyes are often considered safer options.
2) Good ventilation: Make sure you’re in a well-ventilated area when applying the hair dye to minimize exposure to any fumes.
3) Wear gloves: Use gloves to protect your skin and minimize contact with the dye.
4) Timing: It’s generally advisable to wait until the second trimester to dye your hair. This is because the first trimester is a crucial period of development for the baby.
5) Patch test: Perform a patch test on a small area of your skin before applying the dye to your entire scalp. This can help identify any potential allergic reactions.
6) Limit scalp exposure: Try to minimize the amount of dye that comes into contact with your scalp, as it may be more easily absorbed into your system.
Remember, every pregnancy is unique, and individual circumstances may vary. Therefore, it’s essential to consult with your healthcare provider before making a decision about dyeing your hair while pregnant. They will be able to provide you with personalized advice based on your medical history and current condition.
GRADE: Acceptable
It is generally not recommended to undergo laser hair removal while pregnant. Although there is no concrete evidence that laser hair removal is harmful to the fetus, there have not been enough studies done to determine the safety of the procedure during pregnancy.
Additionally, pregnancy can cause changes in your skin, including increased sensitivity and pigmentation changes, which can make laser hair removal less effective or potentially cause adverse effects.
It is important to prioritize the health and safety of both you and your baby during pregnancy, so it is best to avoid unnecessary risks. If you are considering laser hair removal, it is best to wait until after you have given birth and are no longer breastfeeding. In the meantime, there are other hair removal methods, such as shaving or using depilatory creams, that are safe during pregnancy.
GRADE: Not Acceptable (Incomplete)
We found ChatGPT’s laser hair removal response to be not acceptable because it is incomplete. Its omission of a systematic review from 2019 on the safety of cutaneous laser treatment during pregnancy has the potential to deprive women of the necessary information upon which an informed decision regarding laser hair removal may be made [
The safety of marijuana use during pregnancy is a topic of ongoing debate and research. While it is difficult to draw definitive conclusions due to limited and conflicting studies, the general consensus among healthcare professionals is that using marijuana during pregnancy is not recommended.
Marijuana contains compounds called cannabinoids, including delta-9-tetrahydrocannabinol (THC), which can cross the placenta and affect the developing fetus. Some studies suggest that prenatal marijuana exposure may be associated with an increased risk of negative outcomes, such as low birth weight, preterm birth, and developmental and behavioral issues in children. However, it’s important to note that research in this area is still evolving, and more conclusive evidence is needed.
Additionally, marijuana use during pregnancy may also be associated with other potential risks. Smoking marijuana, for example, exposes both the mother and the fetus to the harmful effects of smoke, which can be harmful to lung health. Furthermore, marijuana use can impair cognitive function and attention, which may impact a pregnant woman’s ability to care for herself and her developing baby.
Given the uncertainty and potential risks, it is generally advised that pregnant women avoid using marijuana or any other recreational drugs during pregnancy. If you have concerns or questions about drug use during pregnancy, it is best to consult with a healthcare professional who can provide personalized guidance based on your specific situation.
GRADE: Acceptable
Yes, you can eat sushi while pregnant, but you need to be cautious and take certain precautions to ensure that the sushi is safe to consume. Raw fish used in sushi can potentially be contaminated with bacteria, viruses, or parasites, which can cause foodborne illnesses.
To reduce the risk of foodborne illness, you should make sure that the sushi is made with fresh, high-quality ingredients and prepared in a clean and sanitary environment. You should also avoid eating high-mercury fish, such as swordfish, shark, king mackerel, and tilefish, as these can be harmful to your developing baby.
It’s important to talk to your healthcare provider about your diet during pregnancy and get their advice on the types and amounts of fish that are safe to eat. They can also advise you on any other dietary restrictions you may need to follow during pregnancy.
GRADE: Not Acceptable (Incorrect)
We found ChatGPT’s sushi response to be not acceptable because it is incorrect. It states that women can eat sushi during pregnancy which contradicts ACOG’s clear statement that pregnant women should avoid all raw and undercooked seafood, eggs and meat [
As an AI language model, I’m not a doctor, but I can provide you with some general information. It is generally recommended that pregnant women avoid using any form of nicotine or tobacco products, including vaping. Nicotine is a highly addictive substance that can have harmful effects on both the pregnant person and the developing fetus.
Vaping during pregnancy can expose the fetus to nicotine, as well as other potentially harmful chemicals found in e-cigarette liquids. Nicotine can constrict blood vessels and restrict the oxygen and nutrient supply to the fetus, potentially affecting its growth and development. It has been associated with various risks, including preterm birth, low birth weight, and developmental issues.
Additionally, vaping devices and e-cigarettes may malfunction, causing injury. The long-term effects of vaping on both the pregnant person and the fetus are still being researched, and the lack of conclusive evidence regarding the safety of vaping during pregnancy means it is generally best to err on the side of caution.
If you’re pregnant and considering using any nicotine or tobacco products, it is advisable to consult with your healthcare provider for personalized advice. They can provide you with the most up-to-date and accurate information regarding the potential risks and alternatives to ensure the health and well-being of you and your baby.
GRADE: Not Acceptable (Incomplete)
We found ChatGPT’s vape response to be not acceptable because it is incomplete. Its omission of the CDC’s advisory statement regarding vaping in pregnancy has the potential to mislead. The CDC clearly recommends the discontinuation of e-cigarette and vaping products in all pregnant and postpartum women. This stance is supported by ACOG [
Yes, yoga can be a safe and beneficial form of exercise during pregnancy, but it is important to take certain precautions and modifications to ensure the safety of both you and your growing baby.
It is recommended to consult with your healthcare provider before starting any new exercise program, including yoga. They can provide specific advice based on your individual pregnancy and any potential risks or complications.
When practicing yoga during pregnancy, it is important to avoid certain poses that may put undue stress on the abdomen, such as deep twists, inversions, or poses that involve lying on the stomach. It is also important to avoid overstretching, and to listen to your body and modify poses as needed.
There are many prenatal yoga classes and videos available that are specifically designed for pregnant women, with modified poses and sequences that are safe and effective for pregnancy. These classes may also include breathing techniques and relaxation practices that can help prepare you for labor and delivery.
Overall, with the proper modifications and guidance, yoga can be a beneficial form of exercise during pregnancy that can help improve flexibility, strength, and relaxation.
GRADE: Not Acceptable (Incomplete)
We found ChatGPT’s yoga response to be not acceptable because it is incomplete. Its omission of prospective data from 2015 may lead a woman away from a potentially beneficial exercise activity shown to be safe during pregnancy [
Before engaging in any activities during pregnancy, it’s always best to consult with your healthcare provider for personalized advice. However, I can provide you with some general information about hot tub use during pregnancy.
Hot tubs, jacuzzis, or saunas that maintain a high temperature can potentially pose risks to pregnant women. Spending extended periods in hot water, especially if the water temperature exceeds 100˚F (38˚C), may lead to an increased body temperature, which can be harmful to the developing fetus.
Raising your core body temperature, particularly during the first trimester, has been associated with an increased risk of certain birth defects. It’s also important to note that high water temperatures can potentially cause dehydration, which is not ideal during pregnancy.
If you still wish to use a hot tub during pregnancy, it’s generally recommended to follow these precautions:
1) Limit your time: Keep your hot tub sessions short, typically less than 10 minutes at a time.
2) Lower the temperature: Make sure the water temperature is below 100˚F (38˚C). Ideally, it should be around body temperature or slightly below.
3) Stay hydrated: Drink plenty of water before and after using the hot tub to prevent dehydration.
4) Pay attention to warning signs: If you start feeling dizzy, lightheaded, or overheated, it’s important to exit the hot tub immediately and cool down.
5) Avoid submerging your entire body: To minimize the risk of overheating, you may choose to sit on the edge of the hot tub or keep your lower body in the water while allowing your upper body to remain exposed to cooler air.
Remember, every pregnancy is unique, and certain medical conditions or complications may make hot tub use more risky. Always consult with your healthcare provider to ensure that hot tub use is safe for you and your baby based on your individual circumstances.
GRADE: Acceptable
During pregnancy, it is recommended that women sleep on their left side as much as possible. This position allows for optimal blood flow to the uterus and kidneys, which can help to prevent swelling, back pain, and other discomforts associated with pregnancy.
Sleeping on the left side also helps to relieve pressure on the inferior vena cava, a large vein that runs along the right side of the body and can become compressed by the weight of the uterus when lying on the back. Compression of this vein can cause dizziness, shortness of breath, and a drop in blood pressure.
While sleeping on the left side is preferred, it is important to note that pregnant women should avoid sleeping on their backs after the first trimester. This is because as the uterus grows, it can put pressure on the vena cava and lead to the symptoms mentioned above.
Additionally, it is important for pregnant women to use pillows and other supports to help them stay comfortable while sleeping. This may include placing a pillow between the legs to alleviate pressure on the hips, or using a body pillow for additional support.
GRADE: Not Acceptable (Incomplete)
We found ChatGPT’s sleep position response to be not acceptable because it is incomplete. Its omission of a prospective evaluation of maternal sleeping position published in 2019 has the potential to mislead patients, as supine or non-left sided sleep through 30 weeks was not associated with adverse pregnancy outcomes [
ChatGPT offered verified references for its responses to questions regarding caffeine, deli meat, laser, marijuana use, and sleep position. It provided no specific references for its responses on alcohol, hair dye, hot tub, vaping, and yoga. ChatGPT provided a ghost reference for its response on sushi and an irrelevant reference for its response on air travel. Overall, ChatGPT had references deficiencies for 58% (7/12) of the questions. When ChatGPT could not provide references, it would respond with a disclaimer: I apologize for any confusion, but as an AI language model, I don’t have direct access to external references or a browsing capability. My responses are generated based on a mixture of licensed data, data created by human trainers, and publicly available data. I have been trained on a wide variety of sources, including books, websites, and other texts, to develop a broad understanding of human language, but I do not have access to specific references for individual statements.
ChatGPT and similar LLM technologies have the potential to link the world’s knowledge to anyone with internet access. Since its release, this potential has been excitedly explored not only by academics, researchers, and clinicians but by the public as well. Initially, ChatGPT-3.5 demonstrated its ability to write e-mails, essays, poems, and songs. It tackled a variety of advanced level examinations including the GRE (Graduate Record Examination), SAT (Scholastic Aptitude Test), and LSAT (Law School Admission Test) and excelled. It passed both the Uniform Bar Exam and USMLE (United States Medical Licensing Examination) [
Assessment of the accuracy of ChatGPT in the medical field, however, has produced mixed results. Yeo et al. evaluated ChatGPT on 164 hepatology questions, noting that 75% of responses were correct but less than half were comprehensive [
User beware! LLMs can generate such coherent and “eloquent” responses so as to mislead the user. Human reviewers were asked to differentiate between ChatGPT-generated scientific abstracts and the original abstract and they struggled, incorrectly identifying more than 30% of AI-generated abstracts as original and 14% of original abstracts as AI-generated [
We set out to verify that ChatGPT could be used as an accurate and reliable source of information for pregnant women without specific medical training or expertise. In regard to ChatGPT’s responses to commonly asked pregnancy questions, 50% were not acceptable, being either incomplete or incorrect. In regard to references, more than 50% were deficient, with ChatGPT providing either broken, irrelevant, non-existent, or simply no references at all.
Pregnant women turn to the internet for information regarding their pregnancy. ChatGPT is an attractive and convenient tool that can compile internet resources in a digestible format. However, it is essential to emphasize that it can be a flawed tool in the hands of the lay person without medical knowledge. Our analysis of ChatGPT regarding commonly asked pregnancy questions demonstrates the potential to mislead pregnant women given that its responses on air travel, laser, sushi, vaping, yoga, and sleep position were deemed to be not accurate. A chatbot response that differs from evidence-based information may cause confusion and stress, and it has the potential to cause harm.
New technologies may be powerful and exciting; however, they must be vetted prior to widespread use and integration into society. We concur with Grünebaum et al. that chatbots are a work in progress. Comparison of ChatGPT to the TI-30 calculator, the technological advancement of its time, reveals two very different stories. This minicalculator, launched almost 50 years ago, became the most popular scientific calculator for junior high and high school use in the United States with an estimated 15 million units being used between 1976 and 1983 [
The World Health Organization (WHO) recently issued an advisory statement for patients, healthcare providers, health systems, and policymakers cautioning that “precipitous adoption of untested systems could lead to errors by healthcare workers, cause harm to patients, erode trust in AI and thereby undermine (or delay) the potential long-term benefits and uses of such technologies around the world” [
To address these concerns, healthcare providers and AI developers need to implement strict protocols for using AI models responsibly and transparently. Combining AI’s capabilities with human expertise and oversight can help optimize patient care while mitigating potential risks. Regular updates and advancements in AI technology may also lead to more refined and accurate AI systems for healthcare applications (https://www.chat.OpenAI.com, accessed July 27, 2023).
Our findings underscore the importance of proper testing and evaluation of ChatGPT and other LLMs prior to widespread adoption by healthcare workers and patients. Our qualitative analysis raises significant concerns regarding the use of ChatGPT as a source of information for commonly asked pregnancy questions. Given that ChatGPT’s user experience has been described as “frictionless” with a “wow” factor along with a small learning curve, the best methodologies to evaluate LLMs must be explored [
The strengths of our study are the review panel and grading system. Our panel was a diverse team of medical professionals and students with sufficient knowledge, perspective, and interest in new technology to provide a fair but rigorous review of ChatGPT’s performance. Three reviewers had a specific interest in AI and one reviewer actually used the TI-30 while in junior high school. Our three-tiered grading system was clearly defined and should prove easy to use in future studies.
Several limitations must be considered. We did not assess other LLMs, such as Google’s Bard, that have recently become available to the public, nor did we assess the most current version of ChatGPT, Version 4. ChatGPT-3.5 was chosen because it was freely available at the time of this study and required no subscription fee. We acknowledge the performance improvements by newer versions of ChatGPT with “GPT-4” outperforming “GPT-3.5” on the SAT, scoring a 89th-percentile on the math section versus 70th-percentile, respectively [
Our evaluation of ChatGPT’s responses to commonly asked pregnancy questions confirms the concerns raised by others regarding both content and references. Since women currently use the internet as a source of information, it is likely that they will turn to AI to answer their pregnancy questions as well. While the debate continues regarding what AI is and how it is to be used, technological advances such as ChatGPT simply cannot be ignored. Subject expertise is required in order to evaluate ChatGPT output, and, therefore, we recommend that both professionals and laypersons approach this technology with caution.
The authors declare no conflicts of interest regarding the publication of this paper. The authors received no financial support for the research, authorship, and/or publication of this article.
Wan, C., Cadiente, A., Khromchenko, K., Friedricks, N., Rana, R.A. and Baum, J.D. (2023) ChatGPT: An Evaluation of AI-Generated Responses to Commonly Asked Pregnancy Questions. Open Journal of Obstetrics and Gynecology, 13, 1528-1546. https://doi.org/10.4236/ojog.2023.139129
Glossary of Terminology Relating to Artificial Intelligence