User login
The Internet is becoming a primary source for obtaining medical information. This growing trend may have serious implications for the medical field. As patients increasingly regard the Internet as an essential tool for obtaining health-related information, questions have been raised regarding the quality of medical information available on the Internet.1 Studies have shown that health-related sites often present inaccurate, inconsistent, and outdated information that may have a negative impact on health care decisions made by patients.2
According to the US Census Bureau, 71.7% of American households report having access to the Internet.3 Of those who have access to Internet, approximately 72% have sought health information online over the last year.4 Among people older than age 65 years living in the United States, there has been a growing trend toward using the Internet, from 14% in 2000 to almost 60% in 2013, according to the Pew Research Internet Project.5 Most medical websites are viewed for information on diseases and treatment options.6 Since most patients want to be informed about treatment options, as well as risks and benefits for each treatment, access to credible information is essential for proper decision-making.7
To assess the quality of information on the Internet, we used DISCERN, a standardized questionnaire to aid consumers in judging Internet content.8 The DISCERN instrument, available at www.discern.org.uk, was designed by an expert group in the United Kingdom. First, an expert panel developed and tested the instrument, and then health care providers and self-help group members tested it further.8,9 The questionnaire had been found to have good interrater reliability, regardless of use by health professionals or consumers.8-10
More than 53,000 shoulder arthroplasties are performed in the United States annually, and the number is growing, with the main goal of pain relief from glenohumeral degenerative joint disease.11,12 The Internet has become a quasi–second opinion for patients trying to participate in their care. Given the prevalence of shoulder-related surgeries, it is critical to analyze and become familiar with the quality of information that patients read online in order to direct them to nonbiased, all-inclusive websites. In this study, we provide a summary assessment and comparison of the quality of online information pertaining to shoulder replacement, using medical (total shoulder replacement) and nontechnical (shoulder replacement) search terms.
Methods
Websites were identified using 3 search engines (Google, Yahoo, and Bing) and 2 search terms, shoulder replacement (SR) and total shoulder arthroplasty (TSA), on January 17, 2014. These 3 search engines were used because 77% of health care–related information online searches begin through a search engine (Google, Bing, Yahoo); only 13% begin at a health care–specialized website.4 These search terms were used after consulting with orthopedic residents and attending physicians in a focus group regarding the terminology used with patients. The first 30 websites in each search engine were identified consecutively and evaluated for category and quality of information using the DISCERN instrument.
A total of 180 websites (90 per search term) were reviewed. Each website was evaluated independently by 3 medical students. In the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram, we recorded how websites were identified, screened, and included (Figure 1).13 Websites that were duplicated within each search term and those that were inaccessible were used to determine the total number of noncommercial versus commercial websites, but were excluded from the final analysis. The first part of the analysis involved determining the type of website (eg, commercial vs noncommercial) based upon the html endings. All .com endings were classified as commercial websites; noncommercial included .gov, .org, .edu, and .net endings. Next, each website was categorized based on the target audience. Websites were grouped into health professional–oriented information, patient-oriented, advertisement, or “other.” These classifications were based on those described in previous works.14,15 The “other” category included images, YouTube videos, another search engine, and open forums, which were also excluded from the final analysis because they were not easily evaluable with the DISCERN instrument. Websites were considered health professional–oriented if they included journal articles, scholarly articles, and/or rehabilitation protocols. Patient-directed websites clearly stated the information was directed to patients or provided a general overview. Advertisement included sites that displayed ads or products for sale. Websites were evaluated for quality using the DISCERN instrument (Figure 2).
DISCERN has 3 subdivision scores: the reliable score (composed of the first 8 questions), the treatment options (the next 7 questions), and 1 final question that addresses the overall quality of the website and is rated independently of the first 15 questions. DISCERN uses 2 scales, a binary scale anchored on both extremes with the number 1 equaling complete absence of the criteria being measured, and the number 5 at the upper extreme, representing completeness of the quality being assessed. In between 1 and 5 is a partial ordinal scale measuring from 2 to 4, which indicates the information is present to some extent but not complete. The ordinal scale allows ranking of the criteria being assessed. Summarizing values from each of the 2 scales poses some concern: the scale is not a true binary scale because of the ordinal scale of the middle numbers (2-4), and as such, is not amenable to being an interval scale to calculate arithmetic means. To summarize the values from the 2 scales, we calculated the harmonic mean, the arithmetic mean, the geometric mean, and the median. The means were empirically compared with the median, and we used the harmonic mean to summarize scale values because it was the best approximation of the medians.
Results
A total of 90 websites were assessed with the search term total shoulder arthroplasty and another 90 with shoulder replacement. When 37 duplicate websites for TSA and 52 for SR were eliminated, 53 (59%) and 38 (42%) unique websites were evaluated for each search term, respectively (Figure 1). (These unique websites are included in the Appendix.) Between the 2 search terms, 20 websites were duplicated. Figure 3 shows the distribution of websites by category. Total shoulder arthroplasty provided the highest percentage of health professional–oriented information; SR had the greatest percentage of patient-oriented information. Both TSA and SR had nearly the same number of advertisements and websites labeled “other.” The percentage of noncommercial websites from each search engine is represented in Figure 4. For SR, Google had 40% (12/30) noncommercial websites compared with Yahoo at 53% (16/30) and Bing at 46% (14/30). Total shoulder arthroplasty had 43% (13/30) noncommercial websites on Google, 27% (8/30) on Yahoo, and 40% (12/30) on Bing. In total, SR had more noncommercial websites, 47% (42/90), compared with 37% (33/90) for TSA.
The mean of all 3 raters for reliablity (DISCERN questions 1-8) and treatment options (DISCERN questions 9-15) is represented in the Table. For both search terms, we found that websites identified as health professional–oriented had the highest reliable mean scores, followed by patient-oriented, and advertisement at the lowest (SR: P = .054; TSA: P = .134). For SR, treatment mean scores demonstrated similar results with health professional–oriented websites receiving the highest, followed by patient-oriented and advertisement (P = .005). However, the treatment mean scores for TSA differed with patient-oriented websites receiving higher scores than health professional–oriented websites, but this was not statistically significant (P= .407). Regarding search terms, there were no significant differences between mean reliable and treatment scores across all categories.
The average overall DISCERN score for TSA websites was 2.5 (range, 1-5), compared with 2.3 (range, 1-5) for SR websites. The overall reliable score (DISCERN questions 1-8) for TSA websites was 2.6 and 2.5 for SR websites (P < .001). For TSA websites, 38% (20/53) were classified as good, having an overall DISCERN score ≥3, versus 26% (10/38) of SR websites. The overall DISCERN score for health professional–oriented websites was 2.7, patient-oriented websites received a score of 2.6, and advertisements had the lowest score at 2.4.
Discussion
Both patients and health professionals obtain information on health care subjects through the Internet, which has become the primary resource for patients.15,16 However, there are no strict regulations of the content being written. This creates a challenge for the typical user to find credible and evidence-based information, which is important because misleading information could cause undue anxiety, among other effects.17,18 The aims of this study were to determine the quality of Internet information for shoulder replacement surgeries using the medical terminology total shoulder arthroplasty (TSA) and the nontechnical term shoulder replacement (SR), and to compare the results.
After analyzing the types of websites returned for both total shoulder arthroplasty and shoulder replacement (Figure 4), it was interesting to find that using nonmedical terminology as the search term provided more noncommercial websites compared with total shoulder arthroplasty. Furthermore, Yahoo provided the highest yield of noncommercial websites at 16, with Bing at 14, when using SR as the search term. We believe the increase in noncommercial websites returned for SR was greater than for TSA because SR yielded more patient-oriented websites, which usually had html endings of .edu and .org, as shown in Figure 3 (48% of SR websites offered patient-oriented information).
Although there were more noncommercial websites for SR, the majority of the DISCERN values between the 2 search terms did not differ significantly. This is a direct result of the number of sites (20) that were duplicated across both search terms. However as seen in the Table, TSA had similar reliable mean scores for advertisements and patient-oriented websites but a slightly higher reliable score for health professional–oriented websites. We correlated this with the increased number of health professional–oriented websites returned when using TSA as the search term (Figure 3). The health professional–oriented websites explained their aims and cited their sources more consistently than did patient-oriented sites and advertisements, resulting in higher reliable scores. Although patient-oriented websites frequently lacked citations, they provided information about multiple treatment options, which were more relevant to consumers. This resulted in nearly equivalent reliable scores. Treatment means for advertisements in both SR and TSA were similar. However, treatment means for professional-oriented websites in TSA were lower than those for SR because health professional–oriented websites often were only moderately relevant to consumers, with their focus usually on 1 treatment option or on rehabilitation protocols. Although the DISCERN scores were similar between the search terms, total shoulder arthroplasty provided more websites (20) classified as good—overall DISCERN score, ≥3—than SR did (10). Advertisement websites had similar overall DISCERN scores, which we anticipated because most of the advertisements were duplicated across the search terms.
Using the 2 search terms, academic websites and commercial websites, such as WebMD, consistently received higher reliable and overall DISCERN scores. Advertisement websites, which need to deliver a clear message, frequently scored high on explicitly stating their aims and relevance to consumers, but focused on their products without discussing the benefits of other treatment options. This is significant because Internet search engines, such as Google, offer sponsor links for which organizations pay to appear at the top of the search results. This creates the potential for consumers to receive biased information because most individuals only visit the top 10 websites generated by a search engine.19
We concluded that the quality of online information relating to SR and TSA was highly variable and frequently of moderate-to-poor quality, with most overall DISCERN scores <3. The quality of information found online for this study using the DISCERN instrument is consistent with those studies using DISCERN to evaluate other medical conditions (eg, bunions, chronic pain, general anesthesia, and anterior cruciate ligament reconstruction).2,9,15,19 These studies also concluded that online information varies tremendously in quality and completeness.
This study has several limitations. Websites were searched at a single time point and, because Internet resources are frequently updated, the results of this study could vary. Furthermore, although Google, Yahoo, and Bing are 3 of the most popular search engines, these are not the only resources patients use when searching the Internet for health-related information. Other search engines, such as Pubmed.gov and MSN.com, could provide additional websites for Internet users. Lastly, although DISCERN is validated to address the quality of information available online, it does not evaluate the accuracy of the information.8 Our use of DISCERN involves 2 scales, a binary yes/no (ratings, 1 and 5) and an ordinal scale (ratings, 2-4). As such, a single mean summary statistic cannot be calculated.
Conclusion
The information available on the Internet pertaining to TSA and SR is highly variable and provides mostly moderate-to-poor quality information based on the DISCERN instrument. Many websites failed to describe the benefits and the risks of different treatment options, including nonoperative management. Health care professionals should be aware that patients often refer to the Internet as a primary resource for obtaining medical information. It is important to direct patients to websites that provide accurate information, because patients who educate themselves about their conditions and actively participate in decision-making may have improved health outcomes.20-22 Overall, academic websites and commercial websites, such as WebMD and OrthoInfo, generally had higher DISCERN scores when using either search term. Of major concern is the potential for misleading advertisements or incorrect information that can negatively affect health outcomes. This study found that using nonmedical terminology (SR) provided more noncommercial and patient-oriented websites, especially through Yahoo. This study highlights the need for more comprehensive online information pertaining to shoulder replacement that can better serve as a resource for Internet users.
1. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287(20):2691-2700.
2. Bruce-Brand RA, Baker JF, Byrne DP, Hogan NA, McCarthy T. Assessment of the quality and content of information on anterior cruciate ligament reconstruction on the internet. Arthroscopy. 2013;29(6):1095-1100.
3. Computer and internet use in the United States: population characteristics. US Census Bureau website. http://www.census.gov/hhes/computer/. Accessed December 11, 2015.
4. Fox S, Duggan M. Health online 2013. Pew Research Center website. http://pewinternet.org/Reports/2013/Health-online.aspx. Published January 15, 2013. Accessed November 24, 2015.
5. Smith A. Older adults and technology use. Pew Research Center website. http://www.pewinternet.org/2014/04/03/older-adults-and-technology-use. Published April 3, 2014. Accessed November 24, 2015.
6. Shuyler KS, Knight KM. What are patients seeking when they turn to the internet? Qualitative content analysis of questions asked by visitors to an orthopaedics web site. J Med Internet Res. 2003;5(4):e24.
7. Meredith P, Emberton M, Wood C, Smith J. Comparison of patients’ needs for information on prostate surgery with printed materials provided by surgeons. Qual Health Care. 1995;4(1):18-23.
8. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105-111.
9. Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med. 2010;8(1):59.
10. Griffiths KM, Christensen H. Website quality indicators for consumers. J Med Internet Res. 2005;7(5):e55.
11. Wiater JM. Shoulder joint replacement. American Academy of Orthopedic Surgeons website. http://orthoinfo.aaos.org/topic.cfm?topic=A00094. Updated December 2011. Accessed November 24, 2015.
12. Kim SH, Wise BL, Zhang Y, Szabo RM. Increasing incidence of shoulder arthroplasty in the united states. J Bone Joint Surg Am. 2011;93(24):2249-2254.
13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151(4):W65-W94.
14. Nason GJ, Baker JF, Byrne DP, Noel J, Moore D, Kiely PJ. Scoliosis-specific information on the internet: has the “information highway” led to better information provision? Spine. 2012;37(21):E1364-E1369.
15. Starman JS, Gettys FK, Capo JA, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg Am. 2010;92(7):1612-1618.
16. Bernstein J, Ahn J, Veillette C. The future of orthopaedic information management. J Bone Joint Surg Am. 2012;94(13):e95.
17. Berland GK, Elliott MN, Morales LS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;285(20):2612-2621.
18. Fallowfield LJ, Hall A, Maguire GP, Baum M. Psychological outcomes of different treatment policies in women with early breast cancer outside a clinical trial. BMJ. 1990;301(6752):575-580.
19. Chong YM, Fraval A, Chandrananth J, Plunkett V, Tran P. Assessment of the quality of web-based information on bunions. Foot Ankle Int. 2013;34(8):1134-1139.
20. Brody DS, Miller SM, Lerman CE, Smith DG, Caputo GC. Patient perception of involvement in medical care. J Gen Intern Med. 1989;4(6):506-511.
21. Greenfield S, Kaplan S, Ware JE Jr. Expanding patient involvement in care. Effects on patient outcomes. Ann Intern Med. 1985;102(4):520-528.
22. Kaplan SH, Greenfield S, Ware JE Jr. Assessing the effects of physician-patient interactions on the outcomes of chronic disease. Med Care. 1989;27(3 suppl):S110-S127.
The Internet is becoming a primary source for obtaining medical information. This growing trend may have serious implications for the medical field. As patients increasingly regard the Internet as an essential tool for obtaining health-related information, questions have been raised regarding the quality of medical information available on the Internet.1 Studies have shown that health-related sites often present inaccurate, inconsistent, and outdated information that may have a negative impact on health care decisions made by patients.2
According to the US Census Bureau, 71.7% of American households report having access to the Internet.3 Of those who have access to Internet, approximately 72% have sought health information online over the last year.4 Among people older than age 65 years living in the United States, there has been a growing trend toward using the Internet, from 14% in 2000 to almost 60% in 2013, according to the Pew Research Internet Project.5 Most medical websites are viewed for information on diseases and treatment options.6 Since most patients want to be informed about treatment options, as well as risks and benefits for each treatment, access to credible information is essential for proper decision-making.7
To assess the quality of information on the Internet, we used DISCERN, a standardized questionnaire to aid consumers in judging Internet content.8 The DISCERN instrument, available at www.discern.org.uk, was designed by an expert group in the United Kingdom. First, an expert panel developed and tested the instrument, and then health care providers and self-help group members tested it further.8,9 The questionnaire had been found to have good interrater reliability, regardless of use by health professionals or consumers.8-10
More than 53,000 shoulder arthroplasties are performed in the United States annually, and the number is growing, with the main goal of pain relief from glenohumeral degenerative joint disease.11,12 The Internet has become a quasi–second opinion for patients trying to participate in their care. Given the prevalence of shoulder-related surgeries, it is critical to analyze and become familiar with the quality of information that patients read online in order to direct them to nonbiased, all-inclusive websites. In this study, we provide a summary assessment and comparison of the quality of online information pertaining to shoulder replacement, using medical (total shoulder replacement) and nontechnical (shoulder replacement) search terms.
Methods
Websites were identified using 3 search engines (Google, Yahoo, and Bing) and 2 search terms, shoulder replacement (SR) and total shoulder arthroplasty (TSA), on January 17, 2014. These 3 search engines were used because 77% of health care–related information online searches begin through a search engine (Google, Bing, Yahoo); only 13% begin at a health care–specialized website.4 These search terms were used after consulting with orthopedic residents and attending physicians in a focus group regarding the terminology used with patients. The first 30 websites in each search engine were identified consecutively and evaluated for category and quality of information using the DISCERN instrument.
A total of 180 websites (90 per search term) were reviewed. Each website was evaluated independently by 3 medical students. In the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram, we recorded how websites were identified, screened, and included (Figure 1).13 Websites that were duplicated within each search term and those that were inaccessible were used to determine the total number of noncommercial versus commercial websites, but were excluded from the final analysis. The first part of the analysis involved determining the type of website (eg, commercial vs noncommercial) based upon the html endings. All .com endings were classified as commercial websites; noncommercial included .gov, .org, .edu, and .net endings. Next, each website was categorized based on the target audience. Websites were grouped into health professional–oriented information, patient-oriented, advertisement, or “other.” These classifications were based on those described in previous works.14,15 The “other” category included images, YouTube videos, another search engine, and open forums, which were also excluded from the final analysis because they were not easily evaluable with the DISCERN instrument. Websites were considered health professional–oriented if they included journal articles, scholarly articles, and/or rehabilitation protocols. Patient-directed websites clearly stated the information was directed to patients or provided a general overview. Advertisement included sites that displayed ads or products for sale. Websites were evaluated for quality using the DISCERN instrument (Figure 2).
DISCERN has 3 subdivision scores: the reliable score (composed of the first 8 questions), the treatment options (the next 7 questions), and 1 final question that addresses the overall quality of the website and is rated independently of the first 15 questions. DISCERN uses 2 scales, a binary scale anchored on both extremes with the number 1 equaling complete absence of the criteria being measured, and the number 5 at the upper extreme, representing completeness of the quality being assessed. In between 1 and 5 is a partial ordinal scale measuring from 2 to 4, which indicates the information is present to some extent but not complete. The ordinal scale allows ranking of the criteria being assessed. Summarizing values from each of the 2 scales poses some concern: the scale is not a true binary scale because of the ordinal scale of the middle numbers (2-4), and as such, is not amenable to being an interval scale to calculate arithmetic means. To summarize the values from the 2 scales, we calculated the harmonic mean, the arithmetic mean, the geometric mean, and the median. The means were empirically compared with the median, and we used the harmonic mean to summarize scale values because it was the best approximation of the medians.
Results
A total of 90 websites were assessed with the search term total shoulder arthroplasty and another 90 with shoulder replacement. When 37 duplicate websites for TSA and 52 for SR were eliminated, 53 (59%) and 38 (42%) unique websites were evaluated for each search term, respectively (Figure 1). (These unique websites are included in the Appendix.) Between the 2 search terms, 20 websites were duplicated. Figure 3 shows the distribution of websites by category. Total shoulder arthroplasty provided the highest percentage of health professional–oriented information; SR had the greatest percentage of patient-oriented information. Both TSA and SR had nearly the same number of advertisements and websites labeled “other.” The percentage of noncommercial websites from each search engine is represented in Figure 4. For SR, Google had 40% (12/30) noncommercial websites compared with Yahoo at 53% (16/30) and Bing at 46% (14/30). Total shoulder arthroplasty had 43% (13/30) noncommercial websites on Google, 27% (8/30) on Yahoo, and 40% (12/30) on Bing. In total, SR had more noncommercial websites, 47% (42/90), compared with 37% (33/90) for TSA.
The mean of all 3 raters for reliablity (DISCERN questions 1-8) and treatment options (DISCERN questions 9-15) is represented in the Table. For both search terms, we found that websites identified as health professional–oriented had the highest reliable mean scores, followed by patient-oriented, and advertisement at the lowest (SR: P = .054; TSA: P = .134). For SR, treatment mean scores demonstrated similar results with health professional–oriented websites receiving the highest, followed by patient-oriented and advertisement (P = .005). However, the treatment mean scores for TSA differed with patient-oriented websites receiving higher scores than health professional–oriented websites, but this was not statistically significant (P= .407). Regarding search terms, there were no significant differences between mean reliable and treatment scores across all categories.
The average overall DISCERN score for TSA websites was 2.5 (range, 1-5), compared with 2.3 (range, 1-5) for SR websites. The overall reliable score (DISCERN questions 1-8) for TSA websites was 2.6 and 2.5 for SR websites (P < .001). For TSA websites, 38% (20/53) were classified as good, having an overall DISCERN score ≥3, versus 26% (10/38) of SR websites. The overall DISCERN score for health professional–oriented websites was 2.7, patient-oriented websites received a score of 2.6, and advertisements had the lowest score at 2.4.
Discussion
Both patients and health professionals obtain information on health care subjects through the Internet, which has become the primary resource for patients.15,16 However, there are no strict regulations of the content being written. This creates a challenge for the typical user to find credible and evidence-based information, which is important because misleading information could cause undue anxiety, among other effects.17,18 The aims of this study were to determine the quality of Internet information for shoulder replacement surgeries using the medical terminology total shoulder arthroplasty (TSA) and the nontechnical term shoulder replacement (SR), and to compare the results.
After analyzing the types of websites returned for both total shoulder arthroplasty and shoulder replacement (Figure 4), it was interesting to find that using nonmedical terminology as the search term provided more noncommercial websites compared with total shoulder arthroplasty. Furthermore, Yahoo provided the highest yield of noncommercial websites at 16, with Bing at 14, when using SR as the search term. We believe the increase in noncommercial websites returned for SR was greater than for TSA because SR yielded more patient-oriented websites, which usually had html endings of .edu and .org, as shown in Figure 3 (48% of SR websites offered patient-oriented information).
Although there were more noncommercial websites for SR, the majority of the DISCERN values between the 2 search terms did not differ significantly. This is a direct result of the number of sites (20) that were duplicated across both search terms. However as seen in the Table, TSA had similar reliable mean scores for advertisements and patient-oriented websites but a slightly higher reliable score for health professional–oriented websites. We correlated this with the increased number of health professional–oriented websites returned when using TSA as the search term (Figure 3). The health professional–oriented websites explained their aims and cited their sources more consistently than did patient-oriented sites and advertisements, resulting in higher reliable scores. Although patient-oriented websites frequently lacked citations, they provided information about multiple treatment options, which were more relevant to consumers. This resulted in nearly equivalent reliable scores. Treatment means for advertisements in both SR and TSA were similar. However, treatment means for professional-oriented websites in TSA were lower than those for SR because health professional–oriented websites often were only moderately relevant to consumers, with their focus usually on 1 treatment option or on rehabilitation protocols. Although the DISCERN scores were similar between the search terms, total shoulder arthroplasty provided more websites (20) classified as good—overall DISCERN score, ≥3—than SR did (10). Advertisement websites had similar overall DISCERN scores, which we anticipated because most of the advertisements were duplicated across the search terms.
Using the 2 search terms, academic websites and commercial websites, such as WebMD, consistently received higher reliable and overall DISCERN scores. Advertisement websites, which need to deliver a clear message, frequently scored high on explicitly stating their aims and relevance to consumers, but focused on their products without discussing the benefits of other treatment options. This is significant because Internet search engines, such as Google, offer sponsor links for which organizations pay to appear at the top of the search results. This creates the potential for consumers to receive biased information because most individuals only visit the top 10 websites generated by a search engine.19
We concluded that the quality of online information relating to SR and TSA was highly variable and frequently of moderate-to-poor quality, with most overall DISCERN scores <3. The quality of information found online for this study using the DISCERN instrument is consistent with those studies using DISCERN to evaluate other medical conditions (eg, bunions, chronic pain, general anesthesia, and anterior cruciate ligament reconstruction).2,9,15,19 These studies also concluded that online information varies tremendously in quality and completeness.
This study has several limitations. Websites were searched at a single time point and, because Internet resources are frequently updated, the results of this study could vary. Furthermore, although Google, Yahoo, and Bing are 3 of the most popular search engines, these are not the only resources patients use when searching the Internet for health-related information. Other search engines, such as Pubmed.gov and MSN.com, could provide additional websites for Internet users. Lastly, although DISCERN is validated to address the quality of information available online, it does not evaluate the accuracy of the information.8 Our use of DISCERN involves 2 scales, a binary yes/no (ratings, 1 and 5) and an ordinal scale (ratings, 2-4). As such, a single mean summary statistic cannot be calculated.
Conclusion
The information available on the Internet pertaining to TSA and SR is highly variable and provides mostly moderate-to-poor quality information based on the DISCERN instrument. Many websites failed to describe the benefits and the risks of different treatment options, including nonoperative management. Health care professionals should be aware that patients often refer to the Internet as a primary resource for obtaining medical information. It is important to direct patients to websites that provide accurate information, because patients who educate themselves about their conditions and actively participate in decision-making may have improved health outcomes.20-22 Overall, academic websites and commercial websites, such as WebMD and OrthoInfo, generally had higher DISCERN scores when using either search term. Of major concern is the potential for misleading advertisements or incorrect information that can negatively affect health outcomes. This study found that using nonmedical terminology (SR) provided more noncommercial and patient-oriented websites, especially through Yahoo. This study highlights the need for more comprehensive online information pertaining to shoulder replacement that can better serve as a resource for Internet users.
The Internet is becoming a primary source for obtaining medical information. This growing trend may have serious implications for the medical field. As patients increasingly regard the Internet as an essential tool for obtaining health-related information, questions have been raised regarding the quality of medical information available on the Internet.1 Studies have shown that health-related sites often present inaccurate, inconsistent, and outdated information that may have a negative impact on health care decisions made by patients.2
According to the US Census Bureau, 71.7% of American households report having access to the Internet.3 Of those who have access to Internet, approximately 72% have sought health information online over the last year.4 Among people older than age 65 years living in the United States, there has been a growing trend toward using the Internet, from 14% in 2000 to almost 60% in 2013, according to the Pew Research Internet Project.5 Most medical websites are viewed for information on diseases and treatment options.6 Since most patients want to be informed about treatment options, as well as risks and benefits for each treatment, access to credible information is essential for proper decision-making.7
To assess the quality of information on the Internet, we used DISCERN, a standardized questionnaire to aid consumers in judging Internet content.8 The DISCERN instrument, available at www.discern.org.uk, was designed by an expert group in the United Kingdom. First, an expert panel developed and tested the instrument, and then health care providers and self-help group members tested it further.8,9 The questionnaire had been found to have good interrater reliability, regardless of use by health professionals or consumers.8-10
More than 53,000 shoulder arthroplasties are performed in the United States annually, and the number is growing, with the main goal of pain relief from glenohumeral degenerative joint disease.11,12 The Internet has become a quasi–second opinion for patients trying to participate in their care. Given the prevalence of shoulder-related surgeries, it is critical to analyze and become familiar with the quality of information that patients read online in order to direct them to nonbiased, all-inclusive websites. In this study, we provide a summary assessment and comparison of the quality of online information pertaining to shoulder replacement, using medical (total shoulder replacement) and nontechnical (shoulder replacement) search terms.
Methods
Websites were identified using 3 search engines (Google, Yahoo, and Bing) and 2 search terms, shoulder replacement (SR) and total shoulder arthroplasty (TSA), on January 17, 2014. These 3 search engines were used because 77% of health care–related information online searches begin through a search engine (Google, Bing, Yahoo); only 13% begin at a health care–specialized website.4 These search terms were used after consulting with orthopedic residents and attending physicians in a focus group regarding the terminology used with patients. The first 30 websites in each search engine were identified consecutively and evaluated for category and quality of information using the DISCERN instrument.
A total of 180 websites (90 per search term) were reviewed. Each website was evaluated independently by 3 medical students. In the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram, we recorded how websites were identified, screened, and included (Figure 1).13 Websites that were duplicated within each search term and those that were inaccessible were used to determine the total number of noncommercial versus commercial websites, but were excluded from the final analysis. The first part of the analysis involved determining the type of website (eg, commercial vs noncommercial) based upon the html endings. All .com endings were classified as commercial websites; noncommercial included .gov, .org, .edu, and .net endings. Next, each website was categorized based on the target audience. Websites were grouped into health professional–oriented information, patient-oriented, advertisement, or “other.” These classifications were based on those described in previous works.14,15 The “other” category included images, YouTube videos, another search engine, and open forums, which were also excluded from the final analysis because they were not easily evaluable with the DISCERN instrument. Websites were considered health professional–oriented if they included journal articles, scholarly articles, and/or rehabilitation protocols. Patient-directed websites clearly stated the information was directed to patients or provided a general overview. Advertisement included sites that displayed ads or products for sale. Websites were evaluated for quality using the DISCERN instrument (Figure 2).
DISCERN has 3 subdivision scores: the reliable score (composed of the first 8 questions), the treatment options (the next 7 questions), and 1 final question that addresses the overall quality of the website and is rated independently of the first 15 questions. DISCERN uses 2 scales, a binary scale anchored on both extremes with the number 1 equaling complete absence of the criteria being measured, and the number 5 at the upper extreme, representing completeness of the quality being assessed. In between 1 and 5 is a partial ordinal scale measuring from 2 to 4, which indicates the information is present to some extent but not complete. The ordinal scale allows ranking of the criteria being assessed. Summarizing values from each of the 2 scales poses some concern: the scale is not a true binary scale because of the ordinal scale of the middle numbers (2-4), and as such, is not amenable to being an interval scale to calculate arithmetic means. To summarize the values from the 2 scales, we calculated the harmonic mean, the arithmetic mean, the geometric mean, and the median. The means were empirically compared with the median, and we used the harmonic mean to summarize scale values because it was the best approximation of the medians.
Results
A total of 90 websites were assessed with the search term total shoulder arthroplasty and another 90 with shoulder replacement. When 37 duplicate websites for TSA and 52 for SR were eliminated, 53 (59%) and 38 (42%) unique websites were evaluated for each search term, respectively (Figure 1). (These unique websites are included in the Appendix.) Between the 2 search terms, 20 websites were duplicated. Figure 3 shows the distribution of websites by category. Total shoulder arthroplasty provided the highest percentage of health professional–oriented information; SR had the greatest percentage of patient-oriented information. Both TSA and SR had nearly the same number of advertisements and websites labeled “other.” The percentage of noncommercial websites from each search engine is represented in Figure 4. For SR, Google had 40% (12/30) noncommercial websites compared with Yahoo at 53% (16/30) and Bing at 46% (14/30). Total shoulder arthroplasty had 43% (13/30) noncommercial websites on Google, 27% (8/30) on Yahoo, and 40% (12/30) on Bing. In total, SR had more noncommercial websites, 47% (42/90), compared with 37% (33/90) for TSA.
The mean of all 3 raters for reliablity (DISCERN questions 1-8) and treatment options (DISCERN questions 9-15) is represented in the Table. For both search terms, we found that websites identified as health professional–oriented had the highest reliable mean scores, followed by patient-oriented, and advertisement at the lowest (SR: P = .054; TSA: P = .134). For SR, treatment mean scores demonstrated similar results with health professional–oriented websites receiving the highest, followed by patient-oriented and advertisement (P = .005). However, the treatment mean scores for TSA differed with patient-oriented websites receiving higher scores than health professional–oriented websites, but this was not statistically significant (P= .407). Regarding search terms, there were no significant differences between mean reliable and treatment scores across all categories.
The average overall DISCERN score for TSA websites was 2.5 (range, 1-5), compared with 2.3 (range, 1-5) for SR websites. The overall reliable score (DISCERN questions 1-8) for TSA websites was 2.6 and 2.5 for SR websites (P < .001). For TSA websites, 38% (20/53) were classified as good, having an overall DISCERN score ≥3, versus 26% (10/38) of SR websites. The overall DISCERN score for health professional–oriented websites was 2.7, patient-oriented websites received a score of 2.6, and advertisements had the lowest score at 2.4.
Discussion
Both patients and health professionals obtain information on health care subjects through the Internet, which has become the primary resource for patients.15,16 However, there are no strict regulations of the content being written. This creates a challenge for the typical user to find credible and evidence-based information, which is important because misleading information could cause undue anxiety, among other effects.17,18 The aims of this study were to determine the quality of Internet information for shoulder replacement surgeries using the medical terminology total shoulder arthroplasty (TSA) and the nontechnical term shoulder replacement (SR), and to compare the results.
After analyzing the types of websites returned for both total shoulder arthroplasty and shoulder replacement (Figure 4), it was interesting to find that using nonmedical terminology as the search term provided more noncommercial websites compared with total shoulder arthroplasty. Furthermore, Yahoo provided the highest yield of noncommercial websites at 16, with Bing at 14, when using SR as the search term. We believe the increase in noncommercial websites returned for SR was greater than for TSA because SR yielded more patient-oriented websites, which usually had html endings of .edu and .org, as shown in Figure 3 (48% of SR websites offered patient-oriented information).
Although there were more noncommercial websites for SR, the majority of the DISCERN values between the 2 search terms did not differ significantly. This is a direct result of the number of sites (20) that were duplicated across both search terms. However as seen in the Table, TSA had similar reliable mean scores for advertisements and patient-oriented websites but a slightly higher reliable score for health professional–oriented websites. We correlated this with the increased number of health professional–oriented websites returned when using TSA as the search term (Figure 3). The health professional–oriented websites explained their aims and cited their sources more consistently than did patient-oriented sites and advertisements, resulting in higher reliable scores. Although patient-oriented websites frequently lacked citations, they provided information about multiple treatment options, which were more relevant to consumers. This resulted in nearly equivalent reliable scores. Treatment means for advertisements in both SR and TSA were similar. However, treatment means for professional-oriented websites in TSA were lower than those for SR because health professional–oriented websites often were only moderately relevant to consumers, with their focus usually on 1 treatment option or on rehabilitation protocols. Although the DISCERN scores were similar between the search terms, total shoulder arthroplasty provided more websites (20) classified as good—overall DISCERN score, ≥3—than SR did (10). Advertisement websites had similar overall DISCERN scores, which we anticipated because most of the advertisements were duplicated across the search terms.
Using the 2 search terms, academic websites and commercial websites, such as WebMD, consistently received higher reliable and overall DISCERN scores. Advertisement websites, which need to deliver a clear message, frequently scored high on explicitly stating their aims and relevance to consumers, but focused on their products without discussing the benefits of other treatment options. This is significant because Internet search engines, such as Google, offer sponsor links for which organizations pay to appear at the top of the search results. This creates the potential for consumers to receive biased information because most individuals only visit the top 10 websites generated by a search engine.19
We concluded that the quality of online information relating to SR and TSA was highly variable and frequently of moderate-to-poor quality, with most overall DISCERN scores <3. The quality of information found online for this study using the DISCERN instrument is consistent with those studies using DISCERN to evaluate other medical conditions (eg, bunions, chronic pain, general anesthesia, and anterior cruciate ligament reconstruction).2,9,15,19 These studies also concluded that online information varies tremendously in quality and completeness.
This study has several limitations. Websites were searched at a single time point and, because Internet resources are frequently updated, the results of this study could vary. Furthermore, although Google, Yahoo, and Bing are 3 of the most popular search engines, these are not the only resources patients use when searching the Internet for health-related information. Other search engines, such as Pubmed.gov and MSN.com, could provide additional websites for Internet users. Lastly, although DISCERN is validated to address the quality of information available online, it does not evaluate the accuracy of the information.8 Our use of DISCERN involves 2 scales, a binary yes/no (ratings, 1 and 5) and an ordinal scale (ratings, 2-4). As such, a single mean summary statistic cannot be calculated.
Conclusion
The information available on the Internet pertaining to TSA and SR is highly variable and provides mostly moderate-to-poor quality information based on the DISCERN instrument. Many websites failed to describe the benefits and the risks of different treatment options, including nonoperative management. Health care professionals should be aware that patients often refer to the Internet as a primary resource for obtaining medical information. It is important to direct patients to websites that provide accurate information, because patients who educate themselves about their conditions and actively participate in decision-making may have improved health outcomes.20-22 Overall, academic websites and commercial websites, such as WebMD and OrthoInfo, generally had higher DISCERN scores when using either search term. Of major concern is the potential for misleading advertisements or incorrect information that can negatively affect health outcomes. This study found that using nonmedical terminology (SR) provided more noncommercial and patient-oriented websites, especially through Yahoo. This study highlights the need for more comprehensive online information pertaining to shoulder replacement that can better serve as a resource for Internet users.
1. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287(20):2691-2700.
2. Bruce-Brand RA, Baker JF, Byrne DP, Hogan NA, McCarthy T. Assessment of the quality and content of information on anterior cruciate ligament reconstruction on the internet. Arthroscopy. 2013;29(6):1095-1100.
3. Computer and internet use in the United States: population characteristics. US Census Bureau website. http://www.census.gov/hhes/computer/. Accessed December 11, 2015.
4. Fox S, Duggan M. Health online 2013. Pew Research Center website. http://pewinternet.org/Reports/2013/Health-online.aspx. Published January 15, 2013. Accessed November 24, 2015.
5. Smith A. Older adults and technology use. Pew Research Center website. http://www.pewinternet.org/2014/04/03/older-adults-and-technology-use. Published April 3, 2014. Accessed November 24, 2015.
6. Shuyler KS, Knight KM. What are patients seeking when they turn to the internet? Qualitative content analysis of questions asked by visitors to an orthopaedics web site. J Med Internet Res. 2003;5(4):e24.
7. Meredith P, Emberton M, Wood C, Smith J. Comparison of patients’ needs for information on prostate surgery with printed materials provided by surgeons. Qual Health Care. 1995;4(1):18-23.
8. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105-111.
9. Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med. 2010;8(1):59.
10. Griffiths KM, Christensen H. Website quality indicators for consumers. J Med Internet Res. 2005;7(5):e55.
11. Wiater JM. Shoulder joint replacement. American Academy of Orthopedic Surgeons website. http://orthoinfo.aaos.org/topic.cfm?topic=A00094. Updated December 2011. Accessed November 24, 2015.
12. Kim SH, Wise BL, Zhang Y, Szabo RM. Increasing incidence of shoulder arthroplasty in the united states. J Bone Joint Surg Am. 2011;93(24):2249-2254.
13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151(4):W65-W94.
14. Nason GJ, Baker JF, Byrne DP, Noel J, Moore D, Kiely PJ. Scoliosis-specific information on the internet: has the “information highway” led to better information provision? Spine. 2012;37(21):E1364-E1369.
15. Starman JS, Gettys FK, Capo JA, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg Am. 2010;92(7):1612-1618.
16. Bernstein J, Ahn J, Veillette C. The future of orthopaedic information management. J Bone Joint Surg Am. 2012;94(13):e95.
17. Berland GK, Elliott MN, Morales LS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;285(20):2612-2621.
18. Fallowfield LJ, Hall A, Maguire GP, Baum M. Psychological outcomes of different treatment policies in women with early breast cancer outside a clinical trial. BMJ. 1990;301(6752):575-580.
19. Chong YM, Fraval A, Chandrananth J, Plunkett V, Tran P. Assessment of the quality of web-based information on bunions. Foot Ankle Int. 2013;34(8):1134-1139.
20. Brody DS, Miller SM, Lerman CE, Smith DG, Caputo GC. Patient perception of involvement in medical care. J Gen Intern Med. 1989;4(6):506-511.
21. Greenfield S, Kaplan S, Ware JE Jr. Expanding patient involvement in care. Effects on patient outcomes. Ann Intern Med. 1985;102(4):520-528.
22. Kaplan SH, Greenfield S, Ware JE Jr. Assessing the effects of physician-patient interactions on the outcomes of chronic disease. Med Care. 1989;27(3 suppl):S110-S127.
1. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287(20):2691-2700.
2. Bruce-Brand RA, Baker JF, Byrne DP, Hogan NA, McCarthy T. Assessment of the quality and content of information on anterior cruciate ligament reconstruction on the internet. Arthroscopy. 2013;29(6):1095-1100.
3. Computer and internet use in the United States: population characteristics. US Census Bureau website. http://www.census.gov/hhes/computer/. Accessed December 11, 2015.
4. Fox S, Duggan M. Health online 2013. Pew Research Center website. http://pewinternet.org/Reports/2013/Health-online.aspx. Published January 15, 2013. Accessed November 24, 2015.
5. Smith A. Older adults and technology use. Pew Research Center website. http://www.pewinternet.org/2014/04/03/older-adults-and-technology-use. Published April 3, 2014. Accessed November 24, 2015.
6. Shuyler KS, Knight KM. What are patients seeking when they turn to the internet? Qualitative content analysis of questions asked by visitors to an orthopaedics web site. J Med Internet Res. 2003;5(4):e24.
7. Meredith P, Emberton M, Wood C, Smith J. Comparison of patients’ needs for information on prostate surgery with printed materials provided by surgeons. Qual Health Care. 1995;4(1):18-23.
8. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105-111.
9. Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med. 2010;8(1):59.
10. Griffiths KM, Christensen H. Website quality indicators for consumers. J Med Internet Res. 2005;7(5):e55.
11. Wiater JM. Shoulder joint replacement. American Academy of Orthopedic Surgeons website. http://orthoinfo.aaos.org/topic.cfm?topic=A00094. Updated December 2011. Accessed November 24, 2015.
12. Kim SH, Wise BL, Zhang Y, Szabo RM. Increasing incidence of shoulder arthroplasty in the united states. J Bone Joint Surg Am. 2011;93(24):2249-2254.
13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151(4):W65-W94.
14. Nason GJ, Baker JF, Byrne DP, Noel J, Moore D, Kiely PJ. Scoliosis-specific information on the internet: has the “information highway” led to better information provision? Spine. 2012;37(21):E1364-E1369.
15. Starman JS, Gettys FK, Capo JA, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg Am. 2010;92(7):1612-1618.
16. Bernstein J, Ahn J, Veillette C. The future of orthopaedic information management. J Bone Joint Surg Am. 2012;94(13):e95.
17. Berland GK, Elliott MN, Morales LS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;285(20):2612-2621.
18. Fallowfield LJ, Hall A, Maguire GP, Baum M. Psychological outcomes of different treatment policies in women with early breast cancer outside a clinical trial. BMJ. 1990;301(6752):575-580.
19. Chong YM, Fraval A, Chandrananth J, Plunkett V, Tran P. Assessment of the quality of web-based information on bunions. Foot Ankle Int. 2013;34(8):1134-1139.
20. Brody DS, Miller SM, Lerman CE, Smith DG, Caputo GC. Patient perception of involvement in medical care. J Gen Intern Med. 1989;4(6):506-511.
21. Greenfield S, Kaplan S, Ware JE Jr. Expanding patient involvement in care. Effects on patient outcomes. Ann Intern Med. 1985;102(4):520-528.
22. Kaplan SH, Greenfield S, Ware JE Jr. Assessing the effects of physician-patient interactions on the outcomes of chronic disease. Med Care. 1989;27(3 suppl):S110-S127.
