research articles on electronic health record

• Open access
• Published: 27 October 2021

A narrative review on the validity of electronic health record-based research in epidemiology

• Milena A. Gianfrancesco 1 &
• Neal D. Goldstein   ORCID: orcid.org/0000-0002-9597-5251 2  

BMC Medical Research Methodology volume  21 , Article number:  234 ( 2021 ) Cite this article

11k Accesses

46 Citations

5 Altmetric

Metrics details

Electronic health records (EHRs) are widely used in epidemiological research, but the validity of the results is dependent upon the assumptions made about the healthcare system, the patient, and the provider. In this review, we identify four overarching challenges in using EHR-based data for epidemiological analysis, with a particular emphasis on threats to validity. These challenges include representativeness of the EHR to a target population, the availability and interpretability of clinical and non-clinical data, and missing data at both the variable and observation levels. Each challenge reveals layers of assumptions that the epidemiologist is required to make, from the point of patient entry into the healthcare system, to the provider documenting the results of the clinical exam and follow-up of the patient longitudinally; all with the potential to bias the results of analysis of these data. Understanding the extent of as well as remediating potential biases requires a variety of methodological approaches, from traditional sensitivity analyses and validation studies, to newer techniques such as natural language processing. Beyond methods to address these challenges, it will remain crucial for epidemiologists to engage with clinicians and informaticians at their institutions to ensure data quality and accessibility by forming multidisciplinary teams around specific research projects.

Peer Review reports

The proliferation of electronic health records (EHRs) spurred on by federal government incentives over the past few decades has resulted in greater than an 80% adoption-rate at hospitals [ 1 ] and close to 90% in office-based practices [ 2 ] in the United States. A natural consequence of the availability of electronic health data is the conduct of research with these data, both observational and experimental [ 3 ], due to lower overhead costs and lower burden of study recruitment [ 4 ]. Indeed, a search on PubMed for publications indexed by the MeSH term “electronic health records” reveals an exponential growth in biomedical literature, especially over the last 10 years with an excess of 50,000 publications.

An emerging literature is beginning to recognize the many challenges that still lay ahead in using EHR data for epidemiological investigations. Researchers in Europe identified 13 potential sources of “bias” (bias was defined as a contamination of the data) in EHR-based data covering almost every aspect of care delivery, from selective entrance into the healthcare system, to variation in care and documentation practices, to identification and extraction of the right data for analysis [ 5 ]. Many of the identified contaminants are directly relevant to traditional epidemiological threats to validity [ 4 ]. Data quality has consistently been invoked as a central challenge in EHRs. From a qualitative perspective, healthcare workers have described challenges in the healthcare environment (e.g., heavy workload), imperfect clinical documentation practices, and concerns over data extraction and reporting tools, all of which would impact the quality of data in the EHR [ 6 ]. From a quantitative perspective, researchers have noted limited sensitivity of diagnostic codes in the EHR when relying on discrete codings, noting that upon a manual chart review free text fields often capture the missed information, motivating such techniques as natural language processing (NLP) [ 7 ]. A systematic review of EHR-based studies also identified data quality as an overarching barrier to the use of EHRs in managing the health of the community, i.e. “population health” [ 8 ]. Encouragingly this same review also identified more facilitators than barriers to the use of EHRs in public health, suggesting that opportunities outweigh the challenges. Shortreed et al. further explored these opportunities discussing how EHRs can enhance pragmatic trials, bring additional sophistication to observational studies, aid in predictive modeling, and be linked together to create more comprehensive views of patients’ health [ 9 ]. Yet, as Shortreed and others have noted, significant challenges still remain.

It is our intention with this narrative review to discuss some of these challenges in further detail. In particular, we focus on specific epidemiological threats to validity -- internal and external -- and how EHR-based epidemiological research in particular can exacerbate some of these threats. We note that while there is some overlap in the challenges we discuss with traditional paper-based medical record research that has occurred for decades, the scale and scope of an EHR-based study is often well beyond what was traditionally possible in the manual chart review era and our applied examples attempt to reflect this. We also describe existing and emerging approaches for remediating these potential biases as they arise. A summary of these challenges may be found in Table 1 . Our review is grounded in the healthcare system in the United States, although we expect many of the issues we describe to be applicable regardless of locale; where necessary, we have flagged our comments as specific to the U.S.

Challenge #1: Representativeness

The selection process for how patients are captured in the EHR is complex and a function of geographic, social, demographic, and economic determinants [ 10 ]. This can be termed the catchment of the EHR. For a patient record to appear in the EHR the patient must have been registered in the system, typically to capture their demographic and billing information, and upon a clinical visit, their health details. While this process is not new to clinical epidemiology, what tends to separate EHR-based records from traditional paper-based records is the scale and scope of the data. Patient data may be available for longer periods of time longitudinally, as well as have data corresponding to interactions with multiple, potentially disparate, healthcare systems [ 11 ]. Given the consolidation of healthcare [ 12 ] and aggregated views of multiple EHRs through health information networks or exchanges [ 11 ] the ability to have a complete view of the patients’ total health is increasing. Importantly, the epidemiologist must ascertain whether the population captured within the EHR or EHR-derived data is representative of the population targeted for inference. This is particularly true under the paradigm of population health and inferring the health status of a community from EHR-based records [ 13 ]. For example, a study of Clostridium difficile infection at an urban safety net hospital in Philadelphia, Pennsylvania demonstrated notable differences in risk factors in the hospital’s EHR compared to national surveillance data, suggesting how catchment can influence epidemiologic measures [ 14 ]. Even health-related data captured through health information exchanges may be incomplete [ 15 ].

Several hypothetical study settings can further help the epidemiologist appreciate the relationship between representativeness and validity in EHR research. In the first hypothetical, an EHR-based study is conducted from a single-location federally qualified health center, and in the second hypothetical, an EHR-based study is conducted from a large academic health system. Suppose both studies occur in the same geographic area. It is reasonable to believe the patient populations captured in both EHRs will be quite different and the catchment process could lead to divergent estimates of disease or risk factor prevalence. The large academic health system may be less likely to capture primary care visits, as specialty care may drive the preponderance of patient encounters. However, this is not a bias per se : if the target of inference from these two hypothetical EHR-based studies is the local community, then selection bias becomes a distinct possibility. The epidemiologist must also consider the potential for generalizability and transportability -- two facets of external validity that respectively relate to the extrapolation of study findings to the source population or a different population altogether -- if there are unmeasured effect modifiers, treatment interference, or compound treatments in the community targeted for inference [ 16 ].

There are several approaches for ascertaining representativeness of EHR-based data. Comparing the EHR-derived sample to Census estimates of demography is straightforward but has several important limitations. First, as previously described, the catchment process may be driven by discordant geographical areas, especially for specialty care settings. Second and third, the EHR may have limited or inaccurate information on socioeconomic status, race, and ethnicity that one may wish to compare [ 17 , 18 ], and conversely the Census has limited estimates of health, chiefly disability, fertility, and insurance and payments [ 19 ]. If selection bias is suspected as a result of missing visits in a longitudinal study [ 20 ] or the catchment process in a cross-sectional study [ 21 ], using inverse probability weighting may remediate its influence. Comparing the weighted estimates to the original, non-weighted estimates provides insight into differences in the study participants. In the population health paradigm whereby the EHR is used as a surveillance tool to identify community health disparities [ 13 ], one also needs to be concerned about representativeness. There are emerging approaches for producing such small area community estimates from large observational datasets [ 22 , 23 ]. Conceivably, these approaches may also be useful for identifying issues of representativeness, for example by comparing stratified estimates across sociodemographic or other factors that may relate to catchment. Approaches for issues concerning representativeness specifically as it applies to external validity may be found in these references [ 24 , 25 ].

Challenge #2: Data availability and interpretation

Sub-challenge #2.1: billing versus clinical versus epidemiological needs.

There is an inherent tension in the use of EHR-based data for research purposes: the EHR was never originally designed for research. In the U.S., the Health Information Technology for Economic and Clinical Health Act, which promoted EHRs as a platform for comparative effectiveness research, was an attempt to address this deficiency [ 26 ]. A brief history of the evolution of the modern EHR reveals a technology that was optimized for capturing health details relevant for billing, scheduling, and clinical record keeping [ 27 ]. As such, the availability of data for fundamental markers of upstream health that are important for identifying inequities, such as socioeconomic status, race, ethnicity, and other social determinants of health (SDOH), may be insufficiently captured in the EHR [ 17 , 18 ]. Similarly, behavioral risk factors, such as being a sexual minority person, have historically been insufficiently recorded as discrete variables. It is only recently that such data are beginning to be captured in the EHR [ 28 , 29 ], or techniques such as NLP have made it possible to extract these details when stored in free text notes (described further in “ Unstructured data: clinical notes and reports ” section).

As an example, assessing clinical morbidities in the EHR may be done on the basis of extracting appropriate International Classification of Diseases (ICD) codes, used for billing and reimbursement in the U.S. These codes are known to have low sensitivity despite high specificity for accurate diagnostic status [ 30 , 31 ]. Expressed as predictive values, which depend upon prevalence, presence of a diagnostic code is a likely indicator of a disease state, whereas absence of a diagnostic code is a less reliable indicator of the absence of that morbidity. There may further be variation by clinical domain in that ICD codes may exist but not be used in some specialties [ 32 ], variation by coding vocabulary such as the use of SNOMED for clinical documentation versus ICD for billing necessitating an ontology mapper [ 33 ], and variation by the use of “rule-out” diagnostic codes resulting in false-positive diagnoses [ 34 , 35 , 36 ]. Relatedly is the notion of upcoding, or the billing of tests, procedures, or diagnoses to receive inflated reimbursement, which, although posited to be problematic in EHRs [ 37 ] in at least one study, has not been shown to have occurred [ 38 ]. In the U.S., the billing and reimbursement model, such as fee-for-service versus managed care, may result in varying diagnostic code sensitivities and specificities, especially if upcoding is occurring [ 39 ]. In short, there is potential for misclassification of key health data in the EHR.

Misclassification can potentially be addressed through a validation study (resources permitting) or application of quantitative bias analysis, and there is a rich literature regarding the treatment of misclassified data in statistics and epidemiology. Readers are referred to these texts as a starting point [ 40 , 41 ]. Duda et al. and Shepherd et al. have described an innovative data audit approach applicable to secondary analysis of observational data, such as EHR-derived data, that incorporates the audit error rate directly in the regression analysis to reduce information bias [ 42 , 43 ]. Outside of methodological tricks in the face of imperfect data, researchers must proactively engage with clinical and informatics colleagues to ensure that the right data for the research interests are available and accessible.

Sub-challenge #2.2: Consistency in data and interpretation

For the epidemiologist, abstracting data from the EHR into a research-ready analytic dataset presents a host of complications surrounding data availability, consistency and interpretation. It is easy to conflate the total volume of data in the EHR with data that are usable for research, however expectations should be tempered. Weiskopf et al. have noted such challenges for the researcher: in their study, less than 50% of patient records had “complete” data for research purposes per their four definitions of completeness [ 44 ]. Decisions made about the treatment of incomplete data can induce selection bias or impact precision of estimates (see Challenges #1 , #3 , and #4 ). The COVID-19 pandemic has further demonstrated the challenge of obtaining research data from EHRs across multiple health systems [ 45 ]. On the other hand, EHRs have a key advantage of providing near real-time data as opposed to many epidemiological studies that have a specific endpoint or are retrospective in nature. Such real-time data availability was leveraged during COVID-19 to help healthcare systems manage their pandemic response [ 46 , 47 ]. Logistical and technical issues aside, healthcare and documentation practices are nuanced to their local environments. In fact, researchers have demonstrated how the same research question analyzed in distinct clinical databases can yield different results [ 48 ].

Once the data are obtained, choices regarding operationalization of variables have the potential to induce information bias. Several hypothetical examples can help demonstrate this point. As a first example, differences in laboratory reporting may result in measurement error or misclassification. While the order for a particular laboratory assay is likely consistent within the healthcare system, patients frequently have a choice where to have that order fulfilled. Given the breadth of assays and reporting differences that may differ lab to lab [ 49 ], it is possible that the researcher working with the raw data may not consider all possible permutations. In other words, there may be lack of consistency in the reporting of the assay results. As a second example, raw clinical data requires interpretation to become actionable. A researcher interested in capturing a patient’s Charlson comorbidity index, which is based on 16 potential diagnoses plus the patient’s age [ 50 ], may never find such a variable in the EHR. Rather, this would require operationalization based on the raw data, each of which may be misclassified. Use of such composite measures introduces the notion of “differential item functioning”, whereby a summary indicator of a complexly measured health phenomenon may differ from group to group [ 51 ]. In this case, as opposed to a measurement error bias, this is one of residual confounding in that a key (unmeasured) variable is driving the differences. Remediation of these threats to validity may involve validation studies to determine the accuracy of a particular classifier, sensitivity analysis employing alternative interpretations when the raw data are available, and omitting or imputing biased or latent variables [ 40 , 41 , 52 ]. Importantly, in all cases, the epidemiologists should work with the various health care providers and personnel who have measured and recorded the data present in the EHR, as they likely understand it best.

Furthermore and related to “Billing versus Clinical versus Epidemiological Needs” section, the healthcare system in the U.S. is fragmented with multiple payers, both public and private, potentially exacerbating the data quality issues we describe, especially when linking data across healthcare systems. Single payer systems have enabled large and near-complete population-based studies due to data availability and consistency [ 53 , 54 , 55 ]. Data may also be inconsistent for retrospective longitudinal studies spanning many years if there have been changes to coding standards or practices over time, for example due to the transition from ICD-9 to ICD-10 largely occurring in the mid 2010s or the adoption of the Patient Protection and Affordable Care Act in the U.S. in 2010 with its accompanying changes in billing. Exploratory data analysis may reveal unexpected differences in key variables, by place or time, and recoding, when possible, can enforce consistency.

Sub-challenge #2.3: Unstructured data: clinical notes and reports

There may also be scenarios where structured data fields, while available, are not traditionally or consistently used within a given medical center or by a given provider. For example, reporting of adverse events of medications, disease symptoms, and vaccinations or hospitalizations occurring at different facility/health networks may not always be entered by providers in structured EHR fields. Instead, these types of patient experiences may be more likely to be documented in an unstructured clinical note, report (e.g. pathology or radiology report), or scanned document. Therefore, reliance on structured data to identify and study such issues may result in underestimation and potentially biased results.

Advances in NLP currently allow for information to be extracted from unstructured clinical notes and text fields in a reliable and accurate manner using computational methods. NLP utilizes a range of different statistical, machine learning, and linguistic techniques, and when applied to EHR data, has the potential to facilitate more accurate detection of events not traditionally located or consistently used in structured fields. Various NLP methods can be implemented in medical text analysis, ranging from simplistic and fast term recognition systems to more advanced, commercial NLP systems [ 56 ]. Several studies have successfully utilized text mining to extract information on a variety of health-related issues within clinical notes, such as opioid use [ 57 ], adverse events [ 58 , 59 ], symptoms (e.g., shortness of breath, depression, pain) [ 60 ], and disease phenotype information documented in pathology or radiology reports, including cancer stage, histology, and tumor grade [ 61 ], and lupus nephritis [ 32 ]. It is worth noting that scanned documents involve an additional layer of computation, relying on techniques such as optical character recognition, before NLP can be applied.

Hybrid approaches that combine both narrative and structured data, such as ICD codes, to improve accuracy of detecting phenotypes have also demonstrated high performance. Banerji et al. found that using ICD-9 codes to identify allergic drug reactions in the EHR had a positive predictive value of 46%, while an NLP algorithm in conjunction with ICD-9 codes resulted in a positive predictive value of 86%; negative predictive value also increased in the combined algorithm (76%) compared to ICD-9 codes alone (39%) [ 62 ]. In another example, researchers found that the combination of unstructured clinical notes with structured data for prediction tasks involving in-hospital mortality and 30-day hospital readmission outperformed models using either clinical notes or structured data alone [ 63 ]. As we move forward in analyzing EHR data, it will be important to take advantage of the wealth of information buried in unstructured data to assist in phenotyping patient characteristics and outcomes, capture missing confounders used in multivariate analyses, and develop prediction models.

Challenge #3: Missing measurements

While clinical notes may be useful to recover incomplete information from structured data fields, it may be the case that certain variables are not collected within the EHR at all. As mentioned above, it is important to remember that EHRs were not developed as a research tool (see “ Billing versus clinical versus epidemiological needs ” section), and important variables often used in epidemiologic research may not be typically included in EHRs including socioeconomic status (education, income, occupation) and SDOH [ 17 , 18 ]. Depending upon the interest of the provider or clinical importance placed upon a given variable, this information may be included in clinical notes. While NLP could be used to capture these variables, because they may not be consistently captured, there may be bias in identifying those with a positive mention as a positive case and those with no mention as a negative case. For example, if a given provider inquires about homelessness of a patient based on knowledge of the patient’s situation or other external factors and documents this in the clinical note, we have greater assurance that this is a true positive case. However, lack of mention of homelessness in a clinical note should not be assumed as a true negative case for several reasons: not all providers may feel comfortable asking about and/or documenting homelessness, they may not deem this variable worth noting, or implicit bias among clinicians may affect what is captured. As a result, such cases (i.e. no mention of homelessness) may be incorrectly identified as “not homeless,” leading to selection bias should a researcher form a cohort exclusively of patients who are identified as homeless in the EHR.

Not adjusting for certain measurements missing from EHR data can also lead to biased results if the measurement is an important confounder. Consider the example of distinguishing between prevalent and incident cases of disease when examining associations between disease treatments and patient outcomes [ 64 ]. The first date of an ICD code entered for a given patient may not necessarily be the true date of diagnosis, but rather documentation of an existing diagnosis. This limits the ability to adjust for disease duration, which may be an important confounder in studies comparing various treatments with patient outcomes over time, and may also lead to reverse causality if disease sequalae are assumed to be risk factors.

Methods to supplement EHR data with external data have been used to capture missing information. These methods may include imputation if information (e.g. race, lab values) is collected on a subset of patients within the EHR. It is important to examine whether missingness occurs completely at random or at random (“ignorable”), or not at random (“non-ignorable”), using the data available to determine factors associated with missingness, which will also inform the best imputation strategy to pursue, if any [ 65 , 66 ]. As an example, suppose we are interested in ascertaining a patient's BMI from the EHR. If men were less likely to have BMI measured than women, the probability of missing data (BMI) depends on the observed data (gender) and may therefore be predictable and imputable. On the other hand, suppose underweight individuals were less likely to have BMI measured; the probability of missing data depends on its own value, and as such is non-predictable and may require a validation study to confirm. Alternatively to imputing missing data, surrogate measures may be used, such as inferring area-based SES indicators, including median household income, percent poverty, or area deprivation index, by zip code [ 67 , 68 ]. Lastly, validation studies utilizing external datasets may prove helpful, such as supplementing EHR data with claims data that may be available for a subset of patients (see Challenge #4 ).

As EHRs are increasingly being used for research, there are active pushes to include more structured data fields that are important to population health research, such as SDOH [ 69 ]. Inclusion of such factors are likely to result in improved patient care and outcomes, through increased precision in disease diagnosis, more effective shared decision making, identification of risk factors, and tailoring services to a given population’s needs [ 70 ]. In fact, a recent review found that when individual level SDOH were included in predictive modeling, they overwhelmingly improved performance in medication adherence, risk of hospitalization, 30-day rehospitalizations, suicide attempts, and other healthcare services [ 71 ]. Whether or not these fields will be utilized after their inclusion in the EHR may ultimately depend upon federal and state incentives, as well as support from local stakeholders, and this does not address historic, retrospective analyses of these data.

Challenge #4: Missing visits

Beyond missing variable data that may not be captured during a clinical encounter, either through structured data or clinical notes, there also may be missing information for a patient as a whole. This can occur in a variety of ways; for example, a patient may have one or two documented visits in the EHR and then is never seen again (i.e. right censoring due to lost to follow-up), or a patient is referred from elsewhere to seek specialty care, with no information captured regarding other external issues (i.e. left censoring). This may be especially common in circumstances where a given EHR is more likely to capture specialty clinics versus primary care (see Challenge #1 ). A third scenario may include patients who appear, then are not observed for a long period of time, and then reappear: this case is particularly problematic as it may appear the patient was never lost to follow up but simply had fewer visits. In any of these scenarios, a researcher will lack a holistic view of the patient’s experiences, diagnoses, results, and more. As discussed above, assuming absence of a diagnostic code as absence of disease may lead to information and/or selection bias. Further, it has been demonstrated that one key source of bias in EHRs is “informed presence” bias, where those with more medical encounters are more likely to be diagnosed with various conditions (similar to Berkson’s bias) [ 72 ].

Several solutions to these issues have been proposed. For example, it is common for EHR studies to condition on observation time (i.e. ≥n visits required to be eligible into cohort); however, this may exclude a substantial amount of patients with certain characteristics, incurring a selection bias or limiting the generalizability of study findings (see Challenge #1 ). Other strategies attempt to account for missing visit biases through longitudinal imputation approaches; for example, if a patient missed a visit, a disease activity score can be imputed for that point in time, given other data points [ 73 , 74 ]. Surrogate measures may also be used to infer patient outcomes, such as controlling for “informative” missingness as an indicator variable or using actual number of missed visits that were scheduled as a proxy for external circumstances influencing care [ 20 ]. To address “informed presence” bias described above, conditioning on the number of health-care encounters may be appropriate [ 72 ]. Understanding the reason for the missing visit may help identify the best course of action and before imputing, one should be able to identify the type of missingness, whether “informative” or not [ 65 , 66 ]. For example, if distance to a healthcare location is related to appointment attendance, being able to account for this in analysis would be important: researchers have shown how the catchment of a healthcare facility can induce selection bias [ 21 ]. Relatedly, as telehealth becomes more common fueled by the COVID-19 pandemic [ 75 , 76 ], virtual visits may generate missingness of data recorded in the presence of a provider (e.g., blood pressure if the patient does not have access to a sphygmomanometer; see Challenge #3 ), or necessitate a stratified analysis by visit type to assess for effect modification.

Another common approach is to supplement EHR information with external data sources, such as insurance claims data, when available. Unlike a given EHR, claims data are able to capture a patient’s interaction with the health care system across organizations, and additionally includes pharmacy data such as if a prescription was filled or refilled. Often researchers examine a subset of patients eligible for Medicaid/Medicare and compare what is documented in claims with information available in the EHR [ 77 ]. That is, are there additional medications, diagnoses, hospitalizations found in the claims dataset that were not present in the EHR. In a study by Franklin et al., researchers utilized a linked database of Medicare Advantage claims and comprehensive EHR data from a multi-specialty outpatient practice to determine which dataset would be more accurate in predicting medication adherence [ 77 ]. They found that both datasets were comparable in identifying those with poor adherence, though each dataset incorporated different variables.

While validation studies such as those using claims data allow researchers to gain an understanding as to how accurate and complete a given EHR is, this may only be limited to the specific subpopulation examined (i.e. those eligible for Medicaid, or those over 65 years for Medicare). One study examined congruence between EHR of a community health center and Medicaid claims with respect to diabetes [ 78 ]. They found that patients who were older, male, Spanish-speaking, above the federal poverty level, or who had discontinuous insurance were more likely to have services documented in the EHR as compared to Medicaid claims data. Therefore, while claims data may help supplement and validate information in the EHR, on their own they may underestimate care in certain populations.

Research utilizing EHR data has undoubtedly positively impacted the field of public health through its ability to provide large-scale, longitudinal data on a diverse set of patients, and will continue to do so in the future as more epidemiologists take advantage of this data source. EHR data’s ability to capture individuals that traditionally aren’t included in clinical trials, cohort studies, and even claims datasets allows researchers to measure longitudinal outcomes in patients and perhaps change the understanding of potential risk factors.

However, as outlined in this review, there are important caveats to EHR analysis that need to be taken into account; failure to do so may threaten study validity. The representativeness of EHR data depends on the catchment area of the center and corresponding target population. Tools are available to evaluate and remedy these issues, which are critical to study validity as well as extrapolation of study findings. Data availability and interpretation, missing measurements, and missing visits are also key challenges, as EHRs were not specifically developed for research purposes, despite their common use for such. Taking advantage of all available EHR data, whether it be structured or unstructured fields through NLP, will be important in understanding the patient experience and identifying key phenotypes. Beyond methods to address these concerns, it will remain crucial for epidemiologists and data analysts to engage with clinicians and informaticians at their institutions to ensure data quality and accessibility by forming multidisciplinary teams around specific research projects. Lastly, integration across multiple EHRs, or datasets that encompass multi-institutional EHR records, add an additional layer of data quality and validity issues, with the potential to exacerbate the above-stated challenges found within a single EHR. At minimum, such studies should account for correlated errors [ 79 , 80 ], and investigate whether modularization, or submechanisms that determine whether data are observed or missing in each EHR, exist [ 65 ].

The identified challenges may also apply to secondary analysis of other large healthcare databases, such as claims data, although it is important not to conflate the two types of data. EHR data are driven by clinical care and claims data are driven by the reimbursement process where there is a financial incentive to capture diagnoses, procedures, and medications [ 48 ]. The source of data likely influences the availability, accuracy, and completeness of data. The fundamental representation of data may also differ as a record in a claims database corresponds to a “claim” as opposed to an “encounter” in the EHR. As such, the representativeness of the database populations, the sensitivity and specificity of variables, as well as the mechanisms of missingness in claims data may differ from EHR data. One study that evaluated pediatric quality care measures, such as BMI, noted inferior sensitivity based on claims data alone [ 81 ]. Linking claims data to EHR data has been proposed to enhance study validity, but many of the caveats raised in herein still apply [ 82 ].

Although we focused on epidemiological challenges related to study validity, there are other important considerations for researchers working with EHR data. Privacy and security of data as well as institutional review board (IRB) or ethics board oversight of EHR-based studies should not be taken for granted. For researchers in the U.S., Goldstein and Sarwate described Health Insurance Portability and Accountability Act (HIPAA)-compliant approaches to ensure the privacy and security of EHR data used in epidemiological research, and presented emerging approaches to analyses that separate the data from analysis [ 83 ]. The IRB oversees the data collection process for EHR-based research and through the HIPAA Privacy Rule these data typically do not require informed consent provided they are retrospective and reside at the EHR’s institution [ 84 ]. Such research will also likely receive an exempt IRB review provided subjects are non-identifiable.

Conclusions

As EHRs are increasingly being used for research, epidemiologists can take advantage of the many tools and methods that already exist and apply them to the key challenges described above. By being aware of the limitations that the data present and proactively addressing them, EHR studies will be more robust, informative, and important to the understanding of health and disease in the population.

Availability of data and materials

All data and materials used in this review are described herein.

Abbreviations

Body Mass Index

Electronic Health Record

International Classification of Diseases

Institutional review board/ethics board

Health Insurance Portability and Accountability Act

Natural Language Processing

Social Determinants of Health

Socioeconomic Status

Adler-Milstein J, Holmgren AJ, Kralovec P, et al. Electronic health record adoption in US hospitals: the emergence of a digital “advanced use” divide. J Am Med Inform Assoc. 2017;24(6):1142–8.

Article   PubMed   PubMed Central   Google Scholar  

Office of the National Coordinator for Health Information Technology. ‘Office-based physician electronic health record adoption’, Health IT quick-stat #50. dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php . Accessed 15 Jan 2019.

Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol. 2017;106(1):1–9.

Article   PubMed   Google Scholar  

Casey JA, Schwartz BS, Stewart WF, et al. Using electronic health records for population health research: a review of methods and applications. Annu Rev Public Health. 2016;37:61–81.

Verheij RA, Curcin V, Delaney BC, et al. Possible sources of bias in primary care electronic health record data use and reuse. J Med Internet Res. 2018;20(5):e185.

Ni K, Chu H, Zeng L, et al. Barriers and facilitators to data quality of electronic health records used for clinical research in China: a qualitative study. BMJ Open. 2019;9(7):e029314.

Coleman N, Halas G, Peeler W, et al. From patient care to research: a validation study examining the factors contributing to data quality in a primary care electronic medical record database. BMC Fam Pract. 2015;16:11.

Kruse CS, Stein A, Thomas H, et al. The use of electronic health records to support population health: a systematic review of the literature. J Med Syst. 2018;42(11):214.

Shortreed SM, Cook AJ, Coley RY, et al. Challenges and opportunities for using big health care data to advance medical science and public health. Am J Epidemiol. 2019;188(5):851–61.

In: Smedley BD, Stith AY, Nelson AR, editors. Unequal treatment: confronting racial and ethnic disparities in health care. Washington (DC) 2003.

Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742–52.

Cutler DM, Scott Morton F. Hospitals, market share, and consolidation. JAMA. 2013;310(18):1964–70.

Article   CAS   PubMed   Google Scholar  

Cocoros NM, Kirby C, Zambarano B, et al. RiskScape: a data visualization and aggregation platform for public health surveillance using routine electronic health record data. Am J Public Health. 2021;111(2):269–76.

Vader DT, Weldie C, Welles SL, et al. Hospital-acquired Clostridioides difficile infection among patients at an urban safety-net hospital in Philadelphia: demographics, neighborhood deprivation, and the transferability of national statistics. Infect Control Hosp Epidemiol. 2020;42:1–7.

Google Scholar  

Dixon BE, Gibson PJ, Frederickson Comer K, et al. Measuring population health using electronic health records: exploring biases and representativeness in a community health information exchange. Stud Health Technol Inform. 2015;216:1009.

PubMed   Google Scholar  

Hernán MA, VanderWeele TJ. Compound treatments and transportability of causal inference. Epidemiology. 2011;22(3):368–77.

Casey JA, Pollak J, Glymour MM, et al. Measures of SES for electronic health record-based research. Am J Prev Med. 2018;54(3):430–9.

Polubriaginof FCG, Ryan P, Salmasian H, et al. Challenges with quality of race and ethnicity data in observational databases. J Am Med Inform Assoc. 2019;26(8-9):730–6.

U.S. Census Bureau. Health. Available at: https://www.census.gov/topics/health.html . Accessed 19 Jan 2021.

Gianfrancesco MA, McCulloch CE, Trupin L, et al. Reweighting to address nonparticipation and missing data bias in a longitudinal electronic health record study. Ann Epidemiol. 2020;50:48–51 e2.

Goldstein ND, Kahal D, Testa K, Burstyn I. Inverse probability weighting for selection bias in a Delaware community health center electronic medical record study of community deprivation and hepatitis C prevalence. Ann Epidemiol. 2021;60:1–7.

Gelman A, Lax J, Phillips J, et al. Using multilevel regression and poststratification to estimate dynamic public opinion. Unpublished manuscript, Columbia University. 2016 Sep 11. Available at: http://www.stat.columbia.edu/~gelman/research/unpublished/MRT(1).pdf . Accessed 22 Jan 2021.

Quick H, Terloyeva D, Wu Y, et al. Trends in tract-level prevalence of obesity in philadelphia by race-ethnicity, space, and time. Epidemiology. 2020;31(1):15–21.

Lesko CR, Buchanan AL, Westreich D, Edwards JK, Hudgens MG, Cole SR. Generalizing study results: a potential outcomes perspective. Epidemiology. 2017;28(4):553–61.

Westreich D, Edwards JK, Lesko CR, Stuart E, Cole SR. Transportability of trial results using inverse odds of sampling weights. Am J Epidemiol. 2017;186(8):1010–4.

Congressional Research Services (CRS). The Health Information Technology for Economic and Clinical Health (HITECH) Act. 2009. Available at: https://crsreports.congress.gov/product/pdf/R/R40161/9 . Accessed Jan 22 2021.

Hersh WR. The electronic medical record: Promises and problems. Journal of the American Society for Information Science. 1995;46(10):772–6.

Article   Google Scholar  

Collecting sexual orientation and gender identity data in electronic health records: workshop summary. Washington (DC) 2013.

Committee on the Recommended Social and Behavioral Domains and Measures for Electronic Health Records; Board on Population Health and Public Health Practice; Institute of Medicine. Capturing social and behavioral domains and measures in electronic health records: phase 2. Washington (DC): National Academies Press (US); 2015.

Goff SL, Pekow PS, Markenson G, et al. Validity of using ICD-9-CM codes to identify selected categories of obstetric complications, procedures and co-morbidities. Paediatr Perinat Epidemiol. 2012;26(5):421–9.

Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58(4):323–37.

Gianfrancesco MA. Application of text mining methods to identify lupus nephritis from electronic health records. Lupus Science & Medicine. 2019;6:A142.

National Library of Medicine. SNOMED CT to ICD-10-CM Map. Available at: https://www.nlm.nih.gov/research/umls/mapping_projects/snomedct_to_icd10cm.html . Accessed 2 Jul 2021.

Klabunde CN, Harlan LC, Warren JL. Data sources for measuring comorbidity: a comparison of hospital records and medicare claims for cancer patients. Med Care. 2006;44(10):921–8.

Burles K, Innes G, Senior K, Lang E, McRae A. Limitations of pulmonary embolism ICD-10 codes in emergency department administrative data: let the buyer beware. BMC Med Res Methodol. 2017;17(1):89.

Asgari MM, Wu JJ, Gelfand JM, Salman C, Curtis JR, Harrold LR, et al. Validity of diagnostic codes and prevalence of psoriasis and psoriatic arthritis in a managed care population, 1996-2009. Pharmacoepidemiol Drug Saf. 2013;22(8):842–9.

Hoffman S, Podgurski A. Big bad data: law, public health, and biomedical databases. J Law Med Ethics. 2013;41(Suppl 1):56–60.

Adler-Milstein J, Jha AK. Electronic health records: the authors reply. Health Aff. 2014;33(10):1877.

Geruso M, Layton T. Upcoding: evidence from medicare on squishy risk adjustment. J Polit Econ. 2020;12(3):984–1026.

Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data. New York: Springer-Verlag New York; 2009.

Book   Google Scholar  

Gustafson P. Measurement error and misclassification in statistics and epidemiology: impacts and Bayesian adjustments. Boca Raton: Chapman and Hall/CRC; 2004.

Duda SN, Shepherd BE, Gadd CS, et al. Measuring the quality of observational study data in an international HIV research network. PLoS One. 2012;7(4):e33908.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Shepherd BE, Yu C. Accounting for data errors discovered from an audit in multiple linear regression. Biometrics. 2011;67(3):1083–91.

Weiskopf NG, Hripcsak G, Swaminathan S, et al. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform. 2013;46(5):830–6.

Kaiser Health News. As coronavirus strikes, crucial data in electronic health records hard to harvest. Available at: https://khn.org/news/as-coronavirus-strikes-crucial-data-in-electronic-health-records-hard-to-harvest/ . Accessed 15 Jan 2021.

Reeves JJ, Hollandsworth HM, Torriani FJ, Taplitz R, Abeles S, Tai-Seale M, et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc. 2020;27(6):853–9.

Grange ES, Neil EJ, Stoffel M, Singh AP, Tseng E, Resco-Summers K, et al. Responding to COVID-19: The UW medicine information technology services experience. Appl Clin Inform. 2020;11(2):265–75.

Madigan D, Ryan PB, Schuemie M, et al. Evaluating the impact of database heterogeneity on observational study results. Am J Epidemiol. 2013;178(4):645–51.

Lippi G, Mattiuzzi C. Critical laboratory values communication: summary recommendations from available guidelines. Ann Transl Med. 2016;4(20):400.

Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.

Jones RN. Differential item functioning and its relevance to epidemiology. Curr Epidemiol Rep. 2019;6:174–83.

Edwards JK, Cole SR, Troester MA, Richardson DB. Accounting for misclassified outcomes in binary regression models using multiple imputation with internal validation data. Am J Epidemiol. 2013;177(9):904–12.

Satkunasivam R, Klaassen Z, Ravi B, Fok KH, Menser T, Kash B, et al. Relation between surgeon age and postoperative outcomes: a population-based cohort study. CMAJ. 2020;192(15):E385–92.

Melamed N, Asztalos E, Murphy K, Zaltz A, Redelmeier D, Shah BR, et al. Neurodevelopmental disorders among term infants exposed to antenatal corticosteroids during pregnancy: a population-based study. BMJ Open. 2019;9(9):e031197.

Kao LT, Lee HC, Lin HC, Tsai MC, Chung SD. Healthcare service utilization by patients with obstructive sleep apnea: a population-based study. PLoS One. 2015;10(9):e0137459.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Jung K, LePendu P, Iyer S, Bauer-Mehren A, Percha B, Shah NH. Functional evaluation of out-of-the-box text-mining tools for data-mining tasks. J Am Med Inform Assoc. 2015;22(1):121–31.

Canan C, Polinski JM, Alexander GC, et al. Automatable algorithms to identify nonmedical opioid use using electronic data: a systematic review. J Am Med Inform Assoc. 2017;24(6):1204–10.

Iqbal E, Mallah R, Jackson RG, et al. Identification of adverse drug events from free text electronic patient records and information in a large mental health case register. PLoS One. 2015;10(8):e0134208.

Rochefort CM, Verma AD, Eguale T, et al. A novel method of adverse event detection can accurately identify venous thromboembolisms (VTEs) from narrative electronic health record data. J Am Med Inform Assoc. 2015;22(1):155–65.

Koleck TA, Dreisbach C, Bourne PE, et al. Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review. J Am Med Inform Assoc. 2019;26(4):364–79.

Wang L, Luo L, Wang Y, et al. Natural language processing for populating lung cancer clinical research data. BMC Med Inform Decis Mak. 2019;19(Suppl 5):239.

Banerji A, Lai KH, Li Y, et al. Natural language processing combined with ICD-9-CM codes as a novel method to study the epidemiology of allergic drug reactions. J Allergy Clin Immunol Pract. 2020;8(3):1032–1038.e1.

Zhang D, Yin C, Zeng J, et al. Combining structured and unstructured data for predictive models: a deep learning approach. BMC Med Inform Decis Mak. 2020;20(1):280.

Farmer R, Mathur R, Bhaskaran K, Eastwood SV, Chaturvedi N, Smeeth L. Promises and pitfalls of electronic health record analysis. Diabetologia. 2018;61:1241–8.

Haneuse S, Arterburn D, Daniels MJ. Assessing missing data assumptions in EHR-based studies: a complex and underappreciated task. JAMA Netw Open. 2021;4(2):e210184.

Groenwold RHH. Informative missingness in electronic health record systems: the curse of knowing. Diagn Progn Res. 2020;4:8.

Berkowitz SA, Traore CY, Singer DE, et al. Evaluating area-based socioeconomic status indicators for monitoring disparities within health care systems: results from a primary care network. Health Serv Res. 2015;50(2):398–417.

Kind AJH, Buckingham WR. Making neighborhood-disadvantage metrics accessible - the neighborhood atlas. N Engl J Med. 2018;378(26):2456–8.

Cantor MN, Thorpe L. Integrating data on social determinants of health into electronic health records. Health Aff. 2018;37(4):585–90.

Adler NE, Stead WW. Patients in context--EHR capture of social and behavioral determinants of health. N Engl J Med. 2015;372(8):698–701.

Chen M, Tan X, Padman R. Social determinants of health in electronic health records and their impact on analysis and risk prediction: a systematic review. J Am Med Inform Assoc. 2020;27(11):1764–73.

Goldstein BA, Bhavsar NA, Phelan M, et al. Controlling for informed presence bias due to the number of health encounters in an electronic health record. Am J Epidemiol. 2016;184(11):847–55.

Petersen I, Welch CA, Nazareth I, et al. Health indicator recording in UK primary care electronic health records: key implications for handling missing data. Clin Epidemiol. 2019;11:157–67.

Li R, Chen Y, Moore JH. Integration of genetic and clinical information to improve imputation of data missing from electronic health records. J Am Med Inform Assoc. 2019;26(10):1056–63.

Koonin LM, Hoots B, Tsang CA, Leroy Z, Farris K, Jolly T, et al. Trends in the use of telehealth during the emergence of the COVID-19 pandemic - United States, January-March 2020. MMWR Morb Mortal Wkly Rep. 2020;69(43):1595–9.

Barnett ML, Ray KN, Souza J, Mehrotra A. Trends in telemedicine use in a large commercially insured population, 2005-2017. JAMA. 2018;320(20):2147–9.

Franklin JM, Gopalakrishnan C, Krumme AA, et al. The relative benefits of claims and electronic health record data for predicting medication adherence trajectory. Am Heart J. 2018;197:153–62.

Devoe JE, Gold R, McIntire P, et al. Electronic health records vs Medicaid claims: completeness of diabetes preventive care data in community health centers. Ann Fam Med. 2011;9(4):351–8.

Schmajuk G, Li J, Evans M, Anastasiou C, Izadi Z, Kay JL, et al. RISE registry reveals potential gaps in medication safety for new users of biologics and targeted synthetic DMARDs. Semin Arthritis Rheum. 2020 Dec;50(6):1542–8.

Izadi Z, Schmajuk G, Gianfrancesco M, Subash M, Evans M, Trupin L, et al. Rheumatology Informatics System for Effectiveness (RISE) practices see significant gains in rheumatoid arthritis quality measures. Arthritis Care Res. 2020. https://doi.org/10.1002/acr.24444 .

Angier H, Gold R, Gallia C, Casciato A, Tillotson CJ, Marino M, et al. Variation in outcomes of quality measurement by data source. Pediatrics. 2014;133(6):e1676–82.

Lin KJ, Schneeweiss S. Considerations for the analysis of longitudinal electronic health records linked to claims data to study the effectiveness and safety of drugs. Clin Pharmacol Ther. 2016;100(2):147–59.

Goldstein ND, Sarwate AD. Privacy, security, and the public health researcher in the era of electronic health record research. Online J Public Health Inform. 2016;8(3):e207.

U.S. Department of Health and Human Services (HHS). 45 CFR 46. http://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html .

Download references

Acknowledgements

The authors thank Dr. Annemarie Hirsch, Department of Population Health Sciences, Geisinger, for assistance in conceptualizing an earlier version of this work.

Research reported in this publication was supported in part by the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under Award Number K01AR075085 (to MAG) and the National Institute Of Allergy And Infectious Diseases of the National Institutes of Health under Award Number K01AI143356 (to NDG). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Authors and affiliations.

Division of Rheumatology, University of California School of Medicine, San Francisco, CA, USA

Milena A. Gianfrancesco

Department of Epidemiology and Biostatistics, Drexel University Dornsife School of Public Health, 3215 Market St., Philadelphia, PA, 19104, USA

Neal D. Goldstein

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors conceptualized, wrote, and approved the final submitted version.

Corresponding author

Correspondence to Neal D. Goldstein .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

The authors have no competing interests to declare

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gianfrancesco, M.A., Goldstein, N.D. A narrative review on the validity of electronic health record-based research in epidemiology. BMC Med Res Methodol 21 , 234 (2021). https://doi.org/10.1186/s12874-021-01416-5

Download citation

Received : 02 July 2021

Accepted : 28 September 2021

Published : 27 October 2021

DOI : https://doi.org/10.1186/s12874-021-01416-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Electronic health records
• Data quality
• Secondary analysis

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 25 September 2019

The future of electronic health records

• Jeff Hecht 0

Jeff Hecht is a science writer based in Newton, Massachusetts.

You can also search for this author in PubMed   Google Scholar

Credit: Totto Renna

Advances in medical imaging and the proliferation of diagnostic and screening tests have generated mountains of data on patient health. Digital information technology has seemed poised to revolutionize health care in the United States since 2009, when the Obama administration made the technology part of plans to revive a sinking economy. The US government has now spent tens of billions of dollars on putting patient information at doctors’ fingertips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

Nature 573 , S114-S116 (2019)

doi: https://doi.org/10.1038/d41586-019-02876-y

This article is part of Nature Outlook: Digital health , an editorially independent supplement produced with the financial support of third parties. About this content .

Dinov, I, D. J. Med. Stat. Inform. https://dx.doi.org/10.7243/2053-7662-4-3 (2016).

PubMed   Google Scholar  

Liberatore, K. Pa. Patient Saf. Advis. 15 (Suppl.), 16–24 (2018).

Google Scholar  

Arndt, B. G. et al. Ann. Fam. Med. 15 , 419–426 (2017).

Article   PubMed   Google Scholar  

Shanafelt, T. D. et al. Mayo Clin. Proc. 90 , 1600–1613 (2015).

Download references

Related Articles

• Information technology
• Health care

The dream of electronic newspapers becomes a reality — in 1974

News & Views 07 MAY 24

How scientists are making the most of Reddit

Career Feature 01 APR 24

A global timekeeping problem postponed by global warming

Article 27 MAR 24

AI’s keen diagnostic eye

Outlook 18 APR 24

So … you’ve been hacked

Technology Feature 19 MAR 24

No installation required: how WebAssembly is changing scientific computing

Technology Feature 11 MAR 24

Could bird flu in cows lead to a human outbreak? Slow response worries scientists

News 17 MAY 24

Neglecting sex and gender in research is a public-health risk

Comment 15 MAY 24

Interpersonal therapy can be an effective tool against the devastating effects of loneliness

Correspondence 14 MAY 24

Research Associate - Metabolism

Houston, Texas (US)

Baylor College of Medicine (BCM)

Postdoc Fellowships

Train with world-renowned cancer researchers at NIH? Consider joining the Center for Cancer Research (CCR) at the National Cancer Institute

Bethesda, Maryland

NIH National Cancer Institute (NCI)

Faculty Recruitment, Westlake University School of Medicine

Faculty positions are open at four distinct ranks: Assistant Professor, Associate Professor, Full Professor, and Chair Professor.

Hangzhou, Zhejiang, China

Westlake University

PhD/master's Candidate

PhD/master's Candidate    Graduate School of Frontier Science Initiative, Kanazawa University is seeking candidates for PhD and master's students i...

Kanazawa University

Senior Research Assistant in Human Immunology (wet lab)

Senior Research Scientist in Human Immunology, high-dimensional (40+) cytometry, ICS and automated robotic platforms.

Boston, Massachusetts (US)

Boston University Atomic Lab

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Suggestions or feedback?

MIT News | Massachusetts Institute of Technology

• Machine learning
• Social justice
• Black holes
• Classes and programs

Departments

• Aeronautics and Astronautics
• Brain and Cognitive Sciences
• Architecture
• Political Science
• Mechanical Engineering

Centers, Labs, & Programs

• Abdul Latif Jameel Poverty Action Lab (J-PAL)
• Picower Institute for Learning and Memory
• Lincoln Laboratory
• School of Architecture + Planning
• School of Engineering
• School of Humanities, Arts, and Social Sciences
• Sloan School of Management
• School of Science
• MIT Schwarzman College of Computing

Toward a smarter electronic health record

Press contact :, media download.

*Terms of Use:

Images for download on the MIT News office website are made available to non-commercial entities, press and the general public under a Creative Commons Attribution Non-Commercial No Derivatives license . You may not alter the images provided, other than to crop them to size. A credit line must be used when reproducing images; if one is not provided below, credit the images to "MIT."

Previous image Next image

Electronic health records have been widely adopted with the hope they would save time and improve the quality of patient care. But due to fragmented interfaces and tedious data entry procedures, physicians often spend more time navigating these systems than they do interacting with patients.

Researchers at MIT and the Beth Israel Deaconess Medical Center are combining machine learning and human-computer interaction to create a better electronic health record (EHR). They developed MedKnowts, a system that unifies the processes of looking up medical records and documenting patient information into a single, interactive interface.

Driven by artificial intelligence, this “smart” EHR automatically displays customized, patient-specific medical records when a clinician needs them. MedKnowts also provides autocomplete for clinical terms and auto-populates fields with patient information to help doctors work more efficiently.

“In the origins of EHRs, there was this tremendous enthusiasm that getting all this information organized would be helpful to be able to track billing records, report statistics to the government, and provide data for scientific research. But few stopped to ask the deep questions around whether they would be of use for the clinician. I think a lot of clinicians feel they have had this burden of EHRs put on them for the benefit of bureaucracies and scientists and accountants. We came into this project asking how EHRs might actually benefit clinicians,” says David Karger, professor of computer science in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and senior author of the paper.

The research was co-authored by CSAIL graduate students Luke Murray, who is the lead author, Divya Gopinath, and Monica Agrawal. Other authors include Steven Horng, an emergency medicine attending physician and clinical lead for machine learning at the Center for Healthcare Delivery Science of Beth Israel Deaconess Medical Center, and David Sontag, associate professor of electrical engineering and computer science at MIT and a member of CSAIL and the Institute for Medical Engineering and Science, and a principal investigator at the Abdul Latif Jameel Clinic for Machine Learning in Health. It will be presented at the Association for Computing Machinery Symposium on User Interface Software and Technology next month.

A problem-oriented tool

To design an EHR that would benefit doctors, the researchers had to think like doctors.

They created a note-taking editor with a side panel that displays relevant information from the patient’s medical history. That historical information appears in the form of cards that are focused on particular problems or concepts.

For instance, if MedKnowts identifies the clinical term “diabetes” in the text as a clinician types, the system automatically displays a “diabetes card” containing medications, lab values, and snippets from past records that are relevant to diabetes treatment.

Most EHRs store historical information on separate pages and list medications or lab values alphabetically or chronologically, forcing the clinician to search through data to find the information they need, Murray says. MedKnowts only displays information relevant to the particular concept the clinician is writing about.

“This is a closer match to the way doctors think about information. A lot of times, doctors will do this subconsciously. They will look through a medications page and only focus on the medications that are relevant to the current conditions. We are helping to do that process automatically and hopefully move some things out of the doctor’s head so they have more time to think about the complex part, which is determining what is wrong with the patient and coming up with a treatment plan,” Murray says.

Pieces of interactive text called chips serve as links to related cards. As a physician types a note, the autocomplete system recognizes clinical terms, such as medications, lab values, or conditions, and transforms them into chips. Each chip is displayed as a word or phrase that has been highlighted in a certain color depending on its category (red for a medical condition, green for a medication, yellow for a procedure, etc.)

Through the use of autocomplete, structured data on the patient’s conditions, symptoms, and medication usage is collected with no additional effort from the physician.

Sontag says he hopes the advance will “change the paradigm of how to create large-scale health datasets for studying disease progression and assessing the real-world effectiveness of treatments.”

In practice

After a year-long iterative design process, the researchers tested MedKnowts by deploying the software in the emergency department at Beth Israel Deaconess Medical Center in Boston. They worked with an emergency physician and four hospital scribes who enter notes into the electronic health record.

Deploying the software in an emergency department, where doctors operate in a high-stress environment, involved a delicate balancing act, Agrawal says.

“One of the biggest challenges we faced was trying to get people to shift what they currently do. Doctors who have used the same system, and done the same dance of clicks so many times, form a sort of muscle memory. Whenever you are going to make a change, there is a question of is this worth it? And we definitely found that some features had greater usage than others,” she says.

The Covid-19 pandemic complicated the deployment, too. The researchers had been visiting the emergency department to get a sense of the workflow, but were forced to end those visits due to Covid-19 and were unable to be in the hospital while the system was being deployed.

Despite those initial challenges, MedKnowts became popular with the scribes over the course of the one-month deployment. They gave the system an average rating of 83.75 (out of 100) for usability.

Scribes found the autocomplete function especially useful for speeding up their work, according to survey results. Also, the color-coded chips helped them quickly scan notes for relevant information.

Those initial results are promising, but as the researchers consider the feedback and work on future iterations of MedKnowts, they plan to proceed with caution.

“What we are trying to do here is smooth the pathway for doctors and let them accelerate. There is some risk there. Part of the purpose of bureaucracy is to slow things down and make sure all the i’s are dotted and all the t’s are crossed. And if we have a computer dotting the i’s and crossing the t’s for doctors, that may actually be countering the goals of the bureaucracy, which is to force doctors to think twice before they make a decision. We have to be thinking about how to protect doctors and patients from the consequences of making the doctors more efficient,” Karger says.

A longer-term vision

The researchers plan to improve the machine learning algorithms that drive MedKnowts so the system can more effectively highlight parts of the medical record that are most relevant, Agrawal says.

They also want to consider the needs of different medical users. The researchers designed MedKnowts with an emergency department in mind — a setting where doctors are typically seeing patients for the first time. A primary care physician who knows their patients much better would likely have some different needs.

In the longer-term, the researchers envision creating an adaptive system that clinicians can contribute to. For example, perhaps a doctor realizes a certain cardiology term is missing from MedKnowts and adds that information to a card, which would update the system for all users.

The team is exploring commercialization as an avenue for further deployment.

“We want to build tools that let doctors create their own tools. We don’t expect doctors to learn to be programmers, but with the right support they might be able to radically customize whatever medical applications they are using to really suit their own needs and preferences,” Karger says.

This research was funded by the MIT Abdul Latif Jameel Clinic for Machine Learning in Health.

Share this news article on:

Related links.

• David Karger
• David Sontag
• Computer Science and Artificial Intelligence Laboratory
• Abdul Latif Jameel Clinic for Machine Learning in Health
• Institute for Medical Engineering and Science
• Department of Electrical Engineering and Computer Science

Related Topics

• Health care
• Artificial intelligence
• Electrical Engineering & Computer Science (eecs)
• Computer Science and Artificial Intelligence Laboratory (CSAIL)
• Institute for Medical Engineering and Science (IMES)

Related Articles

3 Questions: John Van Reenen on the impact of technology on health care workers

Algorithm reduces use of riskier antibiotics for UTIs

How well can computers connect symptoms to diseases?

Model improves prediction of mortality risk in ICU patients

Previous item Next item

More MIT News

Researchers develop a detector for continuously monitoring toxic gases

Read full story →

The beauty of biology

Navigating longevity with industry leaders at MIT AgeLab PLAN Forum

Jeong Min Park earns 2024 Schmidt Science Fellowship

New tool empowers users to fight online misinformation

Elaine Liu: Charging ahead

• More news on MIT News homepage →

Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA

• Map (opens in new window)
• Events (opens in new window)
• People (opens in new window)
• Careers (opens in new window)
• Accessibility
• Social Media Hub
• MIT on Facebook
• MIT on YouTube
• MIT on Instagram

• Research article
• Open access
• Published: 04 September 2014

Implementing electronic health records in hospitals: a systematic literature review

• Albert Boonstra 1 ,
• Arie Versluis 2 &
• Janita F J Vos 1  

BMC Health Services Research volume  14 , Article number:  370 ( 2014 ) Cite this article

94k Accesses

170 Citations

55 Altmetric

Metrics details

The literature on implementing Electronic Health Records (EHR) in hospitals is very diverse. The objective of this study is to create an overview of the existing literature on EHR implementation in hospitals and to identify generally applicable findings and lessons for implementers.

A systematic literature review of empirical research on EHR implementation was conducted. Databases used included Web of Knowledge, EBSCO, and Cochrane Library. Relevant references in the selected articles were also analyzed. Search terms included Electronic Health Record (and synonyms), implementation, and hospital (and synonyms). Articles had to meet the following requirements: (1) written in English, (2) full text available online, (3) based on primary empirical data, (4) focused on hospital-wide EHR implementation, and (5) satisfying established quality criteria.

Of the 364 initially identified articles, this study analyzes the 21 articles that met the requirements. From these articles, 19 interventions were identified that are generally applicable and these were placed in a framework consisting of the following three interacting dimensions: (1) EHR context, (2) EHR content, and (3) EHR implementation process.

Conclusions

Although EHR systems are anticipated as having positive effects on the performance of hospitals, their implementation is a complex undertaking. This systematic review reveals reasons for this complexity and presents a framework of 19 interventions that can help overcome typical problems in EHR implementation. This framework can function as a reference for implementers in developing effective EHR implementation strategies for hospitals.

Peer Review reports

In recent years, Electronic Health Records (EHRs) have been implemented by an ever increasing number of hospitals around the world. There have, for example, been initiatives, often driven by government regulations or financial stimulations, in the USA [ 1 ], the United Kingdom [ 2 ] and Denmark [ 3 ]. EHR implementation initiatives tend to be driven by the promise of enhanced integration and availability of patient data [ 4 ], by the need to improve efficiency and cost-effectiveness [ 5 ], by a changing doctor-patient relationship toward one where care is shared by a team of health care professionals [ 5 ], and/or by the need to deal with a more complex and rapidly changing environment [ 6 ].

EHR systems have various forms, and the term can relate to a broad range of electronic information systems used in health care. EHR systems can be used in individual organizations, as interoperating systems in affiliated health care units, on a regional level, or nationwide [ 1 , 2 ]. Health care units that use EHRs include hospitals, pharmacies, general practitioner surgeries, and other health care providers [ 7 ].

The implementation of hospital-wide EHR systems is a complex matter involving a range of organizational and technical factors including human skills, organizational structure, culture, technical infrastructure, financial resources, and coordination [ 8 , 9 ]. As Grimson et al. [ 5 ] argue, implementing information systems (IS) in hospitals is more challenging than elsewhere because of the complexity of medical data, data entry problems, security and confidentiality concerns, and a general lack of awareness of the benefits of Information Technology (IT). Boonstra and Govers [ 10 ] provide three reasons why hospitals differ from many other industries, and these differences might also affect EHR implementations. The first reason is that hospitals have multiple objectives, such as curing and caring for patients, and educating new physicians and nurses. Second, hospitals have complicated and highly varied structures and processes. Third, hospitals have a varied workforce including medical professionals who possess high levels of expertise, power, and autonomy. These distinct characteristics justify a study that focuses on the identification and analysis of the findings of previous studies on EHR implementation in hospitals.

Study aim, theoretical framework, and terminology

In dealing with the complexity of EHR implementation in hospitals, it is helpful to know which factors are seen as important in the literature and to capture the existing knowledge on EHR implementation in hospitals. As such, the objective of this research is to identify, categorize, and analyze the existing findings in the literature on EHR implementation processes in hospitals. This could contribute to greater insight into the underlying patterns and complex relationships involved in EHR implementation and could identify ways to tackle EHR implementation problems. In other words, this study focusses on the identification of factors that determine the progress of EHR implementation in hospitals. The motives behind implementing EHRs in hospitals and the effects on performance of implemented EHR systems are beyond the scope of this paper.

To our knowledge, there have been no systematic reviews of the literature concerning EHR implementation in hospitals and this article therefore fills that gap. Two interesting related review studies on EHR implementation are Keshavjee et al. [ 11 ] and McGinn et al. [ 12 ]. The study of Keshavjee et al. [ 11 ] develops a literature based integrative framework for EHR implementation. McGinn et al. [ 12 ] adopt an exclusive user perspective on EHR and their study is limited to Canada and countries with comparable socio-economic levels. Both studies are not explicitly focused on hospitals and include other contexts such as small clinics and national or regional EHR initiatives.

This systematic review is explicitly focused on hospital-wide, single hospital EHR implementations and identifies empirical studies (that include collected primary data) that reflect this situation. The categorization of the findings from the selected articles draws on Pettigrew’s framework for understanding strategic change [ 13 ]. This model has been widely applied in case study research into organizational contexts [ 14 ], as well as in studies on the implementation of health care innovations [ 15 ]. It generates insights by analyzing three interactive dimensions – context , content , and process – that together shape organizational change. Pettigrew’s framework [ 13 ] is seen as applicable because implementing an EHR artefact is an organization-wide effort. This framework was specifically selected for its focus on organizational change, its ease of understanding, and its relatively general dimensions allowing a broad range of findings to be included. The framework structures and focusses the analysis of the findings from the selected articles.

An organization’s context can be divided into internal and external components. External context refers to the social, economic, political, and competitive environments in which an organization operates. The internal context refers to the structure, culture, resources, capabilities, and politics of an organization. The content covers the specific areas of the transformation under examination. In an EHR implementation, these are the EHR system itself (both hardware and software), the work processes, and everything related to these (e.g. social conditions). The process dimension concerns the processes of change, made up of the plans, actions, reactions, and interactions of the stakeholders, rather than work processes in general. It is important to note that Pettigrew [ 13 ] does not see strategic change as a rational analytical process but rather as an iterative, continuous, multilevel process. This highlights that the outcome of an organizational change will be determined by the context, content, and process of that change. The framework with its three categories, shown in Figure  1 , illustrates the conceptual model used to categorize the findings of this systematic literature review.

Pettigrew ’ s framework [ 13 ] ] and the corresponding categories.

In the literature, several terms are used to refer to electronic medical information systems. In this article, the term Electronic Health Record (EHR) is used throughout. Commonly used terms identified by ISO (the International Organization for Standardization) [ 16 ] plus another not identified by ISO are outlined below and used in our search. ISO considers Electronic Health Record (EHR) to be an overall term for “ a repository of information regarding the health status of a subject of care , in computer processable form ” [ 16 ], p. 13. ISO uses different terms to describe various types of EHRs. These include Electronic Medical Record (EMR), which is similar to an EHR but restricted to the medical domain. The terms Electronic Patient Record (EPR) and Computerized Patient Record (CPR) are also identified. Häyrinen et al. [ 17 ] view both terms as having the same meaning and referring to a system that contains clinical information from a particular hospital. Another term seen is Electronic Healthcare Record (EHCR) which refers to a system that contains all the available health information on a patient [ 17 ] and can thus be seen as synonymous with EHR [ 16 ]. A term often found in the literature is Computerized Physician Order Entry (CPOE). Although this term is not mentioned by ISO [ 16 ] or by Häyrinen et al. [ 17 ], we included CPOE for three reasons. First, it is considered by many to be a key hospital-wide function of an EHR system e.g. [ 8 , 18 ]. Second, from a preliminary analysis of our initial results, we found that, from the perspective of the implementation process, comparable issues and factors emerged from both CPOEs and EHRs. Third, the implementation of a comprehensive electronic medical record requires physicians to make direct order entries [ 19 ]. Kaushal et al. define a CPOE as “ a variety of computer - based systems that share the common features of automating the medication ordering process and that ensure standardized , legible , and complete orders ” [ 18 ], p. 1410. Other terms found in the literature were not included in this review as they were considered either irrelevant or too broadly defined. Examples of such terms are Electronic Client Record (ECR), Personal Health Record (PHR), Digital Medical Record (DMR), Health Information Technology (HIT), and Clinical Information System (CIS).

Search strategies

In order for a systematic literature review to be comprehensive, it is essential that all terms relevant to the aim of the research are covered in the search. Further, we need to include relevant synonyms and related terms, both for electronic medical information systems and for hospitals. By adding an * to the end of a term, the search engines pick out other forms, and by adding “ “ around words one ensures that only the complete term is searched for. Further, by including a ? as a wildcard character, every possible combination is included in the search.

The search used three categories of keywords. The first category included the following terms as approximate synonyms for hospital: “hospital*”, “healthcare”, and “clinic*”. The second category concerned implementation and included the term “implement*”. For the third category, electronic medical information systems, the following search terms were used: “Electronic Health Record*”, “Electronic Patient Record*”, “Electronic Medical Record*”, “Computeri?ed Patient Record*”, “Electronic Healthcare Record*”, “Computeri?ed Physician Order Entry”.

This relatively large set of keywords was necessary to ensure that articles were not missed in the search, and required a large number of search strategies to cover all those keywords. As we were seeking papers about the implementation of electronic medical information systems in hospitals , the search strategies included the terms shown in Table  1 .

The following three search engines were chosen based on their relevance to the field and their accessibility by the researcher: Web of knowledge, EBSCO, and The Cochrane Library. Most search engines use several databases but not all of them were relevant for this research as they serve a wide range of fields. Appendix A provides an overview of the databases used. The reference lists included in articles that met the selection criteria were checked for other possibly relevant studies that had not been identified in the database search.

The articles identified from the various search strategies had to be academic peer-reviewed articles if they were to be included in our review. Further, they were assessed and had to satisfy the following criteria to be included: (1) written in English, (2) full text available online, (3) based on primary empirical data, (4) focused on hospital-wide EHR implementation, and (5) meeting established quality criteria. A long list of abstracts was generated, and all of them were independently reviewed by two of the authors. They independently reviewed the abstracts, eliminated duplicates and shortlisted abstracts for detailed review. When opinions differed, a final decision over inclusion was made following a discussion between the researchers.

Data analysis

The quality of the articles that survived this filtering was assessed by the first two authors using the Standard Quality Assessment Criteria for Evaluating Primary Research Papers [ 18 ]. In other words, the quality of the articles was jointly assessed by evaluating whether specific criteria had been addressed, resulting in a rating of 2 (fully addressed), 1 (partly addressed), or 0 (not addressed) for each criteria. Different questions are posed for qualitative and quantitative research and, in the event of a mixed-method study, both questionnaires were used. Papers were included if they received at least half of the total possible points, admittedly a relatively liberal cut-off point given comments in the Standard Quality Assessment Criteria for Evaluating Primary Research Papers [ 20 ].

The next step was to extract the findings of the reviewed articles and to analyze these with the aim of reaching general findings on the implementation of EHR systems in hospitals. Categorizing these general findings can increase clarity. The earlier introduced conceptual model, based on Pettigrew’s framework for understanding strategic change, includes three categories: context (A), content (B), and process (C). As our review is specifically aimed at identifying findings related to the implementation process, possible motives for introducing such a system, as well as its effects and outcomes, are outside its scope. The authors held frequent discussions between themselves to discuss the meaning and the categorization of the general findings.

Paper selection

Applying the 18 search strategies listed in Table  1 with the various search engines resulted in 364 articles being identified. The searches were carried out on 12 March 2013 for search strategies 1–15 and on 18 April 2013 for search strategies 16–18. The latter three strategies were added following a preliminary analysis of the first set of results which highlighted several other terms and descriptions for information technology in health care. Not surprisingly, many duplicates were included in the 364 articles, both within and between search engines. Using the Refworks functions for identifying exact and close duplicates, 160 duplicates were found. However, this procedure did not identify all the duplicates present and the second author carried out a manual check that identified an additional 23 duplicates. When removing duplicates, we retained the link to the first search engine that identified the article and, as the Web of Knowledge was the first search engine used, most articles appear to have stemmed from this search engine. This left 181 different articles which were screened on title and abstract to check whether they met the selection criteria. When this was uncertain, the contents of the paper were further investigated. This screening resulted in just 13 articles that met all the selection criteria. We then performed two additional checks for completeness. First, checking the references of these articles identified another nine articles. Second, as suggested by the referees of this paper, we also used the term “introduc*” instead of “implement*”, together with the other two original categories of terms, and the term “provider” instead of “physician”, as part of CPOE. Each of these two searches identified one additional article (see Table  1 ). Of these resulting 24 articles, two proved to be almost identical so one was excluded, resulting in 23 articles for a final quality assessment.The results of the quality assessment can be found in Appendix B. The results show that two articles failed to meet the quality threshold and so 21 articles remained for in-depth analysis. Figure  2 displays the steps taken in this selection procedure.

Selection procedure.

To provide greater insight into the context and nature of the 21 remaining articles, an overview is provided in Table  2 . All the studies except one were published after 2000. This reflects the recent increase in effort to implement organization-wide information systems, such as EHR systems, and also increasing incentives from governments to make use of EHR systems in hospitals. Of the 21 studies, 14 can be classified as qualitative, 6 as quantitative, and 1 as a mixed-method study. Most studies were conducted in the USA, with eight in various European countries. Teaching and non-teaching hospitals are almost equally the subject of inquiry, and some researchers have focused on specific types of hospitals such as rural, critical access, or psychiatric hospitals. Ten of the articles were in journals with a five-year impact factor in the Journal Citation Reports 2011 database. There is a huge difference in the number of citations but one should never forget that newer studies have had fewer opportunities to be cited.

Theoretical perspectives of reviewed articles

In research, it is common to use theoretical frameworks when designing an academic study [ 41 ]. Theoretical frameworks provide a way of thinking about and looking at the subject matter and describe the underlying assumptions about the nature of the subject matter [ 42 ]. By building on existing theories, research becomes focused in aiming to enrich and extend the existing knowledge in that particular field [ 42 ]. To provide a more thorough understanding of the selected articles, their theoretical frameworks, if present, are outlined in Table  3 .

It is striking that no specific theoretical frameworks have been used in the research leading to 13 of the 21 selected articles. Most articles simply state their objective as gaining insight into certain aspects of EHR implementation (as shown in Table  1 ) and do not use a particular theoretical approach to identify and categorize findings. As such, these articles add knowledge to the field of EHR implementation but do not attempt to extend existing theories.

Aarts et al. [ 21 ] introduce the notion of the sociotechnical approach: emphasizing the importance of focusing both on the social aspects of an EHR implementation and on the technical aspects of the system. Using the concept of emergent change, they argue that an implementation process is far from linear and predictable due to the contingencies and the organizational complexity that influences the process. A sociotechnical approach and the concept of emergent change are also included in the theoretical framework of Takian et al. [ 37 ]. Aarts et al. [ 21 ] elaborate on the sociotechnical approach when stating that the fit between work processes and the information technology determines the success of the implementation. Aarts and Berg [ 22 ] introduce a model of success or failure in information system implementation. They see creating synergy among the medical work practices, the information system, and the hospital organization as necessary for implementation, and argue that this will only happen if sufficient people accept a change in work practices. Cresswell et al.’s study [ 26 ] is also influenced by sociotechnical principles and draws on Actor-Network Theory. Gastaldi et al. [ 28 ] perceive Electronic Health Records as knowledge management systems and question how such systems can be used to develop knowledge assets. Katsma et al. [ 31 ] focus on implementation success and elaborate on the notion that implementation success is determined by system quality and acceptance through participation. As such, they adopt more of a social view on implementation success rather than a sociotechnical approach. Rivard et al. [ 34 ] examine the difficulties in EHR implementation from a cultural perspective. They not only view culture as a set of assumptions shared by an entire collective (an integration perspective) but also expect subcultures to exist (a differentiation perspective), as well as individual assumptions not shared by a specific (sub-) group (fragmentation perspective). Ford et al. [ 27 ] focus on an entirely different topic and investigate the IT adoption strategies of hospitals using a framework that identifies three strategies. These are the single-vendor strategy (in which all IT is purchased from a single vendor), the best-of-breed strategy (integrating IT from multiple vendors), and the best-of-suit strategy (a hybrid approach using a focal system from one vendor as the basis plus other applications from other vendors).

To summarize, the articles by Aarts et al. [ 21 ], Aarts and Berg [ 22 ], Cresswell et al. [ 26 ], and Takian et al. [ 37 ] apply a sociotechnical framework to focus their research. Gastaldi et al. [ 28 ] see EHRs as a means to renew organizational capabilities. Katsma et al. [ 31 ] use a social framework by focusing on the relevance of an IT system as perceived by the user and the participation of users in the implementation process. Rivard et al. [ 34 ] analyze how organizational cultures can be receptive to EHR implementation. Ford et al. [ 27 ] look at adoption strategies, leading them to focus on the selection procedure for Electronic Health Records. The 13 other studies did not use an explicit theoretical lens in their research.

Implementation-related findings

The process of categorization started by assessing whether a specific finding from a study should be placed in Category A, B, or C. Thirty findings were placed in Category A (context), 31 in Category B (content), and 66 in Category C (process). Comparing and combining the specific findings resulted in several general findings within each category. The general findings are each given a code (category character plus number) and the related code is indicated alongside each specific finding in Appendix C. Findings that were only seen in one article, and thus were lacking support, were discarded.

Category A - context

The context category of an EHR implementation process includes both internal variables (such as resources, capabilities, culture, and politics) and external variables (such as economic, political, and social variables). Six general findings were identified, all but one related to internal variables. An overview of the findings and corresponding articles can be found in Table  4 . The lack of general findings related to external variables reflects our decision to exclude the underlying reasons (e.g. political or social pressures) for implementing an EHR system from this review. Similarly, internal findings related to aspects such as perceived financial benefits or improved quality of care, are outside our scope.

A1: Large (or system-affiliated), urban, not-for-profit, and teaching hospitals are more likely to have implemented an EHR system due to having greater financial capabilities, a greater change readiness, and less focus on profit

The research reviewed shows that larger or system-affiliated hospitals are more likely to have implemented an EHR system, and that this can be explained by their easier access to the large financial resources required. Larger hospitals have more financial resources than smaller hospitals [ 30 ] and system-affiliated hospitals can share costs [ 27 ]. Hospitals situated in urban areas more often have an EHR system than rural hospitals, which is attributed to less knowledge of EHR systems and less support from medical staff in rural hospitals [ 29 ]. The fact that not-for-profit hospitals more often have an EHR system fully implemented and teaching hospitals slightly more often than private hospitals is attributed to the latter’s more wait-and-see approach and the more progressive change-ready nature of public and teaching hospitals [ 27 , 32 ].

A2: EHR implementation requires the selection of a mature vendor who is committed to providing a system that fits the hospital’s specific needs

Although this finding is not a great surprise, it is relevant to discuss it further. A hospital selecting its own vendor can ensure that the system will match the specific needs of that hospital [ 32 ]. Further, it is important to deal with a vendor that has proven itself on the EHR market with mature and successful products. The vendor must also be able to identify hospital workflows and adapt its product accordingly, and be committed to a long-term trusting relationship with the hospital [ 33 ]. With this in mind, the initial price of the system should not be the overriding consideration: the organization should be willing to avoid purely cost-oriented vendors [ 28 ], as costs soon mount if problems arise.

A3: The presence of hospital staff with previous experience of health information technology increases the likelihood of EHR implementation as less uncertainty is experienced by the end-users

In order to be able to work with an EHR system, users must be capable of using information technology such as computers and have adequate typing skills [ 19 , 32 ]. Knowledge of, and previous experience with, EHR systems or other medical information systems reduces uncertainty and disturbance for users, and this results in a more positive attitude towards the system [ 29 , 32 , 37 , 38 ].

A4: An organizational culture that supports collaboration and teamwork fosters EHR implementation success because trust between employees is higher

The influence of organizational culture on the success of organizational change is addressed in almost all the popular approaches to change management, as well as in several of the articles in this literature review. Ash et al. [ 23 , 24 ] and Scott et al. [ 35 ] highlight that a strong culture with a history of collaboration, teamwork, and trust between different stakeholder groups minimizes resistance to change. Boyer et al. [ 25 ] suggest creating a favorable culture that is more adaptive to EHR implementation. However, creating a favorable culture is not necessarily easy: a comprehensive approach including incentives, resource allocation, and a responsible team was used in the example of Boyer et al. [ 25 ].

A5: EHR implementation is most likely in an organization with little bureaucracy and considerable flexibility as changes can be rapidly made

A highly bureaucratic organizational structure hampers change: it slows the process and often leads to inter-departmental conflict [ 19 ]. Specifically, appointing a multidisciplinary team to deal with EHR-related issues can prevent conflict and stimulate collaboration [ 25 ].

A6: EHR system implementation is difficult because cure and care activities must be ensured at all times

During the process of implementing an EHR system, it is of the utmost importance that all relevant information is always available [ 28 , 34 , 39 ]. Ensuring the continuity of quality care while implementing an EHR system is difficult and is an important distinction from many other IT implementations.

Category B - content

The content of the EHR implementation process consists of the EHR system and the corresponding objectives, assumptions, and complementary services. Table  5 lists the five extracted general findings. These focus on both the hardware and software of the EHR system, and its relation to work practices and privacy.

B1: Creating a fit by adapting both the technology and work practices is a key factor in the implementation of EHR

This finding elaborates on the sociotechnical approach identified in the earlier section on the theories adopted in the articles. Several authors [ 21 , 26 , 31 , 37 ] make clear that creating a fit between the EHR system and the existing work practices requires an initial acknowledgement that an EHR implementation is not just a technical project and that existing work practices will change due to the new system. By customizing and adapting the system to meet specific needs, users will become more open to using it [ 19 , 26 , 28 ].

B2: Hardware availability and system reliability, in terms of speed, availability, and a lack of failures, are necessary to ensure EHR use

In several articles, authors highlight the importance of having sufficient hardware. A system can only be used if it is available to the users, and a system will only be used if it works without problems. Ash et al. [ 24 ], Scott et al. [ 35 ], and Weir et al. [ 19 ] refer to the speed of the system as well as to the availability of a sufficient number of adequate terminals see also [ 40 ] in various locations. Systems must be logically structured [ 29 ], reliable [ 32 ], and provide safe information access [ 37 ]. Boyer et al. [ 25 ] also mention the importance of technical aspects but add that these are not sufficient for EHR implementation.

B3: To ensure EHR implementation, the software needs to be user-friendly with regard to ease of use, efficiency in use, and functionality

Some authors distinguish between technical availability and reliability, and the user-friendliness of the software [ 19 , 24 , 32 ]. They argue that it is not sufficient for a system to be available and reliable, it should also be easy and efficient in use, and provide the functionality required for medical staff to give good care. If a system fails to do this, staff will not use the system and will stick to their old ways of working.

B4: An EHR implementation should contain adequate safeguards for patient privacy and confidentiality

Concerns over privacy and confidentiality are recognized by Boyer et al. [ 25 ] and Houser and Johnson [ 29 ] and are considered as a barrier to EHR implementation. Yoon-Flannery et al. [ 40 ] and Takian et al. [ 37 ] also recognize the importance of patient privacy and the need to address this issue by providing training and creating adequate safeguards.

B5: EHR implementation requires a vendor who is willing to adapt its product to hospital work processes

A vendor must be responsive and enable the hospital to develop its product to ensure a good and usable EHR system [ 32 , 33 ]. By so doing, dependence on the vendor decreases and concerns that arise within the hospital can be addressed [ 32 ]. This finding is related to A2 in the sense that an experienced, cooperative, and flexible vendor is needed to deal with the range of interest groups found in hospitals.

Category C - process

This category refers to the actual process of implementing the EHR system. Variables considered are time, change approach, and change management. In our review, this category produced the largest number of general findings (see Table  6 ), as might be expected given our focus on the implementation process. EHR implementation often leads to anxiety, uncertainty, and concerns about a possible negative impact of the EHR on work processes and quality. The process findings, including leadership, resource availability, communication and participation are explicitly aimed at overcoming resistance to EHR implementation. These interventions help to create a positive atmosphere of goal directedness, co-creation and partnership.

C1: Due to their influential position, management’s active involvement and support is positively associated with EHR implementation, and also counterbalances the physicians’ medical dominance

Several authors note the important role that managers play in EHR implementation. Whereas some authors refer to supportive leadership [ 19 , 24 ], others emphasize that strong and active management involvement is needed [ 25 , 32 – 35 ]. Strong leadership is relevant as it effectively counterbalances the physicians’ medical dominance. For instance, Rivard et al. [ 34 ] observe that physicians’ medical dominance and the status and autonomy of other health professionals hinder collaboration and teamwork, and that this complicates EHR implementation. Poon et al. [ 33 ] acknowledge this aspect and argue for strong leadership in order to deal with the otherwise dominant physicians. They also claim that leaders have to set an example and use the system themselves. At the same time, it is motivating that the implementation is managed by leaders who are recognized by the medical staff, for instance by head nurses and physicians or by former physicians and nurses [ 25 , 33 ]. Ovretveit et al. [ 32 ] argue that it helps the implementation if senior management repeatedly declares the EHR implementation to be of the highest priority and supports this with sufficient financial and human resources. Poon et al. [ 33 ] add to this by highlighting that, especially during uncertainties and setbacks, the common vision that guides the EHR implementation has to be communicated to hospital staff. Sufficient human resources include the selection of competent and experienced project leaders who are familiar with EHR implementation. Scott et al. [ 35 ] identify leadership styles for different phases: participatory leadership is valued in selection decisions, whereas a more hierarchical leadership style is preferable in the actual implementation.

C2: Participation of clinical staff in the implementation process increases support for and acceptance of the EHR implementation

Participation of end-users (the clinical staff) generates commitment and enables problems to be quickly solved [ 25 , 26 , 36 ]. Especially because it is very unlikely that the system will be perfect for all, it is important that the clinical staff become the owner, rather than customers, of the system. Clinical staff should participate at all levels and in all steps [ 19 , 28 , 32 , 36 ] from initial system selection onwards [ 35 ]. Ovretveit et al. [ 32 ] propose that this involvement should have an extensive timeframe, starting in the early stages of implementation, when initial vendor requirements are formulated (‘consultation before implementation’), through to the beginning of the use phase. Creating multidisciplinary work groups which determine the content of the EHR and the rules regarding the sharing of information contributes to EHR acceptance [ 25 ] and ensures realistic approaches acceptable to the clinical staff [ 36 ].

C3: Training end-users and providing real-time support is important for EHR implementation success

Frequently, the end-users of a new EHR system lack experience with the specific EHR system or with EHR systems in general. Although it is increasingly hard to imagine society or workplaces without IT, a large specific system, such as an EHR, still requires considerable training on how to use it properly. The importance of training is often underestimated, and inadequate training will create a barrier to EHR use [ 19 , 29 ]. Consequently, adequate training, of appropriate quantity and quality, must be provided at the right times and locations [ 19 , 32 , 36 ]. Simon et al. [ 36 ] add to this the importance of real-time support, preferably provided by peers and super-users.

C4: A comprehensive implementation strategy, offering both clear guidance and room for emergent change, is needed for implementing an EHR system

Several articles highlight aspects of an EHR implementation strategy. A good strategy facilitates EHR implementation [ 19 , 25 ] and consists of careful planning and preparation [ 36 ], a sustainable business plan, effective communication [ 28 , 40 ] and mandatory implementation [ 19 ]. Emergent change is perceived as a key characteristic of EHR implementation in complex organizations such as hospitals [ 21 ], and this suggests an implementation approach based on a development paradigm [ 31 ], which may initially even involve parallel use of paper [ 26 ]. The notion of emergent change has been variously applied, including in the theoretical frameworks of Aarts et al. [ 21 ] and Katsma et al. [ 31 ]. These studies recognize that EHR implementation is relatively unpredictable due to unforeseen contingencies for which one cannot plan. With their emphasis on emergent change with unpredictable outcomes, Aarts et al. [ 21 ] make a case for acknowledging that unexpected and unplanned contingencies will influence the implementation process. They argue that the changes resulting from these contingencies often manifest themselves unexpectedly and must then be dealt with. Additionally, Takian et al. [ 37 ] state that it is crucial to contextualize an EHR implementation so as to be better prepared for unexpected changes.

C5: Establishing an interdisciplinary implementation group consisting of developers, members of the IT department, and end-users fosters EHR implementation success

In line with the arguments for management support and for the participation of clinical staff, Ovretveit et al. [ 32 ], Simon et al. [ 36 ] and Weir et al. [ 19 ] build a case for using an interdisciplinary implementation group. By having all the direct stakeholders working together, a better EHR system can be delivered faster and with fewer problems.

C6: Resistance of clinical staff, in particular of physicians, is a major barrier to EHR implementation, but can be reduced by addressing their concerns

Clinical staff’s attitude is a crucial factor in EHR implementation [ 36 ]. Particularly, the physicians constitute an important group in hospitals. As such, their possible resistance to EHR implementation will form a major barrier [ 29 , 33 ] and may lead to workarounds [ 26 ]. Whether physicians accept or reject an EHR implementation depends on their acceptance of their work practices being transformed [ 22 ]. The likelihood of acceptance will be increased if implementers address the concerns of physicians [ 24 , 28 , 32 , 33 ], but also of other members of clinical staff [ 36 ].

C7: Identifying champions among clinical staff reduces resistance

The previous finding already elaborated on clinical staff resistance and suggested reducing this by addressing their concerns. Another way to reduce their resistance is related to the process of implementation and involves identifying physician champions, typically physicians that are well respected due to their knowledge and contacts [ 32 , 33 ]. Simon et al. [ 36 ] emphasize the importance of identifying champions among each stakeholder group. These champions can provide reassurance to their peers.

C8: Assigning a sufficient number of staff and other resources to the EHR implementation process is important in adequately implementing the system

Implementing a large EHR system requires considerable resources, including human ones. Assigning appropriate people, such as super-users [ 36 ] and a sufficient number of them to that process will increase the likelihood of success [ 19 , 32 , 33 , 36 ]. Further, it is important to have sufficient time and financial resources [ 26 , 32 ]. This finding is also relevant in relation to finding A6 (ensuring good care during organizational change).

These 19 general findings have been identified from the individual findings within the 20 analyzed articles. These findings are all related to one of the three main and interacting dimensions of the framework: six to context, five to content, and eight to process. This identification and explanation of the general findings concludes the results section of this systematic literature review and forms the basis for the discussion below.

This review of the existing academic literature sheds light on the current knowledge regarding EHR implementation. The 21 selected articles all originate from North America or Europe, perhaps reflecting a greater governmental attention to EHR implementation in these regions and, of course, our inclusion of only articles written in English. Two articles were rejected for quality reasons [ 43 , 44 ], see Appendix B. All but one of the selected articles have been published since 2000, reflecting the growing interest in implementing EHR systems in hospitals. Eight articles built their research on a theoretical framework, four of which use the same general lens of the sociotechnical approach [ 21 , 22 , 26 , 37 ]. Katsma et al. [ 31 ] and Rivard et al. [ 34 ] focus more on the social and cultural aspects of EHR implementation, the former on the relevance for, and participation of, users, the latter on three different cultural perspectives. Ford et al. [ 27 ] researched adoption strategies for EHR systems and Gastaldi et al. [ 26 ] consider them as a means to renew organizational capabilities. It is notable that the other reviewed articles did not use a theoretical framework to analyze EHR implementation and made no attempt to elaborate on existing theories.

A total of 127 findings were extracted from the articles, and these findings were categorized using Pettigrew’s framework for strategic change [ 13 ] as a conceptual model including the three dimensions of context, content, and process. To ensure a tight focus, the scope of the review was explicitly limited to findings related to the EHR implementation process, thus excluding the reasons for, barriers to, and outcomes of an EHR implementation.

Some of the findings require further interpretation. Contextual finding A1 relates to the demographics of a hospital. One of the assertions is that privately owned hospitals are less likely than public hospitals to invest in an EHR. The former apparently perceive the costs of EHR implementation to outweigh the benefits. This seems remarkable given that there is a general belief that information technology increases efficiency and reduces process costs, so more than compensating for the high initial investments. It is however important to note that the literature on EHR is ambivalent when it comes to efficiency; several authors record a decrease in the efficiency of work practices [ 25 , 33 , 35 , 38 ], whereas others mention an increase [ 29 , 31 ]. Finding A2 is a reminder of the importance of carefully selecting an appropriate vendor, taking into account experience with the EHR market and the maturity of their products rather than, for example, focussing on the cost price of the system. Given the huge investment costs, the price of an EHR system tends to have a major influence on vendor selection, an aspect that is also promoted by the current European tendering regulations that oblige (semi-) public institutions, like many hospitals, to select the lowest bidder, or the bidder that is economically the most preferable [ 45 ]. The finding that EHR system implementation is difficult because good medical care needs to be ensured at all times (A6) also deserves mention. Essentially, many system implementations in hospitals are different from IT implementations in other contexts because human lives are at stake in hospitals. This not only complicates the implementation process because medical work practices have to continue, it also requires a system to be reliable from the moment it is launched.

The findings regarding the content of the EHR system (Category B) highlight the importance of a suitable software product. A well-defined selection process of the software package and its associated vendor (discussed in A2) is seen as critical (B5). Selection should be based on a careful requirements analysis and an analysis of the experience and quality of the vendor. An important requirement is a sufficient degree of flexibility to customize and adapt the software to meet the needs of users and the work practices of the hospital (finding B1). At the same time the software product should challenge the hospital to rethink and improve its processes. A crucial condition for the acceptance by the diverse user groups of hospitals is the robustness of the EHR system in terms of availability, speed, reliability and flexibility (B2). This also requires adequate hardware in terms of access to computers, and mobile equipment to enable availability at all the locations of the hospital. Perceived ease of use of the system (B4) and the protection of patients’ privacy (B4) are other content factors that can make or break EHR implementation in hospitals.

The findings on the implementation process, our Category C, highlight four aspects that are commonly mentioned in change management approaches as important success factors in organizational change. The active involvement and support of management (C1), the participation of clinical staff (C2), a comprehensive implementation strategy (C4), and using an interdisciplinary implementation group (C5) correspond with three of the ten guidelines offered by Kanter et al. [ 46 ]. These three guidelines are: (1) support a strong leader role; (2) communicate, involve people, and be honest; and (3) craft an implementation plan. As the implementation of an EHR system is an organizational change process it is no surprise that these commonalities are identified in several of the analyzed articles. Three Category C findings (C2, C6, and C7) concern dealing with clinical staff given their powerful positions and potential resistance. Physicians are the most influential medical care providers, and their resistance can delay an EHR implementation [ 23 ], lead to at least some of it being dropped [ 21 , 22 , 34 ], or to it not being implemented at all [ 33 ]. Thus, there is ample evidence of the crucial importance of physicians’ acceptance of an EHR for it to be implemented. This means that clinicians and other key personnel should be highly engaged and motivated to contribute to EHR. Prompt feedback on requests, and high quality support during the implementation, and an EHR that clearly supports clinical work are key issues that contribute to a motivated clinical staff.

Analyzing and comparing the findings enables us to categorize them in terms of subject matter (see Table  7 ). By categorizing the findings in terms of subject, and by totaling the number of articles related to the individual findings on that subject, one can deduce how much attention has been given in the literature to the different topics. This analysis highlights that the involvement of physicians in the implementation process, the quality of the system, and a comprehensive implementation strategy are considered the crucial elements in EHR implementation.

Notwithstanding the useful results, this review and analysis has some limitations. Although we carefully developed and executed the search strategy, we cannot be sure that we found all the relevant articles. Since we focused narrowly on keywords, and these had to be part of an article’s title, we could have excluded relevant articles that used different terminology in their titles. Although searching the reference lists of identified articles did result in several additional articles, some relevant articles might still have been missed. Another limitation is the exclusion of publications in languages other than English. Further, the selection and categorization of specific findings, and the subsequent extraction of general findings, is subjective and depends on the interpretations of the authors, and other researchers might have made different choices. A final limitation is inherent to literature reviews in that the authors of the studies included may have had different motives and aims, and used different methods and interpretative means, in drawing their conclusions.

The existing literature fails to provide evidence of there being a comprehensive approach to implementing EHR systems in hospitals that integrates relevant aspects into an ‘EHR change approach’. The literature is diffuse, and articles seldom build on earlier ones to increase the theoretical knowledge on EHR implementation, notable exceptions being Aarts et al. [ 21 ], Aarts and Berg [ 22 ], Cresswell et al. [ 26 ], and Takian et al. [ 37 ]. The earlier discussion on the various results summarizes the existing knowledge and reveals gaps in the knowledge associated with EHR implementation. The number of EHR implementations in hospitals is growing, as well as the body of literature on this subject. This systematic review of the literature has produced 19 general findings on EHR implementation, which were each placed in one of three categories. A number of these general findings are in line with the wider literature on change management, and others relate to the specific nature of EHR implementation in hospitals.

The findings presented in this article can be viewed as an overview of important subjects that should be addressed in implementing an EHR system. It is clear that EHR systems have particular complexities and should be implemented with great care, and with attention given to context, content, and process issues and to interactions between these issues. As such, we have achieved our research goal by creating a systematic review of the literature on EHR implementation. This paper’s academic contribution is in providing an overview of the existing literature with regard to important factors in EHR implementation in hospitals. Academics interested in this specific field can now more easily access knowledge on EHR implementation in hospitals and can use this article as a starting point and build on the existing knowledge. The managerial contribution lies in the general findings that can be applied as guidelines when implementing EHR in hospitals. We have not set out to provide a single blueprint for implementing an EHR system, but rather to provide guidelines and to highlight points that deserve attention. Recognizing and addressing these aspects can increase the likelihood of getting an EHR system successfully implemented.

Appendix A - List of databases

This appendix provides an overview of all databases included in the used search engines. The databases in italic were excluded for the research as these databases focus on fields not relevant for the subject of EHR implementations.

Web of Knowledge

Web of Science

Biological Abstracts

Journal Citation Reports

Academic Search Premier

AMED - The Allied and Complementary Medicine Database

America : History & Life

American Bibliography of Slavic and East European Studies

Arctic & Antarctic Regions

Art Full Text ( H.W. Wilson )

Art Index Retrospective ( H.W. Wilson )

ATLA Religion Database with ATLASerials

Business Source Premier

Communication & Mass Media Complete

eBook Collection ( EBSCOhost )

Funk & Wagnalls New World Encyclopedia

Historical Abstracts

L ’ Annéephilologique

Library, Information Science & Technology Abstracts

MAS Ultra - School Edition

Military & Government Collection

MLA Directory of Periodicals

MLA International Bibliography

New Testament Abstracts

Old Testament Abstracts

Philosopher ’ s Index

Primary Search

PsycARTICLES

PsycCRITIQUES

Psychology and Behavioral Sciences Collection

Regional Business News

Research Starters - Business

RILM Abstracts of Music Literature

The Cochrane Library

Cochrane Database of Systematic Reviews

Cochrane Central Register of Controlled Trials

Cochrane Methodology Register

Database of Abstracts of Reviews of Effects

Health Technology Assessment Database

NHS Economic Evaluation Database

About The Cochrane Collaboration

Appendix B - Quality assessment

The quality of the articles was assessed with the Standard Quality Assessment Criteria for Evaluating Primary Research Papers [ 18 ]. Assessment was done by questioning whether particular criteria had been addressed, resulting in a rating of 2 (completely addressed), 1 (partly addressed), or 0 (not addressed) points. Table  8 provides the overview of the scores of the articles, (per question) for qualitative studies; Table  9 for quantitative studies; and Table  10 for mixed methods studies. Articles were included if they scored 50% or higher of the total amount of points possible. Based on this assessment, two articles were excluded from the search.

Appendix C - All findings

Table  11 displays all findings from the selected articles. The category number is related to the general finding as discussed in the Results section.

Abramson EL, McGinnis S, Edwards A, Maniccia DM, Moore J, Kaushal R: Electronic health record adoption and health information exchange among hospitals in New York State. J Eval Clin Pract. 2011, 18: 1156-1162.

Article   PubMed   Google Scholar  

Robertson A, Cresswell K, Takian A, Petrakaki D, Crowe S, Cornford T, Sheikh A: Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. Br Med J. 2010, 341: c4564-10.1136/bmj.c4564.

Article   Google Scholar  

Rigsrevisionen: Extract from the report to the Public Accounts Committee on the implementation of electronic patient records at Danish hospitals. 2011, http://uk.rigsrevisionen.dk/media/1886186/4-2010.pdf , 2011

Google Scholar  

Hartswood M, Procter R, Rouncefield M, Slack R: Making a Case in Medical Work: Implications for the Electronic Medical Record. Comput Supported Coop Work. 2003, 12: 241-266. 10.1023/A:1025055829026.

Grimson J, Grimson W, Hasselbring W: The SI Challenge in Health Care. Commun ACM. 2000, 43 (6): 49-55.

Mantzana V, Themistocleous M, Irani Z, Morabito V: Identifying healthcare actors involved in theadoption of information systems. Eur J Inf Syst. 2007, 16: 91-102. 10.1057/palgrave.ejis.3000660.

Boonstra A, Boddy D, Bell S: Stakeholder management in IOS projects: analysis of an attempt to implement an electronic patient file. Eur J Inf Syst. 2008, 17 (2): 100-111. 10.1057/ejis.2008.2.

Jha A, DesRoches CM, Campbell EG, Donelan K, Rao SR, Ferris TF, Shields A, Rosenbaum S, Blumenthal D: Use of Electronic Health Records in US hospitals. N Engl J Med. 2009, 360: 1628-1638. 10.1056/NEJMsa0900592.

Article   CAS   PubMed   Google Scholar  

Heeks R: Health information systems: Failure, success and improvisation. Int J Med Inform. 2006, 75: 125-137. 10.1016/j.ijmedinf.2005.07.024.

Boonstra A, Govers MJ: Understanding ERP system implementation in a hospital by analysing stakeholders. N Technol Work Employ. 2009, 24 (2): 177-193. 10.1111/j.1468-005X.2009.00227.x.

Keshavjee K, Bosomworth J, Copen J, Lai J, Kucukyazici B, Liani R, Holbrook AM: Best practices in EMR implementation: a systematic review. Proceed of the 11th International Symposium on Health Information Mangement Research – iSHIMR. 2006, 1-15.

McGinn CA, Grenier S, Duplantie J, Shaw N, Sicotte C, Mathieu L, Leduc Y, Legare F, Gagnon MP: Comparison of use groups perspectives of barriers and facilitator to implementing EHR – a systematic review. BMC Med. 2011, 9: 46-10.1186/1741-7015-9-46.

Article   PubMed   PubMed Central   Google Scholar  

Pettigrew AM: Context and action in the transformation of the firm. J Manag Stud. 1987, 24 (6): 649-670. 10.1111/j.1467-6486.1987.tb00467.x.

Hartley J: Case Study Research. Chapter 26. Essential Guide to Qualitative Methods in Organizational Research. Edited by: Cassel C, Symon G. 2004, London: Sage

Hage E, Roo JP, Offenbeek MAG, Boonstra A: Implementation factors and their effect on e-health service adoption in rural communities: a systematic literature review. BMC Health Serv Res. 2013, 13 (19): 1-16.

ISO: Health informatics: Electronic health record - Definition, scope and context. Draft Tech Report. 2004, 03-16. ISO/DTR 20514. available at https://www.iso.org/obp/ui/#iso:std:iso:tr:20514:ed-1:v1:en

Häyrinen K, Saranto K, Nykänen P: Definition, structure, content, use and impacts of electronic health records: A review of the research literature. Int J Med Inform. 2008, 77: 291-304. 10.1016/j.ijmedinf.2007.09.001.

Kaushal R, Shojania KG, Bates DW: Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003, 163 (12): 1409-1416. 10.1001/archinte.163.12.1409.

Weir C, Lincoln M, Roscoe D, Turner C, Moreshead G: Dimensions associated with successful implementation of a hospital based integrated order entry system. Proc Annu Symp Comput Appl [Sic] in Med Care Symp Comput Appl Med Care. 1994, 653: 7.

Kmet LM, Lee RC, Cook LS: Standard quality assessment criteria for evaluating primary research papers from a variety of fields. 2004, Alberta Heritage Foundation for Medical Research, http://www.ihe.ca/documents/HTA-FR13.pdf .

Aarts J, Doorewaard H, Berg M: Understanding implementation: The case of a computerized physician order entry system in a large dutch university medical center. J Am Med Inform Assoc. 2004, 11 (3): 207-216. 10.1197/jamia.M1372.

Aarts J, Berg M: Same systems, different outcomes - Comparing the implementation of computerized physician order entry in two Dutch hospitals. Methods Inf Med. 2006, 45 (1): 53-61.

CAS   PubMed   Google Scholar  

Ash J, Gorman P, Lavelle M, Lyman J, Fournier L: Investigating physician order entry in the field: lessons learned in a multi-center study. Stud Health Technol Inform. 2001, 84 (2): 1107-1111.

Ash JS, Gorman PN, Lavelle M, Payne TH, Massaro TA, Frantz GL, Lyman JA: A cross-site qualitative study of physician order entry. J Am Med Inform Assoc. 2003, 10 (2): 188-200. 10.1197/jamia.M770.

Boyer L, Samuelian J, Fieschi M, Lancon C: Implementing electronic medical records in a psychiatric hospital: A qualitative study. Int J Psychiatry Clin Pract. 2010, 14 (3): 223-227. 10.3109/13651501003717243.

Cresswell KM, Worth A, Sheikh A: Integration of a nationally procured electronic health record system into user work practices. BMC Med Inform Decis Mak. 2012, 12: 15-10.1186/1472-6947-12-15.

Ford EW, Menachemi N, Huerta TR, Yu F: Hospital IT Adoption Strategies Associated with Implementation Success: Implications for Achieving Meaningful Use. J Healthc Manag. 2010, 55 (3): 175-188.

PubMed   Google Scholar  

Gastaldi L, Lettieri E, Corso M, Masella C: Performance improvement in hospitals: leveraging on knowledge assets dynamics through the introduction of an electronic medical record. Meas Bus Excell. 2012, 16 (4): 14-30. 10.1108/13683041211276410.

Houser SH, Johnson LA: Perceptions regarding electronic health record implementation among health information management professionals in Alabama: a statewide survey and analysis. Perspect Health Inf Manage/AHIMA, Am Health Inf Manage Assoc. 2008, 5: 6-6.

Jaana M, Ward MM, Bahensky JA: EMRs and Clinical IS Implementation in Hospitals: A Statewide Survey. J Rural Health. 2012, 28: 34-43. 10.1111/j.1748-0361.2011.00386.x.

Katsma CP, Spil TAM, Ligt E, Wassenaar A: Implementation and use of an electronic health record: measuring relevance and participation in four hospitals. Int J Healthc Technol Manag. 2007, 8 (6): 625-643. 10.1504/IJHTM.2007.014194.

Ovretveit J, Scott T, Rundall TG, Shortell SM, Brommels M: Improving quality through effective implementation of information technology in healthcare. Int J Qual Health Care. 2007, 19 (5): 259-266. 10.1093/intqhc/mzm031.

Poon EG, Blumenthal D, Jaggi T, Honour MM, Bates DW, Kaushal R: Overcoming barriers to adopting and implementing computerized physician order entry systems in US hospitals. Health Aff. 2004, 23 (4): 184-190. 10.1377/hlthaff.23.4.184.

Rivard S, Lapointe L, Kappos A: An Organizational Culture-Based Theory of Clinical Information Systems Implementation in Hospitals. J Assoc Inf Syst. 2011, 12 (2): 123-162.

Scott JT, Rundall TG, Vogt TM, Hsu J: Kaiser Permanente’s experience of implementing an electronic medical record: a qualitative study. Br Med J. 2005, 331 (7528): 1313-1316. 10.1136/bmj.38638.497477.68.

Simon SR, Keohane CA, Amato M, Coffey M, Cadet M, Zimlichman E: Lessons learned from implementation of computerized provider order entry in 5 community hospitals: a qualitative study. BMC Med Inform Decis Mak. 2013, 13: 67-10.1186/1472-6947-13-67.

Takian A, Sheikh A, Barber N: We are bitter, but we are better off: case study of the implementation of an electronic health record system into a mental health hospital in England. BMC Health Serv Res. 2012, 12: 484-10.1186/1472-6963-12-484.

Ward MM, Vartak S, Schwichtenberg T, Wakefield DS: Nurses’ Perceptions of How Clinical Information System Implementation Affects Workflow and Patient Care. Cin-Comput Inform Nurs. 2011, 29 (9): 502-511. 10.1097/NCN.0b013e31822b8798.

Ward MM, Vartak S, Loes JL, O’Brien J, Mills TR, Halbesleben JRB, Wakefield DS: CAH Staff Perceptions of a Clinical Information System Implementation. Am J Manage Care. 2012, 18 (5): 244-252.

Yoon-Flannery K, Zandieh SO, Kuperman GJ, Langsam DJ, Hyman D, Kaushal R: A qualitative analysis of an electronic health record (EHR) implementation in an academic ambulatory setting. Inform Prim Care. 2008, 16 (4): 277-284.

Van Aken J, Berends H, Van der Bij H: Problem solving in organizations. 2012, New York, USA: Cambridge University Press

Book   Google Scholar  

Botha ME: Theory development in perspective: the role of conceptual frameworks and models in theory development. J Adv Nurs. 1989, 14 (1): 49-55. 10.1111/j.1365-2648.1989.tb03404.x.

Spetz J, Keane D: Information Technology Implementation in a Rural Hospital: A Cautionary Tale. J Healthc Manag. 2009, 54 (5): 337-347.

Massaro TA: Introducing Physician Order Entry at a Major Academic Medical-Center. Impact Organ Culture Behav Acad Med. 1993, 68 (1): 20-25.

CAS   Google Scholar  

Lundberg S, Bergman M: Tender evaluation and supplier selection methods in public procurement. J Purch Supply Manage. 2013, 19 (2): 73-83. 10.1016/j.pursup.2013.02.003.

Kanter RM, Stein BA, Jick TD: The Challenge of Organizational Change. 1992, New York, USA: Free Press

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1472-6963/14/370/prepub

Download references

Acknowledgement

We acknowledge the Master degree program Change Management at the University of Groningen for supporting this study. We also thank the referees for their valuable comments.

Author information

Authors and affiliations.

Faculty of Economics and Business, University of Groningen, Groningen, The Netherlands

Albert Boonstra & Janita F J Vos

Deloitte Consulting, Amsterdam, The Netherlands

Arie Versluis

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Albert Boonstra .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

AB and JV established the research design and made significant contributions to the interpretation of the results. They supervised AV throughout the study, and participated in writing the final version of this paper. AV contributed substantially to the selection and analysis of included papers, and wrote a preliminary draft of this article. All authors have read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Boonstra, A., Versluis, A. & Vos, J.F.J. Implementing electronic health records in hospitals: a systematic literature review. BMC Health Serv Res 14 , 370 (2014). https://doi.org/10.1186/1472-6963-14-370

Download citation

Received : 23 September 2013

Accepted : 11 August 2014

Published : 04 September 2014

DOI : https://doi.org/10.1186/1472-6963-14-370

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical Staff
• Electronic Health Record
• Electronic Patient Record
• Health Information Technology
• Computerize Physician Order Entry

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Download PDF
• Share X Facebook Email LinkedIn
• Permissions

A Window Into Inpatient Health Care Delivery Through Secure Message Logs—Tracing the Latest Breadcrumbs of the Electronic Health Record

• 1 University of California at San Francisco Health System, San Francisco
• 2 University of California at San Francisco School of Medicine, San Francisco
• 3 Center for Physician Experience and Practice Excellence, Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
• 4 Eisenberg Family Depression Center, University of Michigan, Ann Arbor
• 5 Molecular and Behavioral Neuroscience Institute, University of Michigan, Ann Arbor
• Research Letter Electronic Health Record Messaging Patterns of Health Care Professionals in Inpatient Medicine William Small, MD, MBA; Eduardo Iturrate, MD; Jonathan Austrian, MD; Nicholas Genes, MD, PhD JAMA Network Open

A 2023 study by Barata et al 1 found that across 14 hospitals and 263 outpatient clinics, the volume of messages delivered via the an electronic health record (EHR) secure chat platform increased by 29% from July 2022 to January 2023, 1 highlighting the increasing use and centrality of these platforms as a communication tool for inpatient medical teams. With this context, the study by Small et al 2 provides initial, tantalizing insight into the potential of using analysis of secure messaging platforms to gain insights into the functioning of inpatient care teams. Small et al 2 demonstrate the feasibility of mining this data source to characterize the quantity of messages sent relative to inpatient encounters, patterns of messages sent and received, as well as the time course of messaging during a typical week and hospitalization.

The study by Small et al 2 adds to a growing body of literature that leverages data from EHRs to understand patterns of clinician interactions with the EHR in outpatient 3 and inpatient 4 settings, as well as how that time is associated with outcomes for patients. 5 While prior work has highlighted clinicians interactions with a diversity of EHR functions, the study by Small et al 2 focused specifically on communication among team members. The study by Small et al 2 highlighted that nurses sent the largest proportion of patient messages overall, but that per-person message volume was highest for residents, advanced practice clinicians, and attendings. While traditionally, understanding the sociology of clinical collaboration required direct observations or recordings of care delivery, the data presented by Small et al 2 highlight the new opportunities provided by EHR logs for characterizing interactions among team members.

In addition to helping to elucidate who is part of a clinical team, analyses of patterns and content of secure messages could help facilitate at-scale identification of areas for improved team functioning, delivery of clinical care, and design of EHR tools. Opportunities for improved team functioning could range from opportunities to include additional relevant team members in a conversation to assessment of occasions for improved closed-loop communication. In the future, as analyses of secure messaging text become more prevalent, message content could be used as a basis for quality improvement efforts, such as identifying instances of unclear or ambiguous communication regarding orders or clinical decisions. Additionally, dynamic assessment of message patterns and quantity could elucidate which team members are bearing the greatest burden of the EHR in their clinical work and thus whom EHR optimization efforts should focus on at specific time points.

As depicted by the differential message density per patient across time points during an admission, 2 analysis of message patterns also provides important insight into the workload of inpatient care. While the work of largely cognitive- and coordination-based specialties, such as hospital medicine, traditionally has been more difficult to quantify than the work involved in a surgery or procedure, the information provided by secure messaging logs has the potential to change this paradigm. Messaging logs can help identify the hundreds, if not thousands, of interactions that characterize a thorough and thoughtful admission and, ultimately, the hundreds, if not thousands, of interactions per patient that ensure a safe discharge. Over time, these data could help make an enhanced, data-driven case for the appropriate resources needed to shepherd patients with diverse medical profiles through a successful hospitalization.

Finally, the patterns and content of secure EHR messages have the potential to provide added insight into mechanisms for disparities in patient outcomes. There are known differential health care outcomes for racially and ethnically minoritized patients, such as eg Black or Hispanic patients, cared for in the hospital setting. 6 The mechanisms for these differences are not well understood, and a study by Yan et al 4 previously described that minoritized racial and ethnic groups at 2 academic medical centers were less likely to have high levels of engagement with their EHRs than White patients. In contrast, hospitalized patients of female physicians have been shown to have better outcomes, with the mechanisms for these differences also inadequately characterized. 7 By providing detailed insight into clinical decision-making, care patterns, and clinical teams’ attention, the cadence and content of secure EHR messaging logs have the potential to help us understand how these differences, which can have both positive and negative consequences, arise.

Overall, the work by Small and team 2 is just the beginning. Complementing data from EHR audit logs, the insights derived from secure messaging could help us improve multiple facets of care delivery, from team functioning to team resourcing and disparities in care, if thoughtfully and strategically leveraged.

Published: December 26, 2023. doi:10.1001/jamanetworkopen.2023.49094

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2023 Rotenstein LS et al. JAMA Network Open .

Corresponding Author: Lisa S. Rotenstein, MD, MBA, MSc, Center for Physician Experience and Practice Excellence, Division of General Internal Medicine, Brigham and Women’s Hospital, 75 Francis St, Boston, MA 02215 ( [email protected] ).

Conflict of Interest Disclosures: Dr Rotenstein reported receiving grants from Physicians Foundation, American Medical Association, and FeelBetter; personal fees from Phreesia; and serving on an advisory board for Augmedix AI outside the submitted work. No other disclosures were reported.

See More About

Rotenstein LS , Sen S. A Window Into Inpatient Health Care Delivery Through Secure Message Logs—Tracing the Latest Breadcrumbs of the Electronic Health Record. JAMA Netw Open. 2023;6(12):e2349094. doi:10.1001/jamanetworkopen.2023.49094

Manage citations:

© 2024

Select Your Interests

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

The Role of Electronic Health Records to Identify Risk Factors for Developing Long COVID: A Scoping Review

• Conference paper
• First Online: 13 May 2024
• Cite this conference paper

• Ema Santos 14 ,
• Afonso Fernandes 14 ,
• Manuel Graça 14 &
• Nelson Pacheco Rocha   ORCID: orcid.org/0000-0003-3801-7249 15  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 986))

Included in the following conference series:

• World Conference on Information Systems and Technologies

Coronavirus disease 2019 (COVID-19) was considered a global pandemic from December 2019 to May 2023. A subset of COVID-19 patients develops long-lasting sequelae, commonly referred to as long COVID. This scoping review aimed to identify risk factors for long COVID reported in multiple studies and to determine the role of the secondary use of Electronic Health Records to identify these risk factors. An electronic search was conducted on Scopus, PubMed, and Web of Science, and 46 studies were included in this review after the selection process. Thirty-one risk factors were identified, with the most referred ones being female sex, age, severity of infection and obesity. In terms of data collection, Electronic Health Records were used by 63.0% of the studies, although only 21.7% were retrospective studies exclusively based on the secondary use of Electronic Health Records data. These results show that the potential of clinical research based on the secondary use of data collected from Electronic Health Records is not yet fully achieved, despite the respective advantages when compared with other data collection methods such as remote surveys.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Shah, W., Hillman, T., Playford, E.D., Hishmeh, L.: Managing the long term effects of covid-19: summary of NICE, SIGN, and RCGP rapid guideline. BMJ 372 (2021)

Google Scholar  

Davis, H.E., McCorkell, L., Vogel, J.M., Topol, E.J.: Long COVID: major findings, mechanisms and recommendations. Nat. Rev. Microbiol. 21 , 133–146 (2023)

Article   Google Scholar  

Koc, H.C., Xiao, J., Liu, W., Li, Y., Chen, G.: Long COVID and its Management. Int. J. Biol. Sci. 18 (12), 4768–4780 (2022)

Williams, J.G.: The use of clinical information to help develop new services in a district general hospital. Int. J. Med. Informatics 56 (1–3), 151–159 (1999)

Näher, A.F., et al.: Secondary data for global health digitalisation. Lancet Digital Health, 5 (2), e93-e101 (2023)

Fernández-de-Las-Peñas, C., et al.: Diabetes and the risk of long-term post-COVID symptoms. Diabetes 70 (12), 2917–2921 (2021)

Mady, A.F., Abdelfattah, R.A., Kamel, F.M., Abdel Naiem, A.S.M., AbdelGhany, W.M., Abdelaziz, A.O.: Predictors of long covid 19 syndrome. Egyptian J. Hospital Med. 85 (2), 3604–3608 (2021)

Osikomaiya, B., et al.: Long COVID’: persistent COVID-19 symptoms in survivors managed in Lagos State, Nigeria. BMC Infect. Diseases 21(1), 1–7 (2021)

Peghin, M., et al.: Post-COVID-19 symptoms 6 months after acute infection among hospitalized and non-hospitalized patients. Clinical Microbiol. Infect. 27 (10), 1507–1513 (2021)

Sudre, C.H., et al.: Attributes and predictors of long COVID. Nat. Med. 27 (4), 626–631 (2021)

Vimercati, L., et al.: Association between long COVID and overweight/obesity. J.  Clin. Med. 10 (18), 4143 (2021)

Arjun, M.C., et al.: Characteristics and predictors of Long COVID among diagnosed cases of COVID-19. Plos one 17 (12), e0278825 (2022)

Banić, M.,et al.: Risk factors and severity of functional impairment in long COVID: a single-center experience in Croatia. Croatian Med. J. 63 (1), 27–35 (2022)

Baris, S.A., et alk.: The predictors of long–COVID in the cohort of Turkish Thoracic Society–TURCOVID multicenter registry: one year follow–up results. Asian Pacific J. Tropical Med. 15 (9), 400–409 (2022)

Baruch, J., Zahra, C., Cardona, T., Melillo, T.: National long COVID impact and risk factors. Public Health 213 , 177–180 (2022)

Buonsenso, D., et al.: The prevalence, characteristics and risk factors of persistent symptoms in non-hospitalized and hospitalized children with sars-cov-2 infection followed-up for up to 12 months: a prospective, cohort study in Rome, Italy. J. Clin. Med. 11 (22), 6772 (2022)

Chudzik, M., Babicki, M., Kapusta, J., Kałuzinska-Kołat, Z., Kołat, D., Jankowski, P.: Long-COVID Clinical Features and Risk Factors: a Retrospective Analysis of Patients from the STOP-COVID Registry of the PoLoCOV Study. Viruses 14 (8), 1755 (2022)

Chudzik, M., Lewek, J., Kapusta, J., Banach, M., Jankowski, P., Bielecka-Dabrowa, A.: Predictors of long COVID in patients without comorbidities: data from the polish long-covid cardiovascular (polocov-cvd) study. J. Clin. Med. 11 (17), 4980 (2022)

Cristillo, V., Pilotto, A., Cotti Piccinelli, S., Bonzi, G., et al.: Premorbid vulnerability and disease severity impact on Long-COVID cognitive impairment. Aging Clin. Experim. Res. 34 , 257–260 (2022)

Daitch, V., Yet al.: Characteristics of long-COVID among older adults: A cross-sectional study. Inter. J. Infect. Diseases 125 , 287–293 (2022)

El Otmani, H., Nabili, S., Berrada, M., Bellakhdar, S., El Moutawakil, B., Abdoh Rafai, M.: Prevalence, characteristics and risk factors in a Moroccan cohort of Long-Covid-19. Neurol. Sci. 43 (9), 5175–5180

Fernández-de-Las-Peñas, C., et al.: Female sex is a risk factor associated with long-term post-COVID related-symptoms but not with COVID-19 symptoms: the LONG-COVID-EXP-CM multicenter study. J. Clin. Med. 11 (2), 413 (2022)

Fernández-de-Las-Peñas, C., et al.: Symptoms experienced at the acute phase of SARS-CoV-2 infection as risk factor of long-term post-COVID symptoms: the LONG-COVID-EXP-CM multicenter study. Int. J. Infect. Dis. 116 , 241–244 (2022)

Frontera, J.A., et al.: Life stressors significantly impact long-term outcomes and post-acute symptoms 12-months after COVID-19 hospitalization. J. Neurol. Sci. 443 , 120487 (2022)

Heubner, L., et al.: Extreme obesity is a strong predictor for in-hospital mortality and the prevalence of long-COVID in severe COVID-19 patients with acute respiratory distress syndrome. Sci. Rep. 12 (1), 18418

Hultström, M., et al.: Dehydration is associated with production of organic osmolytes and predicts physical long-term symptoms after COVID-19: a multicenter cohort study. Critical Care 26 (1), 1–9 (2022)

Izquierdo-Condoy, J.S., et al.: Long COVID at different altitudes: a countrywide epidemiological analysis. Inter. J. Environ. Res. Public Health 19 (22), 14673 (2022)

Knight, D.R., Munipalli, B., Logvinov, I.I., Halkar, M.G., Mitri, G., Hines, S.L.: Perception, prevalence, and prediction of severe infection and post-acute sequelae of COVID-19. Am. J. Med. Sci. 363 (4), 295–304 (2022)

Ko, A.C.S.,et al.: Number of initial symptoms is more related to long COVID-19 than acute severity of infection: A prospective cohort of hospitalized patients. Inter. J. Infect. Diseases 118 , 220–223 (2022)

Liao, X., et al.: Long-term sequelae of different COVID-19 variants: the original strain versus the Omicron variant. Global Health Med. 4 (6), 322–326 (2022)

Margalit, I., Yelin, D., Sagi, M., Rahat, M. M., Sheena, L., Mizrahi, N., ... Yahav, D.: Risk factors and multidimensional assessment of long coronavirus disease fatigue: a nested case-control study. Clinical Infectious Diseases, 75(10), 1688–1697 (2022)

Mazza, M.G., Palladini, M., Villa, G., De Lorenzo, R., Querini, P.R., Benedetti, F.: Prevalence, trajectory over time, and risk factor of post-COVID-19 fatigue. J. Psychiatr. Res.Psychiatr. Res. 155 , 112–119 (2022)

Osmanov, I.M., et al.: Risk factors for post-COVID-19 condition in previously hospitalised children using the ISARIC Global follow-up protocol: a prospective cohort study. European Resp. J. 59 (2) (2022)

Pazukhina, E., et al.: Prevalence and risk factors of post-COVID-19 condition in adults and children at 6 and 12 months after hospital discharge: a prospective, cohort study in Moscow (StopCOVID). BMC Med. 20 (1), 244 (2022)

Perlis, R.H., et al.: Prevalence and correlates of long COVID symptoms among US adults. JAMA Netw. Open 5 (10), e2238804-e2238804 (2022)

Pływaczewska-Jakubowska, M., Chudzik, M., Babicki, M., Kapusta, J., Jankowski, P.: Lifestyle, course of COVID-19, and risk of Long-COVID in non-hospitalized patients. Front. Med. 9 , 1036556 (2022)

Righi, E., et al.: Determinants of persistence of symptoms and impact on physical and mental wellbeing in Long COVID: a prospective cohort study. J. Infect. 84 (4), 566–572 (2022)

Rinaldi, R., Basile, M., Salzillo, C., Grieco, D. L., Caffè, A., Masciocchi, C.: Gemelli against COVID Group.: Myocardial injury portends a higher risk of mortality and long-term cardiovascular sequelae after hospital discharge in COVID-19 survivors. J. Clin. Med. 11 (19), 5964 (2022)

Sansone, D., et al.: Persistence of symptoms 15 months since COVID-19 diagnosis: prevalence, risk factors and residual work ability. Life 13 (1), 97 (2022)

Spinicci, M., et al.: Infection with SARS-CoV-2 variants is associated with different long COVID phenotypes. Viruses 14 (11), 2367 (2022)

Su, Y., et al.: Multiple early factors anticipate post-acute COVID-19 sequelae. Cell 185 (5), 881–895 (2022)

Subramanian, A., et al.: Symptoms and risk factors for long COVID in non-hospitalized adults. Nat. Med. 28 (8), 1706–1714 (2022)

Thompson, E.J., Williams, D.M., Walker, A.J., Mitchell, R.E., Niedzwiedz, C.L., Yang, T.C.: Long COVID burden and risk factors in 10 UK longitudinal studies and electronic health records. Nat. Commun. 13 (1), 3528 (2022)

Wang, S., et al.: Associations of depression, anxiety, worry, perceived stress, and loneliness prior to infection with risk of post–COVID-19 conditions. JAMA Psych. 79 (11), 1081–1091 (2022)

Adler, L., et al.: Long COVID symptoms in Israeli children with and without a history of SARS-CoV-2 infection: a cross-sectional study. BMJ Open 13 (2), e064155 (2023)

Bovil, T., Wester, C.T., Scheel-Hincke, L.L., Andersen-Ranberg, K.: Risk factors of post-COVID-19 conditions attributed to COVID-19 disease in people aged≥ 50 years in Europe and Israel. Public Health 214 , 69–72 (2023)

Daines, L., et al.: Characteristics and risk factors for post-COVID-19 breathlessness after hospitalisation for COVID-19. ERJ Open Res. 9 (1) (2023)

Lapa, J., Rosa, D., Mendes, J.P.L., Deusdará, R., Romero, G.A.S.: Prevalence and associated factors of post-COVID-19 syndrome in a Brazilian cohort after 3 and 6 months of hospital discharge. Int. J. Environ. Res. Public Health 20 (1), 848 (2023)

Nørgård, B.M., Zegers, F.D., Juhl, C.B., Kjeldsen, J., Nielsen, J.: Diabetes mellitus and the risk of post-acute COVID-19 hospitalizations—a nationwide cohort study. Diabet. Med.  40 (2), e14986 (2023)

Tene, L., Bergroth, T., Eisenberg, A., David, S.S.B., Chodick, G.: Risk factors, health outcomes, healthcare services utilization, and direct medical costs of patients with long COVID. Int. J. Infect. Dis. 128 , 3–10 (2023)

Zhang, Y., et al.: Identifying environmental risk factors for post-acute sequelae of SARS-CoV-2 infection: An EHR-based cohort study from the recover program. Environ. Adv. 11 , 100352 (2023)

Download references

Acknowledgments

This study was carried out within the scope of the course unit Clinical Information Management of the Master’s in Clinical Bioinformatics at the University of Aveiro.

Author information

Authors and affiliations.

Department of Medical Sciences, University of Aveiro, Aveiro, Portugal

Ema Santos, Afonso Fernandes & Manuel Graça

IEETA, Department of Medical Sciences, University of Aveiro, Aveiro, Portugal

Nelson Pacheco Rocha

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Nelson Pacheco Rocha .

Editor information

Editors and affiliations.

ISEG, Universidade de Lisboa, Lisbon, Portugal

Álvaro Rocha

College of Engineering, The Ohio State University, Columbus, OH, USA

Hojjat Adeli

Institute of Data Science and Digital Technologies, Vilnius University, Vilnius, Lithuania

Gintautas Dzemyda

DCT, Universidade Portucalense, Porto, Portugal

Fernando Moreira

Institute of Information Technology, Lodz University of Technology, Łódz, Poland

Aneta Poniszewska-Marańda

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Santos, E., Fernandes, A., Graça, M., Rocha, N.P. (2024). The Role of Electronic Health Records to Identify Risk Factors for Developing Long COVID: A Scoping Review. In: Rocha, Á., Adeli, H., Dzemyda, G., Moreira, F., Poniszewska-Marańda, A. (eds) Good Practices and New Perspectives in Information Systems and Technologies. WorldCIST 2024. Lecture Notes in Networks and Systems, vol 986. Springer, Cham. https://doi.org/10.1007/978-3-031-60218-4_12

Download citation

DOI : https://doi.org/10.1007/978-3-031-60218-4_12

Published : 13 May 2024

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-60217-7

Online ISBN : 978-3-031-60218-4

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Advancing Health and Wellness with Technology: The Role of Electronic Health Record (EHR) Systems

T he Healthcare industry is undergoing a digital transformation. At the forefront of this transformation are Electronic Health Record (EHR) systems. They are modernizing how patient information is collected, stored, and used.

This technology is significantly improving healthcare. It is improving care and outcomes, streamlining operations, and empowering patients.

EHRs Enhance Patient-Focused Care.

Electronic Health Record systems provide the full picture of a patient’s health profile in real-time. They bring all their care data together into one place. This combined information provides valuable insights that enhance clinical decision-making and care coordination.

Providers can make well-informed care choices quickly with instant access to a patient’s history, medications, diagnoses, allergies, immunizations, and more. EHRs eliminate guesswork. They allow care based on evidence and best practices.

Also, EHRs make care coordination and management easier. With integrated records across providers, care teams can optimize treatment plans, reduce duplicate services, and ensure smooth care transitions. This continuity of care enables patient-focused experiences.

EHRs: A Driver of Better Patient Outcomes

Having instant access to a patient’s complete data is more than just convenient. It transforms care. It lets EHRs dramatically improve outcomes by enabling quick, well-informed decisions aligned with top care standards.

EHRs result in more comprehensive patient data. This allows accurate diagnoses and targeted treatment plans. Better insights from combined records reduce misdiagnoses and potential safety issues.

EHRs also enable real-time clinical decision support tools. These guide providers based on medical evidence. This promotes following clinical guidelines, lowering medical error risk.

Additionally, EHRs enable population health management through advanced analytics. Predictive models identify at-risk patients for conditions. Preventive interventions can then be targeted to needs, improving outcomes.

Streamlining Healthcare Delivery with EHR Technology   

The EHR benefits go beyond clinical settings. By automating workflows, EHRs streamline healthcare delivery. This introduces efficiencies and cost savings so providers can focus on patient care.

With intuitive interfaces and templates, EHRs simplify documentation, order entry, prescriptions, and administrative tasks. Providers spend less time on paperwork and more time with patients.

EHRs eliminate duplicate testing by making results accessible across the entire care spectrum. Orders can be based on full history rather than repeats.

Billing and coding are also automated using EHRs. Claims lag times and denials decrease. Practices can even be reimbursed for meaningful EHR use under value-based payment models.

Empowering Patients in Their Health Journey

As EHRs refine healthcare operations, they also empower patients to actively manage their health. Access to their information promotes proactive wellness.

Patient portals integrated with EHRs provide 24/7 access to test results, care plans, education, and more. This transparency keeps patients informed, engaged, and invested in their outcomes.

EHRs also enable patient-provider communication between visits through secure messaging. Patients can conveniently connect with their care team.

Access to their records lets patients be more proactive in managing chronic conditions, tracking health metrics, and following care plans for better well-being.

EHRs and the Digital Transformation of Healthcare

EHR implementation is key to the digital transformation happening in healthcare. As a foundational part of health IT systems, EHRs provide endless ways to optimize care across settings through connected systems, advanced analytics, telemedicine in healthcare , virtual care, and more.

Already, EHRs are integrating with tech-like wearables to collect patient data for remote monitoring. Smooth data flows between patients and providers will reshape healthcare.

Interoperability between health systems enables connectivity. Data liquidity will empower next-gen analytics for population health insights and personalized guidance.

EHRs bring an era of boundless digital innovation where quality, outcomes, and experiences are dramatically improved. Still, the human aspects of medicine will always remain integral—and be enhanced by technology.

Challenges Faced With EHR Tech

Setting up an EHR system takes serious time and effort. Moving data, setting up software, training staff, and starting new workflows is complex. 

Learning Curve for Users

An EHR brings major changes to providers’ daily routines. Learning to use new interfaces, document electronically, and use the tech in care takes time. At first, EHR use may feel tricky and slow providers down until they get more experience. Staff productivity can dip temporarily.

Information Overload

EHRs make huge amounts of patient data available. However, too much information can occur. Sorting through lots of records to find key details takes time. Customizing systems with templates helps focus data entry and retrieval. Proper EHR training is vital.

Data Security Concerns

EHRs store very sensitive patient information. Data breaches or unauthorized access put patients’ privacy at serious risk. Providers must use strong system security with authentication, encryption, backups, and access controls. Ongoing security updates and audits are essential too.

Interoperability Issues

Ideally, EHRs should integrate seamlessly to share data between different healthcare systems. Limited interoperability remains a barrier. Varying vendor systems using proprietary formats makes transferring records difficult. However, new frameworks like FHIR (Fast Healthcare Interoperability Resources) aim to fix this over time.

Time-Consuming Implementation

Setting up an EHR system takes serious time and effort. Moving data, configuring software, training staff, and starting new workflows is complex. Setup can take several months or years. This can strain resources, reduce productivity, and disrupt operations. Providers also need to ensure no patient care gaps during the transition.

Frequently Asked Questions

How do EHRs improve the quality of patient care and wellness?

EHRs promote quality care by providing instant access to full health data. This eliminates guesswork and enables evidence-based decisions aligned with best practices. It also improves coordination across providers.

What are the real EHR benefits in terms of cost and efficiency?

EHRs provide cost savings and efficiency by streamlining workflows, automating tasks, reducing paperwork, eliminating duplicate testing, improving billing and coding, and enabling value-based reimbursement.

How do EHRs empower patients to manage their health?

EHRs empower patients by enabling 24/7 record access through portals, communication with providers, and supplying data needed for patients to proactively manage conditions, track metrics, and follow care plans.

Bottom Line

In summary, EHR systems are key to healthcare’s digital change. By improving care, outcomes, delivery, and empowering patients, EHRs are critical for advancing health now and in the future.

EHRs have the potential to reshape healthcare through endless innovation while maintaining the human touch we all need.

The post Advancing Health and Wellness with Technology: The Role of Electronic Health Record (EHR) Systems appeared first on Mom and More .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Indian J Ophthalmol
• v.68(3); 2020 Mar

Electronic medical records – The good, the bad and the ugly

Santosh g honavar.

Editor, Indian Journal of Ophthalmology, Centre for Sight, Road No 2, Banjara Hills, Hyderabad, Telangana, India. E-mail: gro.soia@lanruojrotide

“Technology is wonderful and seductive, but when seen as more real than the person to whom it is applied, it may also suppress curiosity; and such curiosity is essential to active thinking and quality care.” – Dr. Faith Fitzgerald

Henry David Thoreau's prophetic statement in Walden (1854) - “ Men have become the tools of their tools ”, has come to be completely realized in the 21 st century, specifically concerning human interface with information technology. The interaction of physicians with electronic medical records (EMR) is the most relevant example of how our inventions have enslaved us. The focus is often on creating a perfect record on EMR, while patient interaction is relegated to the hazy periphery.

Evolution of Medical Records

Medical records have a history of 4000 years in evolution and, in some form, have existed since the beginning of the practice of medicine. Some of the first medical records date back to Hippocrates in the 5 th century BC and medieval physicians.[ 1 , 2 ] Formal medical records appeared in the nineteenth century in Europe in major teaching hospitals and were quickly adopted across the world. The modern medical record was developed in the 20 th century – data about each patient, including clinical data, was recorded, organized in a standardized format and stored.[ 2 ] Major problems with traditional paper medical records include lack of standardization across physicians and healthcare facilities, poor searchability and loss of information.

EMR has been in evolution for several decades now but continues to grossly miss the intended mark of efficient and personalized patient care. The first EMR was developed in 1972 by the Regenstreif Institute in the United States and was then welcomed as a major advancement in medical practice.[ 3 ] The uptake, however, was low, the cost being a major constraint. The vital push came through the American Recovery and Reinvestment Act 2009, spearheaded by Barack Obama, which envisaged incentives to EMR users.[ 3 ] Several EMR packages have since been developed and have become widely available across the world.

EMR – The Good

EMR is considered potentially one of the drivers for the transformation of healthcare. From a patient care perspective, EMR is expected to improve the accuracy of the information, support clinical decision-making and improve the accessibility of information for continuity of care.[ 4 ] From an operational perspective, EMR should generate essential health care statistics crucial to the planning and management of health care services.[ 4 ] User expectations from a good EMR are several – meticulous patient documentation, common templates and order sets, disease coding and billing, regulatory compliance, prevention of medication errors, clinical pathway utilization, optimized workflow, medico-legal defensibility, adaptive learning capability, simplicity, multiple input interfaces (notes, voice transcription, drawings, etc), incorporation of clinical images, seamless connectivity with clinical investigation platforms, input speed at the point of entry, and most importantly, data compilation for analysis and research, all with time-efficiency, and a user- and patient-friendly interface.[ 4 , 5 ] Ideally, EMR should be on a single platform nationwide to enable interoperability and portability horizontally and vertically across the referral chain.

Are computers and clinicians uneasy bedfellows? Probably not. Every sphere of life, including the practice of medicine, has seen extensive computerization and the present generation of doctors are extremely comfortable with digital technology. The uptake of EMR is on the rise and it is here to stay.[ 6 , 7 ] In the United States, ophthalmologists have almost quadrupled their EMR use, from 19% in 2008 to 72% in 2016.[ 7 ] The use of EMR is still in its infancy in India.[ 8 ] The Government of India intends to introduce a uniform system of EMR. An expert committee set up by the government has developed “Electronic Health Record Standards for India”.[ 8 ] With this as the background, there is an immense nascent potential for EMR in India. With major Indian ophthalmic institutes having developed their EMRs and using them in their routine daily practice, and their residents and fellows having been “trained on EMRs”, its use is only likely to increase.

EMR – The Bad and the Ugly

The chief complaint against EMR is that it has undermined personalized face-to-face patient care and the vital doctor-patient interaction - the very soul of medicine - into a new check box-based doctor-computer-patient interaction. Abraham Verghese calls this an “iPatient” phenomenon.[ 9 ] EMR was never designed to facilitate a personalized human narrative, logical thinking, and experience-based clinical analysis. Clinical reasoning being the backbone of a traditional doctor-patient interaction, “a medical record—whether paper or digital—must preserve the information that the physician carefully and thoughtfully elicits from the patient in a form that, above all, facilitates clinical reasoning.”[ 1 ] Current EMRs do not.[ 1 ]

A new report from the National Academy of Medicine is revealing – on an average, nurses and doctors spend 50 percent of their workday treating the screen, not the patient, and the increased work burden associated with EMRs is one of the factors for physician burnout.[ 10 ] A study of emergency room doctors revealed that putting information into the computer consumed more of their time than any other activity. Using a “click” of the computer mouse as the standard of measure, a doctor needed to make 6 clicks of the mouse to order an aspirin, 8 clicks to get a chest x-ray, 15 clicks to provide a prescription, etc., Over 40% of a typical 10-hour emergency room shift was devoted to data entry and 4,000 clicks of the computer mouse.[ 11 ] Immense information on EMR results in high (data) noise to (clinical) signals ratio. Arnold Relman, former editor-in-chief of the New England Journal of Medicine and a physician with 6 decades of experience found EMR “lacking in coherent descriptions of his medical progress, or his complaints and state of mind” when he was a patient himself.[ 5 ]

EMRs seem to have adversely affected the clinical training as Ober and Applegate state, “Our residents often resemble air traffic controllers, focusing more on the logistics of arrivals and departures than on understanding the patient's journey”.[ 5 ] They go on to quote a resident, “Education, rapport, compassion, bedside clinical reasoning, the physical exam, all seem to take a back seat in the current system”.[ 5 ] EMRs seem to be badly designed to the do the job they are meant to do and seem to have failed to make patient care better, more efficient, or more satisfying for the patient or the doctor.

Will We Ever Find the Gold? - Can there be a Perfect EMR?

As there can never be a perfect spouse, there can never be a perfect EMR. EMRs must evolve and the potential users synchronously need to retrain themselves and change their mindset until a sweet spot is reached. “To develop an EMR that meets the needs of the physicians who will use it, we need to better understand how the physicians work, and develop the software with an eye toward solving real problems in practices rather than developing a solution looking for a problem.”[ 12 ] Fortunately, India seems to be leading in the development of stand-alone ophthalmology EMRs, and that too with significant contributions from the users' right at the stage of EMR development. Sankara Nethralaya and Tata Consultancy Services (TCS) have together developed a comprehensive EMR system from scratch. It is natural for people to forget, but Anthony Vipin Das must remember that it took us a lot of effort to initiate and carry forward an in-house coding and development of EMR at the LV Prasad Eye Institute (LVPEI) about 10 years ago. It was meant to be a smart EMR, developed by the ophthalmologists and for the ophthalmologists, appropriately called eyeSmart. I feel redeemed that the seed that I had a small part is sowing and initially nurturing has now grown to be a fruit-bearing tree and is seamlessly used across the LVPEI network for patient care, administration and research. The current issue of the Indian Journal of Ophthalmology carries an article from the LVPEI group reporting their 8-year experience with eyeSmart and the accompanying commentary puts things in perspective.[ 13 , 14 ]

Robert Wachter states in his book The Digital Doctor – ”One of the great challenges in healthcare technology is that medicine is at once an enormous business and an exquisitely human endeavor; it requires the ruthless efficiency of the modern manufacturing plant and the gentle hand-holding of the parish priest; it is about science, but also about art; it is eminently quantifiable and yet stubbornly not.” An ideal EMR should harmoniously bring together the soul of medicine and cutting-edge informatics.

This paper is in the following e-collection/theme issue:

Published on 14.5.2024 in Vol 11 (2024)

Coding of Childhood Psychiatric and Neurodevelopmental Disorders in Electronic Health Records of a Large Integrated Health Care System: Validation Study

Authors of this article:

• Jiaxiao M Shi 1 , PhD ; 
• Vicki Y Chiu 1 , MS ; 
• Chantal C Avila 1 , MA ; 
• Sierra Lewis 1 , MPH ; 
• Daniella Park 1 , MPH ; 
• Morgan R Peltier 2 , PhD ; 
• Darios Getahun 1 , MD, MPH, PhD

1 Department of Research and Evaluation, Kaiser Permanente Southern California, , Pasadena, CA, , United States

2 Department of Psychiatry, Jersey Shore University Medical Center, , Neptune, NJ, , United States

Corresponding Author:

Jiaxiao M Shi, PhD

Background: Mental, emotional, and behavioral disorders are chronic pediatric conditions, and their prevalence has been on the rise over recent decades. Affected children have long-term health sequelae and a decline in health-related quality of life. Due to the lack of a validated database for pharmacoepidemiological research on selected mental, emotional, and behavioral disorders, there is uncertainty in their reported prevalence in the literature.

Objectives: We aimed to evaluate the accuracy of coding related to pediatric mental, emotional, and behavioral disorders in a large integrated health care system’s electronic health records (EHRs) and compare the coding quality before and after the implementation of the International Classification of Diseases, Tenth Revision, Clinical Modification ( ICD-10-CM ) coding as well as before and after the COVID-19 pandemic.

Methods: Medical records of 1200 member children aged 2-17 years with at least 1 clinical visit before the COVID-19 pandemic (January 1, 2012, to December 31, 2014, the ICD-9-CM coding period; and January 1, 2017, to December 31, 2019, the ICD-10-CM coding period) and after the COVID-19 pandemic (January 1, 2021, to December 31, 2022) were selected with stratified random sampling from EHRs for chart review. Two trained research associates reviewed the EHRs for all potential cases of autism spectrum disorder (ASD), attention-deficit hyperactivity disorder (ADHD), major depression disorder (MDD), anxiety disorder (AD), and disruptive behavior disorders (DBD) in children during the study period. Children were considered cases only if there was a mention of any one of the conditions (yes for diagnosis) in the electronic chart during the corresponding time period. The validity of diagnosis codes was evaluated by directly comparing them with the gold standard of chart abstraction using sensitivity, specificity, positive predictive value, negative predictive value, the summary statistics of the F -score, and Youden J statistic. κ statistic for interrater reliability among the 2 abstractors was calculated.

Results: The overall agreement between the identification of mental, behavioral, and emotional conditions using diagnosis codes compared to medical record abstraction was strong and similar across the ICD-9-CM and ICD-10-CM coding periods as well as during the prepandemic and pandemic time periods. The performance of AD coding, while strong, was relatively lower compared to the other conditions. The weighted sensitivity, specificity, positive predictive value, and negative predictive value for each of the 5 conditions were as follows: 100%, 100%, 99.2%, and 100%, respectively, for ASD; 100%, 99.9%, 99.2%, and 100%, respectively, for ADHD; 100%, 100%, 100%, and 100%, respectively for DBD; 87.7%, 100%, 100%, and 99.2%, respectively, for AD; and 100%, 100%, 99.2%, and 100%, respectively, for MDD. The F -score and Youden J statistic ranged between 87.7% and 100%. The overall agreement between abstractors was almost perfect (κ=95%).

Conclusions: Diagnostic codes are quite reliable for identifying selected childhood mental, behavioral, and emotional conditions. The findings remained similar during the pandemic and after the implementation of the ICD-10-CM coding in the EHR system.

Introduction

Children and adolescents are particularly vulnerable to chronic mental and behavioral conditions because their brain continues to develop. Childhood mental and behavioral disorders, including autism spectrum disorder (ASD), attention-deficit hyperactivity disorder (ADHD), disruptive behavior disorders (DBD), anxiety disorder (AD), and major depressive disorder (MDD), are common neurological disorders and are on the rise in recent decades [ 1 - 4 ]. Affected children and adolescents are subjected to long-term negative health and social consequences [ 5 , 6 ], leading to significant health care costs and public health burden [ 7 , 8 ]. Therefore, accurately estimating their incidence and prevalence is important to guide policy-making, resource allocation, and implementation of different intervention programs.

Trends in mental and behavioral disorders are difficult to examine using routinely collected data and often are difficult to compare across studies because of differences in case ascertainment methods. Therefore, reported incidence and prevalence rates vary widely across studies [ 9 ], and the accuracy of case ascertainment has been challenged by researchers [ 10 , 11 ]. This problem is further complicated by the accuracy of coding after the mandatory introduction of the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) coding systems to classify diagnoses and procedures in the United States, which occurred on October 1, 2015 [ 12 ]. The ICD-10-CM provides increased specificity and detail for many health conditions [ 12 ], including childhood psychiatric and neurodevelopmental conditions. Many health care providers adopted the electronic health record (EHR) enacted by the Health Information Technology for Economic and Clinical Health Act (HITECH Act) in 2009 [ 13 , 14 ].

The Kaiser Permanente Southern California (KPSC)–integrated EHR data provide researchers with important health information to perform pharmacoepidemiological studies of children’s behavioral, medical, and psychiatric conditions, including examining the public health impact of childhood psychiatric and neurodevelopmental conditions; however, there is uncertainty over the accuracy of the clinical diagnosis codes requiring validation before their use. Therefore, the objective of this study was to perform an EHR review on the validity of diagnosis codes during 3 time points ( ICD-9-CM and ICD-10-CM coding and before and after the COVID-19 pandemic) for ascertaining psychiatric and neurodevelopmental conditions in a large socioeconomically diverse pediatric population aged 2-17 years [ 15 ]. The validation for the pre- and post–COVID-19 pandemic is important to assess how much the diagnosis coding has been impacted by increasing the use of virtual visits (telephone and video-assisted encounters).

Study Setting

This study was conducted using data on member children extracted from the KPSC EHR. The KPSC health care system provides services to over 4.8 million members in 15 hospitals and 234 medical offices throughout southern California. Mental health services are provided to member patients by qualified providers at in- and outpatient psychiatric care facilities in KPSC settings. Although most members receive their care at KPSC hospitals and <10% use contracting hospitals, all diagnostic, procedural, and pharmacy records are captured and maintained by the KPSC EHR since its full implementation in 2008. The sociodemographic characteristics of KPSC members closely reflect the California population [ 15 ].

Ethical Considerations

This study was conducted with approval by the KPSC Institutional Review Board (IRB# 13114). Informed consent was waived, as the study was low risk and strictly involved the use of internal EHR data, accessible only to authorized personnel when needed.

Study Design and Sample Selection

Data for this validation study were obtained retrospectively from children who were members of the KPSC health care system during three distinct time periods: (1) January 1, 2012, through December 31, 2014; (2) January 1, 2017, through December 31, 2019; and (3) January 1, 2021, through December 31, 2022. We carefully selected the 3 time periods to investigate the medical coding accuracy of selected psychiatric and neurodevelopmental conditions encompassing both the ICD-9-CM and ICD-10-CM periods as well as the pre- and post–COVID-19 pandemic eras. To be included in the validation study cohort for each time period, children must have been enrolled in the KPSC health care system for at least 1 year during the corresponding time period at specific age ranges varying by condition (5-17 years of age for ADHD, 2-17 years of age for ASD, and 3-17 years of age for the other 3 conditions—AD, MDD, and DBD). Children may present with signs and symptoms of the specified conditions very early in life; however, for this study, we used reported and reliable lower age groupings for ASD and ADHD diagnoses [ 16 , 17 ] and widely published age goupings for anxiety and depressive disorders [ 18 , 19 ]. Furthermore, at least 1 clinical visit, including virtual visits, in each corresponding time period was required.

In the KPSC system, diagnosis codes for clinical visits (eg, hospitalization, outpatient office visits, and emergency department visits) from all KPSC facilities are extracted from EHRs and entered into a structured database by professional medical coders from the clinical data management team. In this validation study, coding-based outcomes of interest were ascertained using these clinical diagnosis codes within the 3 specified time periods ( Multimedia Appendix 1 presents the ICD-9-CM and ICD-10-CM codes).

For each of the investigated conditions, we randomly sampled 40 cases according to the following strata: (1) those without a diagnosis (No-Dx) and (2) those with a diagnosis (Dx). Thus, records for a total of 1200 individual children were selected. The accuracy for each of the 2 strata, with or without documented diagnostic records, was expected to be around 85%. A sample size of 120 per diagnostic condition would provide less than a 5% one-sided margin for a 90% CI of the accuracy for each stratum.

Diagnosing ASD, ADHD, DBD, and MDD

The KPSC system has an integrated framework for in- and outpatient as well as emergency department encounter services. During the child’s visit to any of these facilities, the practitioner has access to the child’s diagnoses, but often the diagnosis of ASD, ADHD, and DBD is made in an outpatient setting. The following criteria were used to diagnose and code ASD, ADHD, and DBD within the KPSC setting: (1) a Child Behavior Checklist must be filled out by parents and teachers to describe the child’s behavioral and emotional problems and (2) a clinical interview must be performed by a qualified mental health professional. In a preliminary study conducted for this project, 96% of children with ASD, ADHD, and DBD were found to have had their conditions diagnosed by KPSC child and adolescent psychiatrists, developmental and behavioral pediatricians, child psychologists, and neurologists consistent with the diagnostic criteria from the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition ( DSM-V ). Diagnosis of the remaining 4% was confirmed upon membership, as these cases had been previously diagnosed outside the KPSC system [ 20 , 21 ].

The diagnosis of depression disorder, in the KPSC system, was based on the US Preventive Services Task Force on screening for depression in children and adolescents recommendation [ 22 ]. The KPSC system uses the Patient Health Questionnaire (PHQ-9) and the PHQ-9 modified for Adolescents (PHQ-A). If a patient’s score on the PHQ-9 or PHQ-A does not seem to accurately reflect observed clinical symptoms, DSM-V criteria are recommended for diagnosis.

To ascertain the neurodevelopmental conditions investigated in this study, we relied on systemwide clinical diagnoses made by experts in the field as mentioned above.

Chart Abstraction Process

Trained research associates (abstractors) reviewed EHRs for documentation of a diagnosis (yes/no) of each condition under investigation for the selected sample of children during the study period. Children were considered to have the disorder in the presence of documented evidence of that condition noted in the chart during the corresponding time period of investigation. To ensure data quality and consistency of chart reviews between the 2 abstractors, a total of 180 cases, stratified by the 2 strata (Dx and No-Dx), were randomly selected for reabstraction (90 per abstractor and 36 per condition). There was a total of 4 possible responses for each chart reviewed case, as follows: diagnosis “Yes,” diagnosis “No,” “Unable to ascertain due to blocked notes,” and “Unable to ascertain due to insufficient notes.” The abstractors based their responses on the information they were able to ascertain from the clinical notes. For example, due to the sensitive nature of psychiatric diagnoses, some progress notes may have been blocked; therefore, if the abstractors could not ascertain the diagnosis due to blocked notes, it was coded as such. Similarly, if the abstractors could not ascertain a diagnosis due to a lack of documentation or notes in the KPSC system with underused care (eg, outside claims data), the record was coded as “Unable to ascertain due to insufficient notes.”

The results of this assessment informed our full chart review process of the 1200 medical records as mentioned above. In other words, we excluded those claims data records from the random selection of the validation sample. Furthermore, during chart review, records in which the abstractors encountered blocked notes and were therefore unable to make a determination were flagged. These flagged records (n=38, 3%) were replaced with another randomly selected record for chart review from the same strata. Potential cases that were still unclear in the clinical use records were adjudicated by the study investigator with expertise in the field (DG). The child psychiatric and neurodevelopmental condition cases abstracted through this process served as the gold standard.

Child Characteristics

Characteristics for KPSC member children included age (2-5, 6-11, and 12-17 years), sex (male and female), race/ethnicity (categorized as non-Hispanic White, non-Hispanic Black, Hispanic, Asian/Pacific Islander, other/multiple, and unknown), median household income in US dollars (<30,000; 30,000-49,999; 50,000-69,999; 70,000-89,999; and ≥90,000), and insurance type (Medicaid, commercial through employment, private/individual, and other).

We obtained the characteristics of all children of the state of California residents during the same time periods using publicly available data posted on the Centers for Disease Control and Prevention Wonder website [ 3 ]. Both the KPSC EHR and the Centers for Disease Control and Prevention (Wonder) sources provided information on child characteristics, including age and race/ethnicity. Data on median household income were estimated based on census tracts for KPSC patients.

Statistical Analysis

We described the characteristics of our study population between those with and without the conditions of interest. Furthermore, we investigated how representative the KPSC pediatric population is compared to the state of California pediatric population during the entire study period using frequency distributions. For the purpose of comparison with the California children population, the age of the children for the KPSC population was evaluated based on the date of randomly selected clinical visits for each child during the entire study period.

The agreement between the 2 abstractors was evaluated with the interrater reliability assessment (κ statistic) by using the initial 180 chart reviews. We compared findings from the manual chart review of the 1200 cases, set as the gold standard, with corresponding diagnosis records for he ICD-9-CM and ICD-10-CM coding periods as well as before and after the COVID-19 pandemic through sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). These performance measurements were reported as weighted percentages with corresponding 95% CIs using normalized sampling weights ( W ), as defined below:

W i j = #   i n   s t r a t u m   j 120 × ( # t o t a l   s t u d y   p o p u l a t i o n   i ) ,   i = 1   t o   5 , j = 1 , 2

where i is for each of the 5 conditions and j is for the corresponding two strata (Dx and No-Dx) within each condition. To evaluate the overall performance, we also reported the summary statistics of F -score and Youden J statistic, which are composite measurements of sensitivity and PPV ( F -score) or sensitivity and specificity ( J statistic).

All analyses were conducted using SAS statistical software (version 9.4; SAS Institute, Inc).

An overview of the patient characteristics for our sample, study population, and the state of California pediatric population are shown in Table 1 . Overall, 2,283,931 children from all KPSC hospitals and medical offices were obtained during the 3 time periods. Compared with the state of California pediatric population, KPSC had similar distributions on the children’s sex but had slightly higher percentages of children for ages 2-5 years (25.6% for KPSC vs 24.7% for the state of California) and 6-11 (37.8% for KPSC vs 37.4% for the state of California). In addition, KPSC had slightly lower percentages of children identified as non-Hispanic White (24.5% vs 28.5%) or Asian/Pacific Islander (10.0% vs 13.1%). However, this could be partially attributed to the larger percentage of unknown, missing, or multiple race/ethnicity in the KPSC group (6.0% vs 0.5%, respectively). Overall, the KPSC pediatric population in this study was a good representation of the entire state of California’s pediatric population. Our sample of 1200 children across the 5 conditions was broadly representative of the KPSC study population with respect to age, sex, and race/ethnicity. However, children in the older age groups were slightly oversampled. The overall κ between our 2 abstractors was 95%.

a KPSC: Kaiser Permanente Southern California; the sample is based on the KPSC data from the electronic health records (2012-2014, 2017-2019, and 202-2022).

b Data are from the natality information of Centers for Disease Control and Prevention website [ 3 ].

c Starting age is different for each condition.

d Median household income and insurance type information are not available for the California state data.

Table 2 shows the distribution and their sample sizes for the two strata (Dx and No-Dx) among our entire study population and by the 3 time periods. Based on these comparisons against the chart review results, the performance measurements of our EHRs are shown in Table 3 . The weighted sensitivity, specificity, PPV, and NPV for each of the 5 conditions were as follows:

• ADHD: sensitivity 100%, specificity 99.9%, PPV 99.2%, and NPV 100%
• ASD: sensitivity 100%, specificity 100%, PPV 99.2%, and NPV 100%
• MDD: sensitivity 100%, specificity 100%, PPV 99.2%, and NPV 100%
• AD: sensitivity 87.7%, specificity 100%, PPV 100%, and NPV 99.2%
• DBD: sensitivity 100%, specificity 100%, PPV 100%, and NPV 100%

The corresponding F -score and Youden J statistic were 99.6% and 99.9% for ADHD, 99.6% and 100% for ASD, 99.6% and 100% for MMD, 93.4% and 87.7% for AD, and 100% and 100% for DBD, respectively. Results were similar across the ICD-9-CM and ICD-10-CM coding time periods as well as the before and after the COVID-19 pandemic.

a DBD: disruptive behavior disorders.

b Dx: with confirmed diagnosis codes in electronic data.

c No-Dx: without confirmed diagnosis codes in electronic data.

d ASD: autism spectrum disorder.

e MDD: major depressive disorder.

f ADHD: attention deficit hyperactivity disorder.

a PPV: positive predictive value.

b NPV: negative predictive value.

c DBD: disruptive behavior disorders.

d MDD: major depressive disorder.

e ASD: autism spectrum disorder.

Principal Findings

This validation study was performed to determine the accuracy of clinical diagnosis codes in ascertaining childhood psychiatric and neurodevelopmental cases using data abstracted from the EHR of a large integrated health care system serving a demographically diverse patient population. To our knowledge, the accuracy of data on studied conditions using clinical diagnostic codes has not been validated using EHR data. Furthermore, the extent to which the transition of ICD-9-CM to the ICD-10-CM coding system as well as the pre- and post–COVID-19 pandemic eras have impacted the ascertainment of the studied behavioral and developmental conditions is unclear. Our study showed that within a large integrated health system, there is strong agreement between the diagnosis codes ( ICD-9-CM or ICD-10-CM ) and the patients’ conditions, including ASD, ADHD, DBD, AD, and MDD, both before and after the COVID-19 pandemic eras.

In recent years, EHRs have become important data sources for various epidemiological study designs investigating potential associations between exposures and outcomes that have become standard among researchers and health care providers as part of the American Recovery and Reinvestment Act of 2009 (specifically the HITECH Act) [ 14 ]. Although the EHR has become an important information management and care delivery system tool ensuring quality of care by providing access to comprehensive treatment-related data, its validity and completeness for conducting pharmacoepidemiological studies have been challenged by many due to how information is coded [ 23 ]. In the KPSC health care system, the process of using data coding and coding rules of the medical diagnoses and procedures recorded in patients’ health records is performed by trained medical coders from the clinical data management team. Individual diagnostic conditions in the EHR need to be evaluated critically for accuracy and consistency. Furthermore, whether the accuracy of case ascertainment has been impacted by the introduction of the ICD-10-CM coding system as well as the effect of the COVID-19 pandemic on the quality of data capture needs to be investigated. Therefore, we performed this validation study to evaluate (1) the accuracy of pediatric mental, emotional, and behavioral disorders identified in EHRs and (2) the coding quality before and after the implementation of the ICD-10-CM codings as well as before and after the COVID-19 pandemic.

The findings of this study suggest that the validity of EHR data for the identification of childhood mental, behavioral, and emotional conditions is quite strong and the transition from the ICD-9-CM coding system to the ICD-10-CM coding system as well as coding during the COVID-19 pandemic had minimal impact on the overall accuracy of case ascertainment.

The main strength of this study is the large chart abstraction conducted to assess the validity and reliability of childhood mental, behavioral, and emotional disorders case ascertainment using EHR data extracted from a large integrated health care system. The EHR system database provides an opportunity for neurodevelopmental outcome investigation with a high degree of validity of case ascertainment for epidemiological studies, in addition to current developments by others using natural language processing algorithms [ 24 - 27 ]. This validation study, comprised of a demographically diverse southern California population, is likely generalizable to health care settings with similar EHR database systems. Although the overall agreement between our abstractors was almost perfect (κ=95%), a potential limitation of this study was the use of medical record abstractors who were not blinded to the source of the data. However, a previous study that evaluated the agreement between masked and unmasked medical record abstraction reported no impact of bias in case ascertainment [ 28 ]. A further potential limitation is the fact that we had some records with blocked notes that needed to be replaced because of incompleteness in ascertaining the conditions of interest. However, in close examination, we found that the blocked notes were only 38 (3%) out of the 1200 cases, which we believe will have minimal impact on the overall analysis. In addition, we did not consider oversampling of the older age strata in the summarized performance metrics. Considering that the older age group could be given a more accurate diagnosis coding, the actual concordance of the electronic coding might be slightly lower than what we observed in this study.

Conclusions

Our findings suggest that childhood mental, behavioral, and emotional disorders are reliably coded in the EHRs and can be used for pharmacoepidemiological studies. Furthermore, the completeness of data remained similar during the pre- and postpandemic eras and after the implementation of the ICD-10-CM coding in the EHR system.

Acknowledgments

This research was funded by the Garfield Memorial Fund (project number RNG211534) to DG. The opinions expressed are solely the authors' responsibility and do not necessarily reflect the official views of the funding agency. The Garfield Memorial Fund study team would like to thank Kaiser Permanente members who contributed electronic health information to this study. The authors thank Ms Sole Cardoso for her technical support.

Conflicts of Interest

None declared.

International Classification of Diseases, Ninth/Tenth Revisions, Clinical Modification diagnostic codes to ascertain mental, emotional, and behavioral disorders.

• Khadka N, Peltier MR, Fassett MJ, et al. Rising trends of childhood attention-deficit/hyperactivity disorder in a large integrated healthcare delivery system in Southern California, 2010-2021. J Pediatr. Mar 1, 2024;269:113997. [ CrossRef ] [ Medline ]
• Lebrun-Harris LA, Ghandour RM, Kogan MD, Warren MD. Five-year trends in US children's health and well-being, 2016-2020. JAMA Pediatr. Jul 1, 2022;176(7):e220056. [ CrossRef ] [ Medline ]
• Natality information. Centers for Disease Control and Prevention. URL: https://wonder.cdc.gov/natality.html [Accessed 2023-12-25]
• Bitsko RH, Claussen AH, Lichstein J, et al. Mental health surveillance among children. MMWR. Feb 2022;71:1-42. [ CrossRef ]
• Wang PS, Berglund PA, Olfson M, Kessler RC. Delays in initial treatment contact after first onset of a mental disorder. Health Serv Res. Apr 2004;39(2):393-415. [ CrossRef ] [ Medline ]
• Clement S, Schauman O, Graham T, et al. What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychol Med. Jan 2015;45(1):11-27. [ CrossRef ] [ Medline ]
• Tkacz J, Brady BL. Increasing rate of diagnosed childhood mental illness in the United States: incidence, prevalence and costs. Public Health Pract (Oxf). Oct 15, 2021;2:100204. [ CrossRef ] [ Medline ]
• AAP-AACAP-CHA declaration of a national emergency in child and adolescent mental health. American Academy of Pediatrics. URL: https:/​/www.​aap.org/​en/​advocacy/​child-and-adolescent-healthy-mental-development/​aap-aacap-cha- declaration-of-a-national-emergency- in-child-and-adolescent-mental-health/​ [Accessed 2023-12-25]
• Merikangas KR, He JP, Burstein M, et al. Lifetime prevalence of mental disorders in U.S. adolescents: results from the National Comorbidity Survey Replication--Adolescent Supplement (NCS-A). J Am Acad Child Adolesc Psychiatry. Oct 2010;49(10):980-989. [ CrossRef ] [ Medline ]
• Bethell CD, Garner AS, Gombojav N, Blackwell C, Heller L, Mendelson T. Social and relational health risks and common mental health problems among US children: the mitigating role of family resilience and connection to promote positive socioemotional and school-related outcomes. Child Adolesc Psychiatr Clin N Am. Jan 2022;31(1):45-70. [ CrossRef ] [ Medline ]
• Polanczyk GV, Salum GA, Sugaya LS, Caye A, Rohde LA. Annual research review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry. Mar 2015;56(3):345-365. [ CrossRef ] [ Medline ]
• World Health Organization. ICD-10: International Statistical Classification of Diseases and Related Health Problems: Tenth Revision. 2nd ed. World Health Organization; 2004. URL: https://iris.who.int/handle/10665/42980 [Accessed 2024-05-07]
• HITECH Act. HIPAA Survival Guide. 2009. URL: https://www.hipaasurvivalguide.com/hitech-act-text.php [Accessed 2023-12-25]
• Adler-Milstein J, Jha AK. HITECH Act drove large gains in hospital electronic health record adoption. Health Aff (Millwood). Aug 1, 2017;36(8):1416-1422. [ CrossRef ] [ Medline ]
• Koebnick C, Langer-Gould AM, Gould MK, et al. Sociodemographic characteristics of members of a large, integrated health care system: comparison with US Census Bureau data. Perm J. 2012;16(3):37-41. [ CrossRef ] [ Medline ]
• Baker BL, Neece CL, Fenning RM, Crnic KA, Blacher J. Mental disorders in five-year-old children with or without developmental delay: focus on ADHD. J Clin Child Adolesc Psychol. 2010;39(4):492-505. [ CrossRef ] [ Medline ]
• Lord C, Risi S, DiLavore PS, Shulman C, Thurm A, Pickles A. Autism from 2 to 9 years of age. Arch Gen Psychiatry. Jun 2006;63(6):694-701. [ CrossRef ] [ Medline ]
• Luby JL, Belden AC, Pautsch J, Si X, Spitznagel E. The clinical significance of preschool depression: impairment in functioning and clinical markers of the disorder. J Affect Disord. Jan 2009;112(1-3):111-119. [ CrossRef ] [ Medline ]
• Ingeborgrud CB, Oerbeck B, Friis S, et al. Anxiety and depression from age 3 to 8 years in children with and without ADHD symptoms. Sci Rep. Sep 16, 2023;13(1):15376. [ CrossRef ] [ Medline ]
• Getahun D, Fassett MJ, Peltier MR, et al. Association of perinatal risk factors with autism spectrum disorder. Am J Perinatol. Feb 2017;34(3):295-304. [ CrossRef ] [ Medline ]
• Getahun D, Jacobsen SJ, Fassett MJ, Chen W, Demissie K, Rhoads GG. Recent trends in childhood attention-deficit/hyperactivity disorder. JAMA Pediatr. Mar 1, 2013;167(3):282-288. [ CrossRef ] [ Medline ]
• Siu AL, U.S. Preventive Services Task Force. Screening for depression in children and adolescents: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. Mar 1, 2016;164(5):360-366. [ CrossRef ] [ Medline ]
• Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. Jan 1, 2013;20(1):144-151. [ CrossRef ] [ Medline ]
• Lingren T, Chen P, Bochenek J, et al. Electronic health record based algorithm to identify patients with autism spectrum disorder. PLoS One. 2016;11(7):e0159621. [ CrossRef ] [ Medline ]
• Leroy G, Gu Y, Pettygrove S, Galindo MK, Arora A, Kurzius-Spencer M. Automated extraction of diagnostic criteria from electronic health records for autism spectrum disorders: development, evaluation, and application. J Med Internet Res. Nov 7, 2018;20(11):e10497. [ CrossRef ] [ Medline ]
• Washington P, Wall DP. A review of and roadmap for data science and machine learning for the neuropsychiatric phenotype of autism. Annu Rev Biomed Data Sci. Aug 10, 2023;6:211-228. [ CrossRef ] [ Medline ]
• Slaby I, Hain HS, Abrams D, et al. An electronic health record (EHR) phenotype algorithm to identify patients with attention deficit hyperactivity disorders (ADHD) and psychiatric comorbidities. J Neurodev Disord. Jun 11, 2022;14(1):37. [ CrossRef ] [ Medline ]
• Reisch LM, Fosse JS, Beverly K, et al. Training, quality assurance, and assessment of medical record abstraction in a multisite study. Am J Epidemiol. Mar 15, 2003;157(6):546-551. [ CrossRef ] [ Medline ]

Abbreviations

Edited by John Torous; submitted 31.01.24; peer-reviewed by Ahmed Hassan, Peter Washington, Temitope Adebambo; final revised version received 08.04.24; accepted 09.04.24; published 14.05.24.

© Jiaxiao M Shi, Vicki Y Chiu, Chantal C Avila, Sierra Lewis, Daniella Park, Morgan R Peltier, Darios Getahun. Originally published in JMIR Mental Health (https://mental.jmir.org), 14.5.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/ , as well as this copyright and license information must be included.