Which Of The Following Is An Example Of An Electronic Data Source In Healthcare?

The research question dictates the type of data required, and the researcher must best match the data to the question or determine whether primary data collection is warranted. This chapter discusses considerations for data source selection for comparative effectiveness inquiry (CER). Important considerations for choosing data include whether or not the central variables are available to appropriately define an analytic accomplice and place exposures, outcomes, covariates, and confounders. Data should be sufficiently granular, contain historical information to determine baseline covariates, and represent an adequate duration of followup. The widespread availability of existing data from electronic health records, personal health records, and drug surveillance programs provides an opportunity for answering CER questions without the loftier expense ofttimes associated with primary data drove. If key data elements are unobtainable in an otherwise ideal dataset, methods such every bit predicting absent-minded variables with available information or interpolating for missing time points may exist used. Alternatively, the researcher may link datasets. The process of data linking, which combines information about one individual from multiple sources, increases the richness of information available in a report. This is in contrast to data pooling and networking, which are normally used to increase the size of an observational study. Each data source has advantages and disadvantages, which should be considered thoroughly in light of the enquiry question of involvement, as the validity of the report will be dictated past the quality of the data. This chapter concludes with a checklist of key considerations for selecting a information source for a CER protocol.

Introduction

Identifying advisable data sources to respond comparative effectiveness enquiry (CER) questions is challenging. While the widespread availability of existing data provides an opportunity for answering CER questions without the high expense associated with primary data collection, the information source must exist chosen advisedly to ensure that it tin can accost the study question, that information technology has a sufficient number of observations, that central variables are available, that there is adequate confounder control, and that there is a sufficient length of followup.

This chapter describes information that may be useful for observational CER studies and the sources of these data, including data nerveless for both enquiry and nonresearch purposes. The chapter also explains how the inquiry question should dictate the type of information required and how to best match data to the issue at hand. Considerations for evaluating information quality (e.g., demonstrating data integrity) and privacy protection provisions are discussed. The affiliate concludes past describing new sources of data that may expand the options available to CER researchers to accost questions. Recommendations for "best practices" regarding data selection are included, along with a checklist that researchers may use when developing and writing a CER protocol. To start, however, it is important to consider primary data collection for observational enquiry, since the use of secondary data may be incommunicable or unwise in some situations.

Data Options

Main data are information collected expressly for enquiry. Observational studies, meaning studies with no dictated intervention, require the drove of new data if there are no adequate existing data for testing hypotheses. In contrast, secondary data refer to data that were collected for other purposes and are beingness used secondarily to answer a research question. In that location are other ways to categorize information, but this classification is useful because the types of information collected for inquiry differ markedly from the types of data collected for nonresearch purposes.

Primary Data

Main information are nerveless by the investigator direct from study participants to accost a specific question or hypothesis. Information can be collected by in-person or telephone interviews, mail surveys, or computerized questionnaires. While primary data collection has the advantage of existence able to address a specific study question, information technology is ofttimes time consuming and expensive. The observational research designs that often crave chief data collection are described here. While these designs may likewise incorporate existing data, we depict them here in the context of master data collection. The need to use these designs is determined by the research question; if the research question clearly must be answered with these designs below, primary data collection may be required. Additional detail well-nigh the pick of suitable study pattern for observational CER is presented in chapter two.

Prospective Observational Studies

Observational studies are those in which individuals are selected on the basis of specific characteristics and their progress is monitored. A key concept is that the investigator does non assign the exposure(s) of interest. There are two basic observational designs: (ane) cohort studies, in which choice is based on exposure and participants are followed for the occurrence of a particular outcome, and (2) case-control studies, where selection is based on a disease or condition and participants are contacted to determine a item exposure.

Inside this framework, in that location is a broad variety of possible designs. Participants can be individuals or groups (e.chiliad., schools or hospitals); they tin be followed into the future (prospective information collection) or asked to recall by events (retrospective data collection); and, depending on the specific study questions, elements of the ii basic designs can be combined into a single report (e.g., case-cohort or nested example-command studies). If data is as well collected on those who are either not exposed or exercise not have the consequence of interest, observational studies tin can be used for hypothesis testing.

An example of a prospective observational written report is a recent investigation comparing medication adherence and viral suppression between once-daily and more than-than-once daily pill regimens in a homeless and near-homeless HIV-positive population.¹ Adherence was measured using unscheduled pill-count visits over the six-month study period while viral suppression was determined at the stop of the report. The investigators found that both adherence and viral suppression levels were higher in the once-daily groups compared to the more than-than-once-daily groups. The results of this report are notable as they signal an effective method to treat HIV in a specially hard-to-reach population.

Registries

In the well-nigh general sense, a registry is a systematic collection of information. Registries that are used for research have clearly stated purposes and targeted data collection.

Registries use an observational study pattern that does not specify treatments or require therapies intended to alter patient outcomes. There are generally few inclusion and exclusion criteria to brand the results broadly generalizable. Patients are typically identified when they present for intendance, and the data nerveless more often than not include clinical and laboratory tests and measurements. Registries can be defined by specific diseases or weather condition (east.g., cancer, birth defects, or rheumatoid arthritis), exposures (east.g., to drug products, medical devices, environmental atmospheric condition, or radiation), time periods, or populations. Depending on their purpose and the information collected, registry data tin potentially be used for public health surveillance, to determine incidence rates, to perform risk assessment, to monitor progress, and to improve clinical practice. Registries can besides provide a unique perspective into specialized subpopulations. All the same, like any long-term study, they tin be very expensive to maintain due to the attempt required to remain in contact with the participants over extended periods of time.

Registries have been used extensively for CER. As an example, the United States Renal Data System (USRDS) is a registry of individuals receiving dialysis that includes clinical data also as medical claims. This registry has been used to answer questions about the comparative effectiveness and safety of erythropoiesis-stimulating agents and iron in this patient population,² the comparative effectiveness of dialysis chain facilities,^three and the effectiveness of nocturnal versus daytime dialysis.⁴ Another registry is the Surveillance, Epidemiology, and Stop Results (SEER) registry, which gathers data on Americans with cancer. Much of the SEER registry's value for CER comes from its linkage to Medicare data. Examples of CER studies that make employ of this linked information include an evaluation of the effectiveness of radiofrequency ablation for hepatocellular carcinoma compared to resection or no treatment^five and a comparing of the safe of open versus radical nephrectomy in individuals with kidney cancer.^six A third example is a study that used SEER data to evaluate survival amongst individuals with bladder cancer who underwent early radical cystectomy compared to those patients who did not.⁷

Secondary Information

Much secondary information that are used for CER can be considered byproducts of clinical intendance. The framework developed past Schneeweiss and Avorn is a useful structure with which to consider the secondary sources of information generated within this context.⁸ They described the "record generation procedure," which is the information generated during patient care. Within this framework, data are generated in the creation of the newspaper-based or electronic medical (wellness) record, claims are generated and then that providers are paid for their services, and claims and dispensing records are generated at the pharmacy at the time of payment. Every bit information are not collected specifically for the research question of interest, particular attending must be paid to ensure that data quality is sufficient for the report purpose.

A thorough understanding of the wellness system in which patients receive care and the insurance products they apply is needed for a clear understanding of whether the data are probable to exist consummate or unavailable for the population of involvement. Integrated health delivery systems such as Kaiser Permanente, in which patients receive the majority of their care from providers and facilities within the organization, provide the most complete pic of patient medical care.

Electronic Health Record (EHR) Data

Electronic health records (EHRs) are used by health intendance providers to capture the details of the clinical encounter. They are importantly clinical documentation systems. They are populated with some combination of free text describing findings from the history and the physical exam; results inputted with check-boxes to bespeak positive responses; patient-reported responses to questions for recording symptoms or for screening; prefilled templates that describe normal and abnormal findings; imported text from earlier notes on the patient; and linkages to laboratory results, radiology reports and images; and specialized testing results (such as electrocardiograms, echocardiograms, or pulmonary role exam results). Some EHRs include other features, such equally period sheets of clinical results, particularly those results used in inpatient settings (east.g., blood force per unit area measurements); problem and habits lists, electronic medication administration records; medication reconciliation features; conclusion support systems and/or clinical pathways and protocols; and specialty features for the documentation needs of specialty practices. The variables that might exist accessible from EHR information are shown in Table 8.1.

Table eight.1

Information elements available in electronic wellness records and/or in administrative claims data.

As can be seen from the variable list, the details virtually an individual patient may be all-encompassing. The method of data collection is not standardized and the intervals between visits vary for every patient in accordance with usual medical practice. Of note, medication information captured in EHRs differs from data captured by pharmacy claims. While pharmacy claims comprise information on medications dispensed (including the national drug code [NDC] to identify the medication, dispensing date, days' supply, and amount dispensed), EHRs more typically contain information on medications prescribed past a clinician. Medication data from EHRs are often captured as function of the patient's medication list, which may include the medication name, order engagement, force, units, quantity, and frequency. Again depending on the specific EHR organisation, inpatient medication orders may or may non be available but are not typically. As EHRs differ substantially, information technology is important to understand what fields are captured in the EHR nether consideration, and to realize that completeness of specific fields may vary depending on how individual health care providers apply the EHR.

An additional challenge with EHR data is that patients may receive intendance at unlike facilities, and information regarding their health may exist entered separately into multiple systems that are not integrated and are inconsistent across practices. If a patient has an emergency room visit at a hospital that is not his usual site of care, it is unlikely to exist recorded in the electronic medical record that houses the majority of his clinical information. Additionally, for a patient who resides in two or more cities during the twelvemonth, the electronic medical record at each institution may be incomplete if the institutions exercise not share a mutual data system.

Paper-Based Records

Although time-intensive to access, the apply of newspaper-based records is sometimes required. Many practices nonetheless do not have EHRs; in 2009, it was estimated that but half of outpatient practices in the U.South. were using EHRs.^x Exclusion of sites without electronic records may bias study results because these sites may have different patient populations or because at that place may be regional differences in exercise. These information may be particularly valuable if patient-reported information is needed (such every bit severity of hurting, quality of symptoms, mental health concerns, and habits). The richness of information in newspaper-based records may exceed that in EHR data particularly if the electronic data is template driven. Additionally, paper-based records are valuable equally a source of primary data for validating data that is available elsewhere such as in administrative claims. With a newspaper medical record, the researcher tin exam the sensitivity and specificity of the data independent in claims data by reviewing the paper record to see if the diagnosis or procedure was described. In that situation, the paper-based record would be considered the reference standard for diagnoses and procedures.

Administrative Data

Administrative health insurance data are typically generated as function of the process of obtaining insurance reimbursement. Presently, medical claims are most often coded using the International Classification of Disease (ICD) and the Mutual Procedural Terminology (CPT) systems. The ICD, Ninth Revision, Clinical Modification (ICD-9-CM) is the official arrangement of assigning codes to diagnoses and procedures associated with hospital utilization in the Us. Much of Europe is using ICD-10 already, while the United States currently uses ICD-9 for everything except bloodshed information; the U.s.a. will kickoff using ICD-10 in October 2013.¹¹ The ICD coding terminology includes a numerical list of codes identifying diseases, also as a classification system for surgical, diagnostic, and therapeutic procedures. The National Center for Health Statistics and the Centers for Medicare and Medicaid Services (CMS) are responsible for overseeing modifications to the ICD. For outpatient encounters, the CPT is used for submitting claims for services. This terminology was initially developed past the American Medical Clan in 1966 to encourage the use of standard terms and descriptors to document procedures in the medical record, to communicate authentic information on procedures and services to agencies concerned with insurance claims, to provide the basis for a computer-oriented system to evaluate operative procedures, and for actuarial and statistical purposes. Shortly, this organization of terminology is the required nomenclature to written report outpatient medical procedures and services to U.S. public and private health insurance programs, as the ICD is the required system for diagnosis codes and inpatient infirmary services.¹² The diagnosis-related group (DRG) nomenclature is a arrangement to classify hospital cases by their ICD codes into 1 of approximately 500 groups expected to have like hospital resources utilize; it was adult for Medicare as part of the prospective payment system. The DRG organisation can be used for inquiry too, simply with the recognition that there may exist clinical heterogeneity inside a DRG. In that location is no correlate of the DRG for outpatient care.

When using these claims for research purposes, the validity of the coding is of the highest importance. This is described in more detail below. The validity of codes for procedures exceeds the validity of diagnostic codes, as procedural billing is more closely tied to reimbursement. Understandably, the motivation for coding procedures correctly is high. For diagnosis codes, all the same, a diagnosis that is nether evaluation (east.one thousand., a medical visit or a test to "rule out" a particular condition) is duplicate from a diagnosis that has been confirmed. Consequently, researchers tend to wait for sequences of diagnoses, or diagnoses followed past treatments appropriate for those diagnoses, in order to identify conditions of involvement. Although Medicare requires an appropriate diagnosis lawmaking to accompany the procedure code to authorize payment, other insurers accept looser requirements. In that location are few external motivators to lawmaking diagnoses with loftier precision, so the validity of these codes requires an understanding of the health insurance system's approach to documentation.¹³ ^- ²⁰ Investigators using claims information for CER should validate the key diagnostic and process codes in the study. There are many examples of validation studies in the literature upon which to pattern such a study.¹⁸ ^, ²¹ ^- ²² Additional codes are available in some datasets—for example, the "present on admission" code that has been required for Medicare and Medicaid billing since October 2007—which may help in further refinement of algorithms for identifying key exposures and outcomes.

Pharmacy Data

Outpatient pharmacy data include claims submitted to insurance companies for payment likewise as the records on drug dispensing kept past the pharmacy or by the pharmacy benefits manager (PBM). Claims submitted to the insurance visitor utilize the NDC as the identifier of the product. The NDC is a unique, 10-digit, 3-segment number that is a standard product identifier for human being drugs in the United States. Included in this number are the agile ingredient, the dosage form and route of administration, the strength of the product, and the package size and type. The U.S. Food and Drug Assistants (FDA) has authority over the NDC codes. Claims submitted to insurance companies for payment for drugs are submitted with the NDC lawmaking as well as data about the supply dispensed (e.g., how many days the prescription is expected to comprehend), and the amount of medication dispensed. This information can be used to provide a detailed moving-picture show of the medications dispensed to the patient. Medications for which a claim is non submitted or is not covered past the insurance plan (due east.g., over-the-counter medications) are not available. It should exist noted that claims data are by and large weak for medical devices, due to a lack of uniform coding, and claims often practise non include drugs that are not dispensed through the chemist's (e.chiliad., injections administered in a dispensary).

Large national PBMs, such every bit Medco Wellness Solutions or Caremark, administer prescription drug programs and are responsible for processing and paying prescription drug claims. They are the interface between the pharmacies and the payers, though some larger health insurers manage their own pharmacy data. PBM models differ substantially, but well-nigh maintain formularies, contract with pharmacies, and negotiate prices with drug manufacturers. The differences in formularies across PBMs may offer researchers the advantage of natural experiments, as some patients will non be dispensed a detail medication fifty-fifty when indicated, while other patients will be dispensed the medication, solely due to the formulary differences of their PBMs. Some PBMs own their ain mail service-order pharmacies, eliminating the local pharmacies' role in distributing medications. PBMs more recently have taken on roles of disease management and outcomes reporting, which generates additional data that may be attainable for research purposes. Figure 8.1 illustrates the flow of data into PBMs from health plans, pharmaceutical manufacturers, and pharmacies. PBMs contain a potentially rich source of data for CER, provided that these data can exist linked with outcomes. Examples of CERs that have been done using PBM data include two studies that evaluate patient adherence to medications equally their outcome. One compared adherence to different antihypertensive medications using data from Medco Wellness Solutions. The researchers identified differential adherence to antihypertensive drugs, which has implications for their effectiveness in practice.²³ Some other study compared costs associated with a step-therapy intervention that controlled admission to angiotensin-receptor blockers with the costs associated with open access to these drugs.²⁴ Data came from iii wellness plans that contracted with one PBM and one health plan that contracted with a unlike PBM.

Figure viii.1

How pharmacy benefits managers fit inside the payment system for prescription drugs. From the Congressional Budget Function, based in office on Full general Accounting Office, Pharmacy Benefit Managers: Early on Results on Ventures with Drug Manufacturers. GAO/HEHS-96-45. (more...)

Frequently, PBM data are accessible through health insurers along with related medical claims, thus enabling single-source access to data on both treatment and outcomes. Data from the U.South. Department of Veterans Affairs (VA) Chemist's Benefits Manager, combined with other VA data or linked to Medicare claims, are a valuable resource that has generated comparative effectiveness and safety information.²⁵ ^- ²⁶

Regulatory Data

FDA has a vast store of data from submissions for regulatory approval from manufacturers. While the majority of the submissions are not in a format that is usable for enquiry (e.g., paper-based submissions or PDFs), increasingly the submissions are in formats where the information may exist used for purposes beyond that for which they were collected, including CER. Additionally, FDA is committed to converting many of its older datasets into research-appropriate data. FDA presently has a contractor working on conversion of 101 trials into useable data that volition exist stored in their clinical trial repository.²⁷ Information technology too has pilot projects underway that are exploring the benefits and risks of providing external researchers access to their information for CER. It is recognized that issues of using proprietary data or trade-secret information will ascend, and that there may be regulatory and data-security challenges to address. A limitation of using these trials for CER is that they are typically efficacy trials rather than effectiveness trials. However, when combined using techniques of meta-analysis, they may provide a comprehensive picture of a drug'due south efficacy and brusque-term safety.

Repurposed Trial Information or Data From Completed Observational Studies

A vast amount of data is nerveless for clinical enquiry in studies funded past the Federal government. Past police, these information must exist made available upon request to other researchers, every bit this was information nerveless with taxpayer dollars. This is an exceptional source of existing information. To illustrate, the Cardiovascular Health Study is a large accomplice study that was designed to identify risk factors for coronary heart disease and stroke by means of a population-based longitudinal accomplice study.²⁸ The written report investigators nerveless diverse outcomes including information on hospitalization, specifically heart failure associated hospitalizations. Thus, the data from this study tin can be used to answer comparative effectiveness questions virtually interventions and their effectiveness on preventing heart failure complications, even though this was not a primary aim of the original cohort study. A limitation is that the researcher is limited to only the data that were collected—an important consideration when selecting a dataset. Some of the datasets have associated biospecimen repositories from which specimens can be requested for additional testing.

Completed studies with publicly bachelor datasets oftentimes can be identified through the National Institutes of Wellness found that funded the study. For example, the National Center Lung and Blood Institute has a searchable site (at https://biolincc.nhlbi.nih.gov/home/) where datasets can be identified and requested. Similarly, the National Found of Diabetes and Digestive and Kidney Diseases has a repository of datasets as well as instructions for requesting data (at https://www.niddkrepository.org/niddk/jsp/public/resource.jsp).

Considerations for Selecting Data

Required Data Elements

The enquiry question must bulldoze the choice of data. Often, however, every bit the question is developed, it becomes clear that a particular slice of information is critical to answering the question. For example, a question about interventions that reduce the amount of albuminuria will almost certainly crave admission to laboratory data that include measurement of this outcome. Reliance on ICD-9 codes or use of a statement in the medical tape that "albuminuria decreased" will be insufficiently specific for research purposes. Similarly, a written report question nearly racial differences in outcomes from coronary interventions requires data that include documentation of race; this requirement precludes use of most authoritative data from individual insurers that do not collect this information. If the relevant information are not available in an existing information source, this may exist an indication that master information collection or linking of datasets is in order. It is recommended that the investigator specify a priori what the minimum requirements of the data are before the data are identified, equally this volition help avoid the endeavor of making suboptimal data work for a given study question.

If some central information elements seem to be unobtainable in an otherwise suitable dataset, ane might consider means to supplement the available data. These strategies may be methodological, such as predicting absent-minded information variables with data that are available, or interpolating for missing time points. The authors recently completed a written report in which the presence of obesity was predicted for individuals in the dataset based on ICD-nine codes.²⁹ In such instances, it is desirable to provide a reference to support the quality of information obtained by such an approach.

Alternatively, there may exist a need to link datasets or to use already linked datasets. SEER-Medicare is an instance of an already linked dataset that combines the richness of the SEER cancer diagnosis data with claims data from Medicare.³⁰ Unique patient identifiers that can be linked across datasets (such as Social Security numbers) provide opportunities for powerful linkages with other datasets.³¹ Other methods take been developed that do not rely on the existence of unique identifiers.³² As described above, linking medical data with environmental data, population-level data, or census information provides rich datasets for addressing research questions. Privacy concerns raised by individual contributors can profoundly increase the complexity and time needed for a study with linked information.

Information linking combines data on the same person from multiple sources to increase the richness of data available in a study. This is in contrast to information pooling and networking, tools primarily used to increase the size of an observational study.

Time Period and Duration of Followup

In an ideal state of affairs, researchers take like shooting fish in a barrel admission to low-cost, clinically rich data about patients who have been continuously observed for long periods of time. This is seldom the case. Often, the question being addressed is sensitive to the time the data were nerveless. If the question is about a newly available drug or device, information technology will be essential that the data capture the time period of relevance. Other questions are less sensitive to secular changes; in these cases, older information may exist adequate.

Inadequate length of followup for individuals is often the cardinal time chemical element that makes data unusable. How long is necessary depends on the research question; in about cases, information virtually outcomes associated with specific exposures requires a period of followup that takes the natural history of the outcomes into account. Data from registries or from clinical care may be platonic for studies requiring long followup. Commercial insurers see large amounts of turnover in their covered patient populations, which often makes the length of fourth dimension that data are bachelor on a given individual relatively brusk. This is also the case with Medicaid data. The populations in data from commercial insurers or Medicaid, withal, are so large that reasonable numbers of relevant individuals with long followup can oft be identified. It should be noted that when a study population is restricted to patients with longer than typical periods of followup within a database, the representativeness of those patients should be assessed. Individuals insured by Medicare are typically insured by Medicare for the remainder of their lives, so these data are often advisable for longitudinal inquiry, especially when they tin can be coupled with data on drug use. Similarly, the VA wellness system is oftentimes a source of information for CER because of the relatively stable population that is served and the detail of the clinical information captured in the system's electronic records.

Table 8.2 provides the types of questions, with an instance for each, that an investigator should enquire when choosing data.

Tabular array 8.2

Questions to consider when choosing information.

Ensuring Quality Data

When because potential data resource for a written report, an important element is the quality of the information in the resource. Using databases with large amounts of missing information, or that do not have rigorous and standardized data editing, cleaning, and processing procedures increases the risk of inconclusive and potentially invalid study results.

Missing Information

One of the biggest concerns in any investigation is missing data. Depending on the elements and if there is a design in the type and extent of missingness, missing information can compromise the validity of the resource and any studies that are done using that data. It is important to understand what variables are more than or less likely to be missing, to define a priori an acceptable percent of missing information for central information elements required for assay, and to be aware of the efforts an organization takes to minimize the amount of missing information. For example, data resources that obtain data from medical or insurance claims will by and large have higher completion rates for information elements used in reimbursement, while optional items will exist completed less frequently. A data resources may also have unlike standards for individual versus group-level exam. For case, while ethnicity might be the only missing variable in an private record, it could be absent for a significant percentage of the report population.

Some investigators impute missing information elements under sure circumstances. For example, in a longitudinal resource, data that were previously nowadays may be carried forrard if the latest update of a patient's information is missing. Statistical imputation techniques may be used to judge or gauge missing information past modeling the characteristics of cases with missing data to those who have such data.³³ ^- ³⁵ Data that have been generated in this manner should be clearly identified so that they can be removed for sensitivity analyses, equally may be appropriate. Additional information about methods for handling missing information in analysis is covered in chapter ten.

Changes That May Alter Information Availability and Consistency Over Fourth dimension

Any data resource that collects data over time is likely to eventually come across changes in the information that will bear upon longitudinal analyses. These changes could be either a atypical event or a gradual shift in the data and can be triggered past the system that maintains the database or past events beyond the command of that organization including adjustments in diagnostic practices, coding and reimbursement modifications, or increased affliction awareness. Investigators should be enlightened of these changes as they may take a substantial effect on the study pattern, time period, and execution of the project.

Sudden changes in the database may exist dealt with by using trend breaks. These are points in time where the database is discontinuous, and analyses that cross over these points will need to be interpreted with care. Examples of trend break events might exist major database upgrades and/or redesigns or changes in data suppliers. Other trend break events that are exterior the influence of the maintenance organization might be medical coding upgrades (e.one thousand., ICD-nine to ICD-x), announcements or presentations at conferences (due east.one thousand., Women'due south Wellness Initiative findings) that may lead to changes in medical practice, or high contour drug approvals or withdrawals.

More than gradual events tin besides bear upon the data availability. Software upgrades and changes might effect in more data being bachelor for recently added participants versus individuals who were captured in prior versions. Changes in reimbursement and recommended practise could lead to shifts in utilize of ICD-9 codes, or to more than or less information being entered for individuals.

Validity of Key Data Definitions

Validity assessment of key information in an investigation is an of import just sometimes overlooked consequence in wellness care research using secondary information. There is a need to assess not only the general definition of key variables, just also their reliability and validity in the particular database chosen for the analysis. In some cases, particularly for data resource commonly used for research, other researchers or the organization may have validated outcomes of health events (e.1000., middle set on, hospitalization, or mortality).³⁶ Creating the best definitions for key variables may require the involvement of knowledgeable clinicians who might suggest that the occurrence of a specific process or a prescription would strengthen the specificity of a diagnosis. Knowing the validity of other key variables, such as race/ethnicity, inside a specific dataset is essential, specially if results will be described in these subgroups.

Ideally, validity is examined past comparison study data to additional or alternative records that represent a "golden standard," such as paper-based medical records. We described in the Authoritative Information department in a higher place how validity of diagnoses associated with administrative claims might exist assessed relative to paper-based records. EHRs and non–claims-based resource practise non ever allow for this type of assessment, only a more accommodating validation procedure has not all the same been developed. When a patient's principal wellness care record is electronic, in that location may not be a paper trail to follow. Commonly, all activity is integrated into one record, so there is no additional documentation. On the other mitt, if the data resource pulls information from a switch company (an organisation that specializes in routing claims between the indicate of service and an insurance visitor), there may be no machinery to discover additional medical information for patients. In those cases, the information included in the database is all that is bachelor to researchers.

Information Privacy Issues

Data privacy is an ongoing business in the field of wellness care inquiry. Most researchers are familiar with the Health Insurance Portability and Accountability Human activity (HIPAA), enacted in 1996 in function to standardize the security and privacy of health care information. HIPAA coined the term "protected wellness information" (PHI), defined every bit whatsoever individually identifiable health information (45 CFR 160.103). HIPAA requires that patients exist informed of the use of their PHI and that covered entities (generally, health intendance clearinghouses, employer-sponsored health plans, wellness insurers, and medical service providers) track the use of PHI. HIPAA also provides a mechanism for patients to report when they feel these regulations have been violated.³⁷

In practical terms, this has resulted in an increase in the corporeality and complexity of documentation and permissions required to conduct healthcare research and a decrease in patient recruitment and participation levels.³⁸ ^- ³⁹ While many data resource have established procedures that allow for access to data without personal identifiers, obtaining permission to use identifiable information from existing data sources (e.g., from nautical chart review) or for master data collection can exist time consuming. Additionally, some organizations will non allow research to proceed beyond a sure point (east.k., beginning or completing statistical analyses, dissemination, or publication of results) without proper institutional review lath approvals in place. If a non-U.Due south. data resources is being used, researchers will demand to be aware of differences between U.Due south. privacy regulations and those in the country where the information resource resides.

Adherence to HIPAA regulations can too impact report blueprint considerations. For case, since birth, admission, and discharge dates are all considered to be PHI, researchers may demand to utilize a patient's age at admission and length of stay equally unique identifiers. Alternatively, a limited information fix that includes PHI but no direct patient identifiers such as proper noun, address, or medical record numbers may be defined and transferred with appropriate data use agreements in place. Organizations may have their own unique limits on data sharing and pooling. For example, in the VA system, the general records and records for status-specific treatment, such every bit HIV handling, may not be pooled. Additional information regarding HIPAA regulations every bit they apply to data used for inquiry may be constitute on the National Institutes of Health Web site.^twoscore

Emerging Problems and Opportunities

Data From Outside of the United states of america

Where appropriate, not-U.S. databases may exist considered to address CER questions, particularly for longitudinal studies. One of the main reasons is that, unlike the majority of U.Due south. health care systems, several countries with unmarried-payer systems, such every bit Canada, the Great britain, and the Netherlands, have regional or national EMR systems. This makes it much easier to obtain complete, long-term medical records and to follow individuals in longitudinal studies.⁴¹

The Clinical Do Research Datalink (CPRD) is a collection of anonymized master intendance medical records from selected general practices beyond the United kingdom. These data have been linked to many other datasets to address comparative effectiveness questions. An example is a study that linked the CPRD to the Myocardial Ischaemia National Audit Project registry in England and Wales. The researchers answered questions about the risks associated with discontinuing clopidogrel therapy after a myocardial infarction (performed when the database was called General Practise Research Database).⁴²

While the selection of a non-U.South. information source may be the right choice for a given study, there are a number of things to consider when designing a study using one of these resources.

One of the main considerations is if the study question can be appropriately addressed using a non-U.South. resources. Questions that should be addressed during the study blueprint procedure include:

Is the exposure of interest similar betwixt the report and target population? For example, if the exposure is a drug product, is information technology bachelor in the aforementioned dose and class in the data resource? Is it used in the same way and frequency as in the The states?
Are at that place any differences in availability, toll, practice, or prescribing guidelines between the written report and target populations? Has the product been bachelor in the study population and the Usa for similar periods of time?
What is the difference between the wellness intendance systems of the written report and target populations? Are in that location differences in diagnosis methods and handling patterns for the outcome of involvement? Does the outcome of involvement occur with the aforementioned frequency and severity in the study and target populations?
Are the comparator treatments similar to those that would be available and used in the United States?

An additional consideration is information admission. Access to some resources, such every bit the United kingdom of great britain and northern ireland's CPRD, can exist purchased by interested researchers. Others, such as Canada's regional health care resources, may require the personal interest of and an official association with investigators in that country who are authorized to use the system. If a non-U.S. information resource is appropriate for a proposed study, the researcher will demand to become familiar with the process for accessing the data and allow for any actress time and effort required to obtain permission to use it.

A sound justification for selecting a non-U.S. data resource, a solid understanding of the similarities and differences of the non-U.S. versus the U.S. systems, too as conscientious discussion of whether the results of the study can exist generalized to U.Southward. populations will help other researchers and wellness care practitioners interpret and utilise the results of non-U.Due south.-based inquiry to their detail situations.

Signal of Care Data Collection and Interactive Vocalism Response/Other Technologies

Traditionally, the information used in epidemiologic studies have been gathered at one bespeak in fourth dimension, cleaned, edited, and formatted for research use at a later bespeak. As technology has developed, all the same, data collected close to the point of care increasingly have been available for analysis. Prescription claims tin be available for research in as little equally i week.

In conjunction with a shortened turnaround time for data availability, the indicate at which information are coded and edited for research is also occurring closer to when the patient received care. Many people are familiar with health intendance encounters where the doc takes notes, which are then transcribed and coded for use. With the advent of EHRs, health information is at present coded and transcribed into a searchable format at the time of the visit; that is, the information is direct coded as it is collected, rather than being transcribed afterward.

Another innovation is using computers to collect data. Computer-aided information collection has been used in national surveys since the 1990s⁴³ and also in types of research (such as risky behaviors, addiction, and mental health) where respondents might not exist comfortable responding to a personal interviewer.⁴⁴ ^- ⁴⁶

The advantages of these new and timely data streams are more than detailed data, sometimes available in real or near-real time that can exist used to spot trends or patterns. Since data tin can be recorded at the time of care by the wellness care provider, this may help minimize miscoding and misinterpretation. Computerized data collection and Interactive Voice Response are becoming easier and less expensive to use, enabling investigators to reach more participants more hands. Some disadvantages are that these information streams are ofttimes specialized (e.thousand., bedside prescribing), and, without linkage to other patient characteristics, it can be hard to rail unique patients. Also, depending on the survey population, it can be challenging to maintain current telephone numbers.⁴⁷ ^- ⁴⁸

Data Pooling and Networking

A major claiming in health research is studying rare outcomes, particularly in clan with common exposures. Two methods that can be used to accost this challenge are information pooling and networking. Information pooling is combining data, at the level of the unit of analysis (i.east., individual), from several sources into a unmarried accomplice for analysis. Pooled data may besides include data from unanalyzed and unpublished investigations, helping to minimize the potential for publication bias. All the same, pooled analyses crave close coordination and can be very hard to complete due to differences in written report methodology and collection practices. An example is an analysis that pooled primary data from four cohorts of breast cancer survivors to ask a new question well-nigh the effectiveness of physical activity. The researchers had to ensure the comparability of the definitions of physical activity and its intensity in each cohort.⁴⁹ Another instance is a written report that pooled data from four different data systems including from Medicare, Medicaid, and a private insurer to assess the comparative safety of biological products in rheumatologic diseases. The authors depict their assessment of the comparability of covariates across the data systems.⁵⁰ Researchers must exist sensitive to whether additional informed consent of individuals is needed for using their data in combination with other information. Furthermore, privacy concerns sometimes exercise non allow for the actual combination of raw study information.⁵¹

An alternative to data pooling is data networking, sometimes referred to as virtual data networks or distributed inquiry networks. These networks have become possible every bit engineering science has developed to allow more sophisticated linkages. In this situation, common protocols, data definitions, and programming are developed for several data resource. The results of these analyses are combined in a central location, but individual study data do not leave the original data resource site. The advantage of this is that data security concerns may be fewer. As with information pooling, the differences in definitions and use of terminology requires that there be careful adjudication earlier the data is combined for analyses. Examples of data networking are the HMO Enquiry Network and FDA's Sentinel Initiative.⁵² ^- ⁵⁴

The advantage of these methods is the ability to create large datasets to written report rare exposures and outcomes. Information pooling can be preferable to meta-analyses that combine the results of published studies considering unified guidelines can be developed for inclusion criteria, exposures, and outcomes, and analyses using individual patient level data allow for adjustment for differences across datasets. Often, creation and maintenance of these datasets can exist time consuming and expensive, and they by and large require all-encompassing administrative and scientific negotiation, but they tin can be a rich resource for CER.

Personal Health Records

Although they are not presently used for enquiry to a significant extent, personal wellness records (PHRs) an alternative to electronic medical records. Typically, PHRs are electronically stored health records that are initiated past the patient. The patient enters information about his or her wellness care encounters, test results, and, potentially, responses to surveys or documentation of medication utilise. Many of these electronic formats are Web-based and therefore easily accessible by the patient when receiving health care in diverse settings. The application that is used by the patient may exist one for which he or she has purchased access, or it may be sponsored past the health care setting or insurer with which the patient has contact. Other PHRs, such as HealthVault and NoMoreClipboard, can be accessed freely. 1 case of a widely used PHR is MyHealtheVet, which is the personal wellness tape provided by the VA to the veterans who use its wellness care system.⁵⁵ MyHealtheVet is an integrated system in which the patient-entered data are combined with the EHR and with health management tools.

While at that place is ongoing research almost how to all-time improve patient outcomes through the creative apply of personal health records, there is also interest in how to best apply the rich data independent inside the personal wellness records for enquiry. Outstanding bug remain regarding data buying, but there is consensus that the data entered in the personal health record belongs to the patient and cannot be accessed without patient consent, which may include explicit documentation of the level of information-sharing that the patient would allow, at the time of entering data into the record. Many PHRs request that the patient state to whom he or she grants permission to access portions of the data.

Work is underway to standardize information collection across PHRs through the use of common terminologies such as the SNOMED CT (Systematized Nomenclature of Medicine—Clinical Terms) system. Before long, the National Library of Medicine (NLM) PHR projection is validating and improving the NLM'due south clinical vocabularies and studying consumers' employ of PHR systems. In 2010, the NLM researchers reviewed and enhanced the controlled vocabulary for more than two,000 condition names and synonyms and more than than 300 surgery procedure names by enriching the synonymy, providing the consumer-friendly name when feasible, and adding SNOMED codes, when available, to these items.⁵⁶

Patient-Reported Outcomes

Patient-reported outcomes (PROs) may occasionally be bachelor in paper-based records and EHRs, simply they are non soon plant in administrative data. Wu et al. described several strategies that could be employed to increase the availability of PROs in administrative data.⁵⁷ The first is to encourage routine drove of PROs in clinical intendance by requiring information technology for compliance with information quality assurance guidelines. The Infirmary Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey administered past CMS assesses patient'south perspectives on their hospital care and could be a required activity. Another strategy, as described by Wu et al., is the required participation of all Medicare managed intendance plans with Medicare Advantage contracts in the Medicare Health Outcomes Survey, which collects data like to that in the SF-12 Short-Form Health Survey. A tertiary example may be provider reimbursement for collecting symptom-related outcome data, and thus its required reporting in administrative data. None of these approaches are currently widely used. Creative interventions to increase the availability of PROs in administrative data, ideally collected with validated tools and instruments, would exist valuable to CER. Primary information collection of PRO information remains the nearly common means of ensuring that required PRO data are available on the patient population of interest at the required time points and of adequate completeness in club to acquit CER.

Conclusion

The choice of study data needs to be driven by the research question. Not all research questions can be answered with existing data, and some questions will thus crave principal data drove. For questions amenable to the employ of secondary data, observational research with existing data tin exist efficient and powerful. Investigators accept a growing number of options from which to choose when looking for appropriate data, from clinical data to claims information to existing trial or cohort data. Each selection has strengths and limitations, and the researcher is urged to make a careful match. In the end, the validity of the written report is but as proficient every bit the quality of the information.

Checklist: Guidance and cardinal considerations for data source pick for a CER protocol

View in own window

Guidance	Key Considerations	Check
Suggest data source(s) that include information required to address the primary and secondary research questions.	– Ensure that the data resource is appropriate for addressing the study question. – Ensure that the key variables needed to conduct the study are available in the data source.	□
Draw details of the data source(s) selected for the study.	– Nature of the information (claims, paper, or electronic medical records; if prospective, how the information is/was collected and from whom). – Coding arrangement(due south) that may exist used (e.m., ICD9 or ICD10; HCPCS; etc.) – Population included in the data source (ages, geography, etc.). – Other features (e.g., wellness programme membership; retention rate [i.due east., boilerplate duration of followup for members in the database, proportion of patients with followup sufficiently long for the study purpose]). – Time period covered by the information source(s). If non-U.S., draw relevant differences in health care and how this volition affect the results.	□
Draw validation or other quality assessments that have been conducted on the data source that are relevant to the data elements required for the report.	– If validation/quality assessments have not previously been performed, propose a method to assess data quality.	□
Depict what patient identifiers are necessary for the research purpose, how they will be protected, and what permissions/waivers will exist required.		□
Provide details on whatever data linkage arroyo, and the quality/accuracy of the linkage, if applicable.	– Provide enough particular to clarify the quality of the linkage approach.	□

: HCPCS = Healthcare Common Process Coding System, ICD = International Classification of Disease

References

1.: Bangsberg DR, Ragland One thousand, Monk A, et al. A single tablet regimen is associated with higher adherence and viral suppression than multiple tablet regimens in HIV+ homeless and marginally housed people. AIDS. 2010 Nov 27;24(eighteen):2835–forty. [PMC free article: PMC3540404] [PubMed: 21045636]
2.: Brookhart MA, Schneeweiss S, Avorn J, et al. Comparative bloodshed risk of anemia management practices in incident hemodialysis patients. JAMA. 2010 March iii;303(9):857–64. [PubMed: 20197532]
three.: Zhang Y, Cotter DJ, Thamer K. The effect of dialysis chains on mortality among patients receiving hemodialysis. Health Serv Res. 2011 June;46(3):747–67. [PMC gratuitous article: PMC3097400] [PubMed: 21143480]
5.: Massarweh NN, Park JO, Yeung RS, et al. Comparative assessment of the safe and effectiveness of radiofrequency ablation amongst elderly Medicare beneficiaries with hepatocellular carcinoma. Ann Surg Oncol. 2012 Apr;xix(4):1058–65. [PMC complimentary commodity: PMC3264841] [PubMed: 21947695]
6.: Tan HJ, Wolf JS Jr., Ye Z, et al. Complications and failure to rescue afterwards laparoscopic versus open up radical nephrectomy. J Urol. 2011 October;186(iv):1254–60. [PubMed: 21849185]
7.: Amble D, Egleston B, Wong YN, et al. Use of radical cystectomy equally initial therapy for the treatment of high-grade T1 urothelial carcinoma of the bladder: A SEER database analysis. Urol Oncol. 2011 September 7 [epub ahead of print] [PubMed: 21906968]
8.: Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic enquiry on therapeutics. J Clin Epidemiol. 2005 April;58(four):323–37. [PubMed: 15862718]
nine.: Botsis T, Hartvigsen 1000, Chen F, et al. Secondary Utilize of EHR: Data Quality Issues and Informatics Opportunities. AMIA Summits Transl Sci Proc. 2010 Mar 1;2010:1–v. [PMC free article: PMC3041534] [PubMed: 21347133]
x.: Hsiao CJ, Hing Due east, Socey TC. Electronic medical tape/electronic health record systems of office-based physicians: United states, 2009 and preliminary 2010 state estimates. National Heart for Health Statistics; Hyattsville, MD: 2009.
xi.
12.
xiii.: Fisher ES, Whaley FS, Krushat WM, et al. The accuracy of Medicare's hospital claims information: progress has been made, but issues remain. Am J Public Health. 1992 February;82(ii):243–8. [PMC free article: PMC1694279] [PubMed: 1739155]
14.: Quan H, Parsons GA, Ghali WA. Validity of process codes in International Nomenclature of Diseases, 9th revision, clinical modification administrative data. Med Intendance. 2004 August;42(eight):801–9. [PubMed: 15258482]
xv.: Quan H, Parsons GA, Ghali WA. Assessing accurateness of diagnosis-type indicators for flagging complications in administrative data. J Clin Epidemiol. 2004 April;57(4):366–72. [PubMed: 15135837]
xvi.: Quan H, Li B, Saunders LD, et al. Assessing validity of ICD-9-CM and ICD-10 authoritative information in recording clinical conditions in a unique dually coded database. Health Serv Res. 2008 August;43(4):1424–41. [PMC costless commodity: PMC2517283] [PubMed: 18756617]
17.: Segal JB, Ness PM, Powe NR. Validating billing data for RBC transfusions: a brief study. Transfusion. 2001 Apr;41(4):530–iii. [PubMed: 11316905]
18.: Segal JB, Powe NR. Accuracy of identification of patients with allowed thrombocytopenic purpura through administrative records: a information validation study. Am J Hematol. 2004 Jan;75(1):12–7. [PubMed: 14695627]
19.: Strom BL. Data validity issues in using claims data. Pharmacoepidemiol Drug Saf. 2001 Baronial;10(5):389–92. [PubMed: 11802582]
20.: Thirumurthi Southward, Chowdhury R, Richardson P, et al. Validation of ICD-ix-CM diagnostic codes for inflammatory bowel disease amidst veterans. Dig Dis Sci. 2010 September;55(9):2592–viii. [PubMed: 20033847]
21.: Stein BD, Bautista A, Schumock GT, et al. The validity of ICD-9-CM diagnosis codes for identifying patients hospitalized for COPD exacerbations. Breast. 2012 Jan;141(one):87–93. [PMC free commodity: PMC3251268] [PubMed: 21757568]
24.: Yokoyama One thousand, Yang West, Preblick R, et al. Effects of a step-therapy program for angiotensin receptor blockers on antihypertensive medication utilization patterns and toll of drug therapy. J Manag Care Pharm. 2007 April;13(3):235–44. [PubMed: 17407390]
25.: Hachem C, Morgan R, Johnson 1000, et al. Statins and the adventure of colorectal carcinoma: a nested case-command study in veterans with diabetes. Am J Gastroenterol. 2009 May;104(5):1241–8. [PMC free article: PMC4144072] [PubMed: 19352344]
26.: Iqbal SU, Cunningham F, Lee A, et al. Divalproex sodium vs. valproic acrid: drug utilization patterns, persistence rates and predictors of hospitalization amid VA patients diagnosed with bipolar disorder. J Clin Pharm Ther. 2007 December;32(6):625–32. [PubMed: 18021341]
27.
28.: Fried LP, Borhani NO, Enright P, et al. The Cardiovascular Health Study: blueprint and rationale. Ann Epidemiol. 1991 February;1(three):263–76. [PubMed: 1669507]
29.: Clark JM, Chang HY, Bolen SD, et al. Development of a claims-based risk score to identify obese individuals. Popul Wellness Manag. 2010 August;13(4):201–7. [PubMed: 20443698]
thirty.
31.: Jutte DP, Roos LL, Brownell MD. Administrative record linkage equally a tool for public health research. Annu Rev Public Health. 2011 April 21;32:91–108. [PubMed: 21219160]
33.: Greenland South, Finkle WD. A disquisitional await at methods for handling missing covariates in epidemiologic regression analyses. Am J Epidemiol. 1995 December 15;142(12):1255–64. [PubMed: 7503045]
34.: Klebanoff MA, Cole SR. Apply of multiple imputation in the epidemiologic literature. Am J Epidemiol. 2008 August 15;168(4):355–7. [PMC free article: PMC2561989] [PubMed: 18591202]
35.: Mackinnon A. The use and reporting of multiple imputation in medical research - a review. J Intern Med. 2010 December;268(six):586–93. [PubMed: 20831627]
36.: Foundation for the National Institues of Wellness. Observational Medical Outcomes Partnership. [May 8, 2011]. Available at: http://omop.fnih.org/node/22.
37.: Gunn PP, Fremont AM, Bottrell Yard, et al. The Health Insurance Portability and Accountability Human action Privacy Rule: a practical guide for researchers. Med Care. 2004 April;42(4):321–7. [PubMed: 15076808]
twoscore.
41.: Schneeweiss Southward, Setoguchi S, Brookhart A, et al. Hazard of expiry associated with the use of conventional versus singular antipsychotic drugs among elderly patients. CMAJ. 2007 February 27;176(five):627–32. [PMC gratuitous article: PMC1800321] [PubMed: 17325327]
42.: Boggon R, van Staa TP, Timmis A, et al. Clopidogrel discontinuation after astute coronary syndromes: frequency, predictors and associations with expiry and myocardial infarction—a hospital registry-primary care linked accomplice (MINAP-GPRD). Eur Center J. 2011 Oct;32(xix):2376–86. [PMC gratuitous article: PMC3184230] [PubMed: 21875855]
43.
44.: Perlis TE, Des Jarlais DC, Friedman SR, et al. Audio-computerized self-interviewing versus face up-to-confront interviewing for research information collection at drug corruption treatment programs. Addiction. 2004 July;99(7):885–96. [PubMed: 15200584]
45.: Fairley CK, Sze JK, Vodstrcil LA, et al. Calculator-assisted self interviewing in sexual wellness clinics. Sexual activity Transm Dis. 2010 Nov;37(11):665–eight. [PubMed: 20975481]
46.: Richens J, Copas A, Sadiq ST, et al. A randomised controlled trial of reckoner-assisted interviewing in sexual wellness clinics. Sexual activity Transm Infect. 2010 August;86(4):310–4. [PubMed: 20551234]
47.: Corkrey R, Parkinson L. Interactive voice response: review of studies 1989-2000. Behav Res Methods Instrum Comput. 2002 August;34(3):342–53. [PubMed: 12395550]
48.: Abu-Hasaballah Yard, James A, Aseltine RH Jr. Lessons and pitfalls of interactive voice response in medical research. Contemp Clin Trials. 2007 September;28(5):593–602. [PubMed: 17400520]
49.: Beasley JM, Kwan ML, Chen WY, et al. Meeting the physical activity guidelines and survival after chest cancer: findings from the afterwards breast cancer pooling project. Breast Cancer Res Treat. 2012 January;131(ii):637–43. [PMC gratuitous article: PMC3272362] [PubMed: 21935600]
50.: Herrinton LJ, Curtis JR, Chen L, et al. Study pattern for a comprehensive assessment of biologic condom using multiple healthcare data systems. Pharmacoepidemiol Drug Saf. 2011 Nov;20(xi):1199–209. [PMC free commodity: PMC3312584] [PubMed: 21919113]
52.: Selby JV. Linking automatic databases for enquiry in managed intendance settings. Ann Intern Med. 1997 October 15;127(8 Pt 2):719–24. [PubMed: 9382386]
53.: Platt R, Davis R, Finkelstein J, et al. Multicenter epidemiologic and health services research on therapeutics in the HMO Research Network Middle for Education and Enquiry on Therapeutics. Pharmacoepidemiol Drug Saf. 2001 Baronial;x(5):373–7. [PubMed: 11802579]
54.: Harcourt SE, Smith GE, Elliot AJ, et al. Use of a large full general exercise syndromic surveillance organisation to monitor the progress of the influenza A(H1N1) pandemic 2009 in the Uk. Epidemiol Infect. 2011 April 8:1–6. [PubMed: 21473803]
55.: Nazi KM, Hogan TP, Wagner Thursday, et al. Embracing a health services inquiry perspective on personal health records: lessons learned from the VA My HealtheVet system. J Gen Intern Med. 2010 January;25 1 Suppl 1:62–7. [PMC free commodity: PMC2806958] [PubMed: 20077154]
56.
57.: Wu AW, Snyder C, Clancy CM, et al. Adding the patient perspective to comparative effectiveness research. Wellness Aff (Millwood). 2010 Oct;29(10):1863–71. [PubMed: 20921487]

*: Disclaimer: The views expressed are the authors' and not necessarily those of the Food and Drug Assistants.