Abstract

DATA AND METHODS

We conducted a factorial experiment to simultaneously measure the effects of: (a) patient attributes (age, gender, race and socioeconomic status); (b) physician characteristics (gender and years of clinical experience); and (c) separate healthcare systems (the US, the UK, or Germany) on physician diagnostic certainty and subsequent medical decision making when providers are presented with identical signs and symptoms indicative of CHD. Experiments were conducted in the US (Massachusetts), the UK (the West Midlands, SE London and Surrey), and Germany (Northern Rhine/Westfalia region) (42). A full factorial of combinations of patient age (55 vs. 75), gender, race (white vs. black in the US; white vs. Afro Caribbean in the UK; and white only in Germany) and SES (lower vs. higher social class, depicted by occupation as a cleaner/janitor vs. a teacher) was used for the videotaped vignette scenarios (2⁴ = 16 unique vignettes). The decision to omit the race factor from the German experiment was based on discussions with our German colleagues, who advised us that the physicians in our sampling area saw few Black patients in their everyday practices. Audios for the vignettes were dubbed into German and backtranslated to ensure accuracy. One of the 16 combinations was shown to each physician.

CHD was selected for the vignettes because: a) it is among the most common and costly problems presented by older patients to primary care providers (43); b) it represents a clinically well-defined medical condition; c) it admits a range of diagnostic, therapeutic and lifestyle actions; and d) its reported prevalence differs among the US, the UK, and Germany. Scripts for the vignettes were developed from several tape-recorded role-playing sessions with experienced clinical advisors, and professional actors were trained (under experienced physician supervision) to realistically portray a patient presenting with the signs/symptoms of disease to a primary care provider. Patients in the vignette presented with signs and symptoms that were consistent with CHD, including chest pain worsening with exertion, pain in the back between the shoulder blades, stress, and elevated blood pressure. All patients included a non-verbal “Levine’s fist,” a well-known gesture indicative of cardiac pain. Because CHD is a spectrum condition, and live patients do not typically present as clear-cut textbook cases of specific conditions, the vignette also built in several red herring symptoms potentially indicative of a gastrointestinal diagnosis. To this end, the patient also complained of indigestion, feeling worse after a large or spicy meal, having pain similar to heartburn but unresponsive to antacids, and feeling full and “gassy.” This was done not to specifically make the physicians’ diagnostic task more difficult, but to more accurately represent how actual patients present, based on advice from our clinical advisors. The vignette also incorporated references to the patient’s mood, including the spouse’s report that the patient has been difficult to be around and the patient’s self-report of feeling irritated and having decreased energy (see Appendix A for an illustrative excerpt from the vignette).

After viewing the videotaped vignette, physicians were asked, “What do you think is going on with this patient?”, and for each possibility, they were asked for their level of certainty on a scale of 0–100 (where 0 = no certainty and 100 = complete certainty). Physicians were asked through an open-ended question from the interviewer to list their full set of differential diagnoses, and if CHD was present anywhere on that list they were counted as having considered the condition and the accompanying certainty level was included in the present analysis. Physicians were also asked how they would manage the patient in terms of asking for addition information, performing physical examination, ordering tests, prescribing medications, giving lifestyle advice, and referring to other physicians.

Based on the theoretical approach outlined above and our concerns about the social construction of epidemiologic base rates, we do not appeal to population rates of disease to ascertain the “correctness” of a given diagnosis. Rather, we assume that while a physician’s diagnostic priors (among many other sources of learning) may inform decision making, patient-specific clinical information should be used above and beyond those pre-existing base rates. Therefore, the vignette for this study was purposely designed to present a set of signs and symptoms sufficient to trigger a CHD diagnosis, regardless the epidemiologic prevalence in any sub-group of the population. In this sense, the vignette provides sufficient information to suspect the condition, regardless of the social characteristics of the patient.

The study used a probability sample of physicians selected from within each of four strata within each country. To be eligible for selection, physicians had to: (a) be internists or family practitioners in the US and in Germany, or general practitioners in the UK (to most accurately capture the types of non-specialist physicians most likely to treat undiagnosed cases of CHD in each country); (b) be trained at an accredited medical school in the country in which they practiced (no international medical graduates were included); and (c) be currently in clinical practice more than half-time. Within each country, physicians were stratified into four equal cells by gender and level of experience, with “less” experience defined as those with ≤12 years since graduation from medical school in the US or UK (≤ 7 years since licensure in Germany) and those with “more” experience having ≥22 years since graduation from medical school in the US or UK (≥17 years since licensure in Germany). These cutoff dates were chosen to act as a proxy for clinical experience (which has been shown to affect clinical decision making, as discussed above); to standardize as much as possible the amounts of clinical experience across countries; and to create a clear separation between the strata. Twelve strata of physician characteristics (gender, years of clinical experience [<12 or >22 years] and health care system [US/UK/Germany]) were defined, with 32 physicians included in each stratum from each country. This configuration generated a total of 384 physicians required to complete the design the experiment (16 vignettes x 12 physician strata x 2 replications = 384)(Table 1).

Screening telephone calls were conducted to identify eligible subjects and an hour-long appointment was scheduled for the interviewer to come to the physician’s place of work to show the vignette and administer related questions. The interviews were conducted in 2001–2 (128 throughout Massachusetts, 64 from The Midlands and 64 throughout Surrey and SE London, England) and in 2005–6 in Germany; no more than one physician was selected from each practice. Each physician subject was provided a modest stipend to partially offset lost revenue [$100 (US), £50 (UK), 100 Euros (Germany)] and to acknowledge his or her participation. Quality control interviews and site visits were conducted and selected tape-recorded interviews were reviewed by supervisors on a regular basis.

Analysis of variance was used to test the main effects and two-way interactions of the design variables (patient gender, race, age, and SES, and physician gender and level of experience) on the diagnostic certainty (0–100, with 0 for not at all certain and 100 for completely certain). If the physician did not make a CHD diagnosis, his or her certainty for these diagnoses was set to 0. To determine the effect of certainty on clinical decision making, logistic regression was used for dichotomous variables (e.g., whether or not an EKG was ordered) and analysis of covariance was used for continuous variables (e.g., number of days to next appointment). Each model included as explanatory variables the design variables, certainty, and the interaction of the design variables and certainty. Using backwards elimination, non-significant effects (at the 0.05 level) other than certainty were removed from the model, leaving a parsimonious model. Due to the challenges of multiple testing, we emphasize consistency across results and focus on identifying general patterns of physician certainty and treatment decisions. Furthermore, the results we observe at the p < .01 level are unlikely to change. To facilitate interpretation, we present actual p-values, unadjusted for multiple testing. To further facilitate interpretation of results, we indicate in Tables 2 and ​and33 the number of expected and observed significant results.

RESULTS

2. Which physicians were the most certain?

We next examined whether physician characteristics (gender and years of experience) were associated with certainty, independent of the health care system in which providers practiced (Table 2). While there were no significant main effects for physician characteristics, we did observe an interaction between patient race and physician level of experience (Figure 2a). With white patients, physicians had comparable certainty levels (54.1 vs. 53.0, less and more experienced, respectively). With black patients, however, more experienced physicians had increased certainty (56.9) while those with less experience were less certain (44.5).

Open in a separate window

Open in a separate window

Open in a separate window

Open in a separate window

Figure 2

Figure 2a. Two-way interaction effect of patient race and physician level of experience on physicians’ diagnostic certainty for coronary heart disease (CHD), 0 to 100 (United States and United Kingdom only; P = 0.0500).

Figure 2b. Two-way interaction effect of patient age and patient gender on physicians’ diagnostic certainty for CHD, 0 to 100 (P = 0.0086).

Figure 2c. Two-way interaction effect of patient age and country on physicians’ diagnostic certainty for CHD, 0 to 100 (P = 0.0100).

Figure 2d. Two-way interaction effect of patient race and country on physicians’ diagnostic certainty for CHD, 0 to 100 (P = 0.0021).

DISCUSSION

We observed significant differences between health care systems, with US physicians having the highest levels of certainty. As expected, physicians were least certain of their CHD diagnoses with younger female patients (12, 44). In addition, there was racial variation depending on the physician’s level of experience and variation by health care system depending on patient age and race. Increased certainty was associated with differences in information seeking as well as increased test ordering, prescriptions, referrals to cardiologists, and shorter time to follow up.

Previous studies of medical practice variation have largely focused on either system-level social and economic patterns in CDM or on physicians’ individual-level cognitive processing as mechanisms that may generate such variation. The factorial design of our experiment allowed for unconfounded estimates of the simultaneous effects of patient characteristics, provider attributes, and healthcare systems (as represented by country) on physicians’ certainty of CHD diagnoses and their subsequent clinical actions. With our analytic approach, we were able to capture physicians at all points on the certainty spectrum rather than excluding those who did not list CHD among their differential diagnosis selections. While those who did not consider CHD among their differential diagnosis selections did not provide an explicit certainty value, we know their certainty concerning the presence of CHD was low because they were allowed to list a full set of diagnoses (we recognize that physicians may have high certainty that CHD is absent from the vignette, but this is beyond the scope of the present analysis). Therefore, if they were treated as missing they would not be missing at random. By increasing the variability, our estimates are rendered more conservative due to a decrease in power.

The sample size of 384 allows us to detect a difference in certainty of 12 points with 97% power, assuming a standard deviation of 30 (which, assuming a unimodal Beta distribution of 0–1, is the upper bound of a standard deviation). That is, a true 12 point difference in certainty between two groups will be detected 97 percent of the time at α = 0.05. Because the experiment was replicated, a pure error term with 192 degrees of freedom was used to test all effects using analysis of variance. Due to the omission of the race factor for the German experiment (explained above), only the US and UK data are considered for the effect of patient race on certainty. For all other analyses, all data were considered.

The main effects related to patient characteristics (decreased certainty with female and younger patients) partially corroborate Bayesian perspectives and studies of uncertainty and statistical discrimination (31, 45) suggesting that when physicians are uncertain, they are likely to make diagnostic decisions that are consistent with existing epidemiologic base rates. In this case prior assumptions overwhelm the presenting patient-specific data, thereby contributing to the reification of some types of existing health statistics. Most importantly, these findings extend previous work by showing that certainty—not simply identifying a diagnosis, but having diagnostic certainty about that condition—has an independent effect on clinical actions (46). Therefore, these results suggest that having CHD on the differential diagnosis list is necessary, but not sufficient, for physicians to take appropriate therapeutic actions; this result is consistent with the notion that physicians need to pass certainty thresholds in order to order more tests or treat a patient (28). By extension, improving disparities in CHD outcomes is not just a matter of physicians learning to more appropriately consider CHD in specific populations (for example, women), but also to be able to do so with sufficient certainty to trigger appropriate clinical actions to improve morbidity and mortality outcomes.

However, these perspectives do not fully explain our results and the persistence of some between-health care system differences implies that features of the broader sociological, cultural, and organizational environments are also relevant to decision making. Beyond diagnostic certainty, observed cross-national variation in CHD diagnosis and treatment may also be a function of differences in a series of influences that are beyond the scope of the present analysis or our study more generally. These include both patients’ and physicians’ cultural expectations for medical practice and treatment as well as biological variations in the prevalence of CHD and related conditions at a population level, such that physicians in different healthcare systems may be differentially equipped to identify CHD with certainty. For example, previous research has shown cross-cultural differences in the relationship between symptoms and underlying conditions (47, 48). Other possible explanations include funding mechanisms, expectations for physicians to achieve diagnostic certainty in a brief period of time, modes of practice, and access to resources across the three countries. For example, economic reimbursement policies may translate to more pressure on US physicians to achieve a firm diagnosis and management plan during the initial patient consultation so they can be paid, whereas in the UK and Germany physicians may tend to make these decisions over a series of consultations close together in time. Similarly, ready availability of technological equipment in Massachusetts in conjunction with a fear of lawsuits for missed diagnoses may lead to increased testing in the US relative to physicians in the UK or Germany, while increased rates of referral among UK and German physicians (25) may explain lower rates of testing and prescription treatment relative to the US. Increased regulation in the UK with pay-for-performance may also contribute to the U.K. physicians’ higher certainty relative to their German counterparts, where professional pressures and regulations are not yet as explicit as in the U.S. (49). Most generally, physician learning is known to be related to the local context of practice, such that physicians may either self-select into environments with practice styles like their own, or they may adapt to the local culture—either type of pattern could exist within the local contexts that were selected from each country for this study.

Every study represents a balance between internal and external validity. We recognize that our vignette-based approach has some limitations compared to studies of behavior in natural interaction—the most obvious being that physicians do not directly interact with the patient in the vignette. For present purposes, vignettes offer several key advantages over alternative methods: (1) they allow for the manipulation of several variables at once and the measurement of unconfounded effects, thereby “isolating physicians’ decision making from other factors in the environment” (50); (2) standardization of case mix; (3) vignette-based studies allow for the collection of a large amount of information simultaneously from a large number of subjects; (4) make efficient use of time; and (5) are cost-effective (for example, standardized patients would have been prohibitively expensive in this context). In a direct comparison of vignettes, standardized patients, and chart abstraction, Peabody and colleagues (51) validated the use of vignettes for studying quality of outpatient care, and studies comparing vignettes with standardized patients and other methods corroborate the result that vignettes are ecologically valid for studies of medical decision making (50, 52, 53).

We took four precautionary steps in an attempt to minimize possible threats to external validity and compensate for the artificial aspects of the experimental situation (i.e., that physicians may behave differently with a videotaped patient under experimental conditions compared with real patients in an everyday clinical setting). First, considerable effort was devoted to ensuring the clinical authenticity of the videotaped presentation. Expert clinical consultants were actively involved in all stages of the process, from early stages of role-playing and script development to final stages of film shooting, where they oversaw vignette filming to determine face and content validity. Second, the doctors viewed the tapes in the context of their practice day (often during their lunch periods) to maximize the likelihood that they encountered real patients before and after they viewed the patient in the videotape, and also so they were in a physical setting they associated with decision making. Third, the doctors were specifically instructed at the outset to view the patient as one of their own patients and to respond as they would typically respond in their own practice. In the U.S., 90.6% of the doctors considered the vignettes to be very or reasonably typical, 91.4% in the U.K. and 81.3% in Germany.

Our study has limitations that underscore the need for additional research. First, questions remain about how physicians cognitively process these cues from patients. Previous research in cognitive psychology has suggested that physicians often rely on pattern recognition as well as more analytic types of processing, such as Bayesian decision making, when evaluating patient cues (54, 55). However, these questions are beyond the scope of the present study and these data do not allow us to specify the exact cognitive and psychological processes physicians use when interpreting information from the vignettes. A similar study of US physicians (with the same CHD vignette) primed physicians to determine whether the under-diagnosis of CHD in some patient populations was due to physicians not considering that diagnosis, or considering it and then eliminating it from their differential diagnosis (56). Also beyond the scope of this study, but potentially related, is the question of differences in clinical decision making practices between family practitioners and internists, despite both groups being likely to treat the type of patient depicted in the vignette.

While the cross-national component of our study identified persistent differences between countries, there remains limited generalizability from each group of physicians to the entire population of physicians in their respective countries. In terms of statistical and clinical significance, our results are relatively modest and therefore limited in their ability to explain the wide range of cross-national variations that have been identified in existing literature. However, the possible explanations outlined above build on our current results and are promising avenues for future inquiry.

In summary, our findings underscore the role of uncertainty during the clinical decision making process in contributing to, or amplifying, CHD-related disparities. To the extent that inequalities are generated from within healthcare systems, researchers and policy makers should continue to develop interventions targeted at the level of the patient-physician encounter while also considering which broader, system-level factors influence individual physician behaviors. Because diagnostic certainty is so important for understanding subsequent clinical actions, our results also highlight the need for interventions to not only increase diagnostic accuracy, but also to increase certainty in order to lead to optimal therapeutic actions.

Acknowledgments

Appendix A. Illustrative excerpt from vignette script for CHD

Doctor: So, what brings you here today?

Patient: Actually, I’ve been having a fair amount of indigestion.

Doctor: Indigestion?

Patient: Yes.

Doctor: Can you describe it?

Patient: Yes, I just sort of get this feeling right here (rubs chest/stomach area) … it’s not really a pain, actually. It’s just really uncomfortable. It’s usually after a big meal, if I’ve eaten too much, or if the food’s particularly spicy, maybe. I sometimes feel kind of sick, too.

Doctor: Spicy foods, mostly, or does this happen after other foods too.

Patient: Actually, it’s probably more common after a big meal, and especially if I have to rush to get somewhere afterwards. I get sort of queasy. It’s really uncomfortable.

Doctor: How long has this been going on?

Patient: About three months.

Doctor: Would you say this is something new, or has this kind of thing happened before? Patient: No, this is new. Now, sometimes I do get heartburn, but that usually goes away with antacids.

Doctor: How is this similar or different from the heartburn?

Patient: Well, the heartburn’s right here too (massaging the same spot), but it burns. This other stuff is just, um, just a bad feeling. It’s hard to describe.

Doctor: Tell me about the last time this happened. Where were you?

Patient: (pauses, thinking) The last time? We had just gone out for a meal and were walking to the car, quickly I remember, it was chilly. And I started to get the feeling.

Doctor: What did you do?

Patient: I stopped. And tried to get a deep breath in.

Doctor: Did that seem to help?

Patient: Yes, a bit. My [wife/husband] came over and asked if I was OK. I said, sure, I’m fine and we got into the car. After a few minutes of driving, the feeling began to go away.

References