The first author performed qualitative content analysis of the interviews and observational field notes, and organized the observational field notes and 86 documents containing relevant information, as well as other documents and field notes. Follow-up interviews were conducted with department managers to validate this analysis. NVivo was used for codification (1100 codes were extracted and categorized). Coding and subsequent analysis were done in English to involve all the authors in strengthening trustworthiness and mitigating the bias risk
It is important to highlight that all steps are recursive. More than standard criteria, qualitative research depends on the researcher’s prudence, transparency, conscience, and attitudes. 24 Ensuring research validity can be laborious; our processual framework seems to help researchers with this relevant task.
As we pointed out, all the steps involved in processual validity matter, and the proposed integrated approach helps to appropriately link researchers, fields, and data to obtain high-quality results. Both cases illustrate the five processual validity steps, demonstrating the benefits of adopting good research procedures to promote reflexivity and increase the rigor of the study and the quality of the results.
Analyzing the research procedures adopted in Case 1, we first highlight participants’ representativeness, a detailed explanation of the data used, and the researcher’s prolonged exposure to the field. One of the co-authors works as a family doctor. Understanding the field helps the researcher appropriate the “field’s language,” which improves open coding procedures when analyzing the data. The researcher then used double coding techniques (provisional and final analysis grid) and several data sources, which facilitated triangulation and helped engage in reflection and achieve saturation. The adoption of dialectical hermeneutics to analyze the data made it possible to combine interpretive and critical elements, dive deeply into the phenomenon, and promote systematic reflexivity regarding how to make medical students’ internships more effective and more closely aligned with the most vulnerable population’s primary health care needs, thus improving the study’s social relevance. Analyzing the dimensions of validity in Case 1, the transparency of the hermeneutic data analysis process was weak. Missing details compromise reliability and transferability. Rigor in hermeneutic research is much more processual than in methodological research, once the researcher is at the center of the process. 14 In addition, discussion of the results in the ex-post step of the field research has not been detailed, weakening the confirmability of the findings.
In Case 2, the researcher also had long field exposure, but the methodological approach was different, as it was a positivist case study. The data were coded and analyzed using ex-ante analytical categories from two theoretical frameworks and qualitative content analysis. The authors of this article fulfilled all five processual validity steps of the proposed approach, showing high rigor, objectivity, and credibility to secure evidence. A significant difference between studies is related to the delimitation of the five steps identified in Case B and is a consequence of a more objectivist approach.
In a sense, processual validity attempts to create harmony and equilibrium between the “big picture” of the event’s context and the subtle details. In addition, the field and scientific peers can influence approximation, mainly at the last step (E) in the framework. Therefore, a sense of objectivity and intersubjectivity is a condition sine qua non for the quality of qualitative research.
This paper argues for the benefits of adopting a processual approach to validity in health care research. However, both case studies are exploratory, with inherent limitations concerning data and literature. Consequently, it was impossible to present more explanatory conclusions, and further research is required.
Regardless of the qualitative researcher’s epistemological or ontological position, the procedural approach to qualitative studies’ validity proved useful and applicable. According to the proposed framework, we aimed to improve the quality of qualitative research in health care, while helping to clarify some misunderstandings and providing conditions for continuous research development. 9 , 38 The qualitative researcher is not just at the center of the research process; they are also the manager of an equilibrium between the creativity of the arts and the rigidity of science. Hence, every step matters for continuous co-construction. 23 , 37 High-quality qualitative research can yield insightful contributions to providing secure evidence for health care decisions. 39 Thus, in a modified quote, “Validity is not a destination; it is a way of thinking, doing, and living research.”
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
Paulo Hayashi Jr https://orcid.org/0000-0003-2718-244
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Email citation, add to collections.
Your saved search, create a file for external citation management software, your rss feed.
Affiliation.
Much contemporary dialogue has centered on the difficulty of establishing validity criteria in qualitative research. Developing validity standards in qualitative research is challenging because of the necessity to incorporate rigor and subjectivity as well as creativity into the scientific process. This article explores the extant issues related to the science and art of qualitative research and proposes a synthesis of contemporary viewpoints. A distinction between primary and secondary validity criteria in qualitative research is made with credibility, authenticity, criticality, and integrity identified as primary validity criteria and explicitness, vividness, creativity, thoroughness, congruence, and sensitivity identified as secondary validity criteria.
PubMed Disclaimer
Full text sources.
NCBI Literature Resources
MeSH PMC Bookshelf Disclaimer
The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.
Post prepared and written by Joe Tise, PhD, Senior Education Researcher
In this series we have discovered the many ways in which evidence of validity can be produced and ways in which reliable data can be produced. To be sure, the bulk of this series was focused on quantitative research, but any mixed-methods or qualitative researcher will tell you that quantitative research only tells us one piece of the puzzle.
Qualitative research is needed to answer questions not suited for quantitative research, and validity and reliability need to be considered in qualitative research too. Qualitative research includes numerous methodological approaches, such as individual and focus group interviews, naturalistic observations, artifact analysis, and even open-ended survey questions. Unlike quantitative research–which utilizes forms, surveys, tests, institutional data, etc.–in qualitative research, the researcher often is the data collection mechanism and the analysis mechanism.
Researchers usually don’t run a statistical analysis on qualitative data; instead, a researcher typically analyzes the qualitative data, extracts meaning from it, and answers a research question from that meaning. Though this is similar to quantitative research, some of the analysis methods can be viewed as more subjective.
So, how can we know that results obtained from a qualitative analysis reflect some truth, and not the researcher’s personal biases, experiences, or lenses?
Reliability and validity are equally important to consider in qualitative research. Ways to enhance validity in qualitative research include:
Building reliability can include one or more of the following:
The most well-known measure of qualitative reliability in education research is inter-rater reliability and consensus coding. I want to make a distinction between two common measures of inter-rater reliability: percent agreement and Cohen’s Kappa.
Percent agreement refers to the percentage of coding instances in which two raters assign the same code to a common “piece” of data. Because this is a simple percentage, it’s more intuitive to understand. But it also does not account for chance–in any deductive coding framework (i.e., when all possible codes are already defined), there is a random chance that two coders will apply the same code without actually “seeing” the same thing in the data.
By contrast, Cohen’s Kappa is designed to parse out the influence of chance agreement, and for this reason Cohen’s Kappa will always be smaller than the percent agreement for a given dataset. Many qualitative data analysis software packages (e.g., NVivo) will calculate both percent agreement and Cohen’s Kappa.
In consensus coding, multiple raters code the same data, discuss the codes that may apply, and decide together how to code the data. With consensus coding, the need for inter-rater agreement/reliability metrics is circumvented, because by definition, you will always have 100% agreement/reliability. The major downside of consensus coding is, of course, the time and effort needed to engage it. With large sets of qualitative data, consensus coding may not be feasible.
For a deeper dive into these topics, there are many excellent textbooks that explore the nuances of qualitative validity and reliability. Below, you’ll find a selection of recommended resources, as well as others that provide detailed insights into strengthening qualitative research methods.
Corbin, J., & Strauss, A. (2015). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (4th ed.). Sage Publications. Creswell, J. W., & Báez, J. C. (2021). 30 Essential Skills for the Qualitative Researcher (2nd ed.). Sage Publications. Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches . Sage Publications. Saldaña, J. (2013). An introduction to codes and coding. In The coding manual for qualitative researchers (pp. 1–40). Sage Publications.
Comments are closed
Cookie | Duration | Description |
---|---|---|
cookielawinfo-checkbox-analytics | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics". |
cookielawinfo-checkbox-functional | 11 months | The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-others | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other. |
cookielawinfo-checkbox-performance | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance". |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
How do we assess and assure Validity in Qualitative Research ? This can be a bit of a tricky topic, as qualitative research involves humans understanding humans, a necessarily subjective practice from the get-go. Nevertheless, there are some questions the researcher can ask and some techniques he or she can employ to establish a reasonable level of validity.
Whether it is employed in business or the social sciences, it is often used to inform decisions that have important implications, thus assuring a high level of validity is essential. While the results should never be extrapolated over a larger population, (as they never come from a large enough sample to be statistically significant), validity can be established such that it can be used to inform meaningful decisions.
One measure of validity in qualitative research is to ask questions such as: “Does it make sense?” and “Can I trust it?” This may seem like a fuzzy measure of validity to someone disciplined in quantitative research, for example, but in a science that deals in themes and context, these questions are important.
Steps in Ensuring Validity
The first step in ensuring validity is choosing a well-trained and skilled moderator (or facilitator). A good moderator will check personal bias and expectations at the door. He or she is interested in learning as much candid information from the research participants as possible, and respectful neutrality is a must if the goal is valid qualitative research. For this reason, organizations often employ moderators from outside the group or organization to help ensure that the responses are genuine and not influenced by “what we want to hear.” For some academic applications, the moderator will disclose his or her perspectives and biases in the reporting of the data as a matter of full disclosure.
While a good moderator is key, a good sample group is also essential. Are the participants truly members of the segment from which they are recruited? Ethical recruiting is an important issue in qualitative research, as data collected from individuals who are not truly representative of their segment will not lead to valid results.
Another way to promote validity is to employ a strategy known as triangulation. To accomplish this, the research is done from multiple perspectives. This could take the form of using several moderators, different locations, multiple individuals analyzing the same data . . . essentially any technique that would inform the results from different angles. For some applications, for example, an organization may choose to run focus groups in parallel through two entirely different researchers and then compare the results.
Validity in qualitative research can also be checked by a technique known as respondent validation. This technique involves testing initial results with participants to see if they still ring true. Although the research has been interpreted and condensed, participants should still recognize the results as authentic and, at this stage, may even be able to refine the researcher’s understanding.
When the study permits, deep saturation into the research will also promote validity. If responses become more consistent across larger numbers of samples, the data becomes more reliable.
Another technique to establish validity is to actively seek alternative explanations to what appear to be research results. If the researcher is able to exclude other scenarios, he is or she is able to strengthen the validity of the findings. Related to this technique is asking questions in an inverse format.
While the techniques to establish validity in qualitative research may seem less concrete and defined than in some of the other scientific disciplines, strong research techniques will, indeed, assure an appropriate level of validity in qualitative research.
Additional Webpages Related to Validity in Qualitative Research
BMC Health Services Research volume 24 , Article number: 969 ( 2024 ) Cite this article
27 Accesses
Metrics details
The Stop Cancer PAIN Trial was a phase III pragmatic stepped wedge cluster randomised controlled trial which compared effectiveness of screening and guidelines with or without implementation strategies for improving pain in adults with cancer attending six Australian outpatient comprehensive cancer centres ( n = 688). A system for pain screening was introduced before observation of a ‘control’ phase. Implementation strategies introduced in the ‘intervention’ phase included: (1) audit of adherence to guideline recommendations, with feedback to clinical teams; (2) health professional education via an email-administered ‘spaced education’ module; and (3) a patient education booklet and self-management resource. Selection of strategies was informed by the Capability, Opportunity and Motivation Behaviour (COM-B) Model (Michie et al., 2011) and evidence for each strategy’s stand-alone effectiveness. A consultant physician at each centre supported the intervention as a ‘clinical champion’. However, fidelity to the intervention was limited, and the Trial did not demonstrate effectiveness. This paper reports a sub-study of the Trial which aimed to identify factors inhibiting or enabling fidelity to inform future guideline implementation initiatives.
The qualitative sub-study enabled in-depth exploration of factors from the perspectives of personnel at each centre. Clinical champions, clinicians and clinic receptionists were invited to participate in semi-structured interviews. Analysis used a framework method and a largely deductive approach based on the COM-B Model.
Twenty-four people participated, including 15 physicians, 8 nurses and 1 clinic receptionist. Coding against the COM-B Model identified ‘capability’ to be the most influential component, with ‘opportunity’ and ‘motivation’ playing largely subsidiary roles. Findings suggest that fidelity could have been improved by: considering the readiness for change of each clinical setting; better articulating the intervention’s value proposition; defining clinician roles and responsibilities, addressing perceptions that pain care falls beyond oncology clinicians’ scopes of practice; integrating the intervention within existing systems and processes; promoting patient-clinician partnerships; investing in clinical champions among senior nursing and junior medical personnel, supported by medical leaders; and planning for slow incremental change rather than rapid uptake.
Future guideline implementation interventions may require a ‘meta-implementation’ approach based on complex systems theory to successfully integrate multiple strategies.
Registry: Australian New Zealand Clinical Trials Registry; number: ACTRN 12615000064505; data: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspxid=367236&isReview=true .
Peer Review reports
Pain is a common and burdensome symptom in people with cancer [ 1 ]. Barriers to pain care occur at all ‘levels’, including the patient and family (e.g., misconceptions regarding opioids), clinician (e.g. lack of expertise), service (e.g. inadequate referral processes) and healthcare system (e.g. lack of coordination) [ 2 , 3 , 4 , 5 , 6 , 7 , 8 ]. A recent systematic review suggests that around 40% of cancer patients with pain may not receive adequate management [ 9 ]. Research has demonstrated that routine screening and implementation of evidence-based guidelines has potential to improve quality of cancer pain care and outcomes [ 10 , 11 , 12 , 13 , 14 ]. However, experience suggests that clinicians are unlikely to utilise screening results or follow guidelines unless these are supported by targeted strategies [ 15 , 16 ].
The Stop Cancer PAIN Trial (ACTRN 12615000064505) was a phase III pragmatic stepped wedge cluster randomised controlled trial conducted between 2014 and 2019 which compared the effectiveness of screening and guidelines with or without implementation strategies for improving pain in adults with cancer attending six outpatient comprehensive cancer centres in Australia ( n = 688) [ 17 , 18 ]. A pen/paper system to screen for pain using 0–10 numerical rating scales (NRS) for worst and average intensity over the past 24 h was introduced to each centre prior to observation of a ‘control’ phase, in which clinicians were also made aware of the Australian Cancer Pain Management in Adults guidelines [ 19 ]. At the beginning of the training phase, trial investigators presented at staff meetings on the importance of better managing pain and the rationale and evidence base for the intervention components. Implementation strategies (collectively termed the ‘intervention’) were then introduced in a ‘training’ phase and maintained during an ‘intervention’ phase as follows: (1) audit of adherence to key guideline recommendations [ 19 ] and feedback delivered to clinical teams in one or two cycles; (2) health professional education via a ‘Qstream’ email-administered ‘spaced education’ module [ 20 ]; and (3) a patient education booklet and self-management resource for completion together with a clinician that included goal setting, a pain diary and pain management plan [ 21 , 22 ]. Selection of these strategies was informed by the Capability, Opportunity and Motivation Behaviour (COM-B) Model of behaviour change [ 23 ], and evidence that each strategy had been separately effective for supporting guideline implementation for other health conditions. The intervention was supported at each centre by a consultant physician who agreed to be a ‘clinical champion’ [ 24 ].
As reported previously [ 18 ], the Stop Cancer PAIN Trial found no significant differences between the intervention and the control phases on the trial’s primary outcome - the proportion of patients with moderate-severe worst pain intensity who reported a 30% decrease at 1-week follow-up. Fidelity to the intervention was lower than anticipated and variable between centres: only 2/6 centres had two audit cycles rather than one; completion rates for the health professional spaced education varied from 12% to 74% between centres; and the proportion of patients reporting receipt of written information of any kind rose to an average of only 30% (20-44%) versus 22% (2-30%) in the control phase. Unexpectedly, secondary measures of mean, worst and average pain over a 4-week follow-up period improved by 0.5 standard deviation during control as well as intervention phases. However, the lack of a comparison group with no screening system made it difficult to conclude whether improvement in the control phase was due to effects from screening, a Hawthorne effect, or some other explanation.
The current paper reports a sub-study of the Stop Cancer PAIN Trial which aimed to identify factors influencing fidelity to the intervention that might warrant consideration by similar initiatives in the future.
The intervention, methods and results of the Stop Cancer PAIN trial have been described in previous open-access articles [ 17 , 18 ]. The sub-study used a qualitative approach with pragmatic orientation to enable in-depth exploration of factors influencing success from the perspectives of clinicians at each participating centre [ 25 ]. Clinician views canvassed at interview were considered the most efficient means of identifying barriers and enablers among complex contextual factors at each centre, including personnel’s knowledge, attitudes and beliefs towards pain care and the intervention.
The sub-study was approved by the Southwestern Sydney Local Health District Human Research Ethics Committee (HREC/14/LPOOL/479) as part of the overall trial. All participants gave written informed consent to participate.
Reporting adheres to the consolidated criteria for reporting qualitative research (COREQ) [ 26 ].
Participants were eligible if they were employed on a permanent basis either full- or part-time at a participating centre in a role that provided clinical care to cancer patients or patient-focused administrative support. The clinical champion at each centre was invited to participate by the research team. Other personnel were invited by means of email circulars and verbal invitations during meetings. Given the diverse range of roles at each centre, no limit was set on sample size to canvass as many perspectives as possible.
Data were collected by means of semi-structured interviews conducted by one of two researchers, a female pharmacist with experience of medical education for pain management (LR), and a male social scientist with a doctorate (TL). Both interviewers had prior experience in qualitative research and knew some participants through their project roles.
Participants were fully aware of the study purpose before consenting. Interviews were conducted face-to-face or by telephone, with the participant and interviewer being the only people present. Interviews began with open questions about ‘what worked’ and ‘didn’t work’ across the intervention before focusing on each implementation strategy in more detail and important contextual factors at their centre (see Table 1 for a topic guide, which was developed specifically for this study). Interviewers explicitly invited criticism, expressing a tone of open enquiry and neutrality throughout. Prompts were used as necessary to explore factors identified by participants in more detail. Factors identified at previous interviews were raised at subsequent ones for verification, inviting participants to disagree or agree as they felt appropriate. No requests were received to return transcripts to participants for comment. Interviews were audio-recorded and transcribed verbatim.
Analysis used the framework method [ 27 ] and a largely deductive approach based on the same theoretical framework used during intervention design - the COM-B Model [ 23 ]. Based on a systematic review, the COM-B Model posits that behaviour change requires three conditions, namely ‘capability’ (including both psychological and physical capacity), ‘opportunity’ (all the factors that lie outside the individual that make the behaviour possible or prompt it) and ‘motivation’ (including habitual processes, emotional responding, as well as analytical decision-making). Initial line-by-line coding categorized data against these conditions according to which best described relationships between factors and behaviours within and across implementation strategies and the levels of patient, clinician and centre. While the COM-B model originally focused directly on human behaviour, it became clear during coding that behaviour was substantially influenced by centre, specialty and disciplinary factors, so these were also considered appropriate foci for coding against COM conditions. To enhance credibility, the same data were coded in different ways where multiple interpretations seemed plausible until coding of further interviews identified consistencies to help with disambiguation. Charting of codes for data within and between centres enabled mapping between the relative contributions made by each condition, summarised as lessons learned for guiding similar initiatives in the future. Dependability was increased by ensuring coding was conducted by two members of the research team (NR, MG) who had no previous involvement in the project but were experienced in qualitative research. Review and discussion with two team members who were involved in the project throughout (TL and ML) was intended to balance ‘outsider’ and ‘insider’ perspectives to guard against bias from preconceived interpretations whilst also referencing contextual understanding. Both Excel 2019 (Microsoft) and NVivo V12 (QSR) software were used to help manage different stages of the analytic process.
Twenty-four people participated across the six centres, ranging from one to six participants. Fifteen were physicians (of whom six were clinical champions), eight were nurses, and one was a clinic receptionist. This response rate ranged from 2 to 27% of eligible personnel at each centre. See Table 2 for a more detailed summary of participant roles at each centre. Interviews were a median of 20 min long, with an inter-quartile range of 13 to 28 min.
Coding against the COM-B Model identified ‘capability’ to be the component having most influence over intervention success, with ‘opportunity’ and ‘motivation’ playing largely subsidiary roles.
Capabilities: Pertinent capabilities were reported to include: a pre-existing, centre-level culture of continuous improvement, communication pathways between senior management and other personnel, established roles and responsibilities for pain care among disciplines and specialties, systems and processes that could readily accommodate the intervention, and a culture of involving patients as partners in care. These capabilities influenced the degree to which personnel and patients had the opportunity and motivation to fully engage with the intervention.
Opportunity and motivation: These elements were most frequently discussed by participants in terms of ‘time’ that personnel could commit to pain care relative to other responsibilities. Clinical champions were perceived to play a critical role in supporting intervention success but were under-resourced at every centre and challenged by turnover in the role at two. In addition to more systemic drivers, individual personnel’s motivation was influenced by the degree to which they accepted the intervention’s value proposition at the outset and perceived this to be demonstrated over time.
Interactions between capability, opportunity and motivation are explored below in terms of their implications for similar future initiatives. Findings suggest that fidelity could have been improved by: considering the readiness for change of each clinical setting; better articulating the intervention’s value proposition; defining clinician roles and responsibilities, addressing perceptions that pain care falls beyond oncology clinicians’ scopes of practice; integrating the intervention within existing systems and processes; promoting patient-clinician partnerships; investing in clinical champions among senior nursing and junior medical personnel, supported by medical leaders; and planning for slow incremental change rather than rapid uptake.
The degree to which centres had a pre-existing culture of continuous improvement was considered important in providing a fertile context for the intervention. At Centre 5, there was a consensus that change of any kind was difficult to instigate, even according to the head of department: “… because it’s new - because we’re so entrenched in our ways ” (C5P04 [Centre 5, participant 04] medical oncologist, head of department and clinical champion). At another, the complex centre-level nature of the intervention was perceived to pose particular challenges compared to oncology drug trials with which they were more familiar: “ we haven’t been a principal site [in a trial of this kind] previously and I think that’s sort of opened up some gaps in knowledge for us and some opportunities for learning in the future … what kind of support we’d need to come with that trial to help it be a success in this culture ” (C3P02 palliative care physician and clinical champion).
Interviews highlighted the importance of articulating the intervention’s value proposition to every member of the workforce and maintaining engagement by demonstrating benefits over time. At Centre 5, some participants perceived that the intervention had been imposed by management rather than generated from clinical priorities: “…senior staff say [to researchers] ‘come to our clinics, but we expect everyone else to do the work’ ” (C5P05 radiation oncologist). This was compounded by a perceived lack of communication about the project, which limited personnels’ opportunity to take a more active role even when they were motivated to do so: “ I would have facilitated [the intervention] … but I didn’t know about it ” (C5P01 nurse practitioner). Eliciting and maintaining engagement was said to be additionally challenged at this centre by high staff turnover, especially among junior medical officers on rotation: “ it was very accepted by the junior medical staff [but] I think, unfortunately, when there’s a relatively high turnover of staff … ” (C5P07 radiation oncology trainee). At two other centres, turnover among personnel required a transition in the role of clinical champion, interrupting support for the intervention while the new incumbents familiarised themselves with the role.
Across centres, participants reported reservations among some of their colleagues regarding the project’s fundamental premises, including the assumption that pain care needed improving at their centre (“ they actually felt this trial was a little bit insulting for their clinical skills. There was a bit of eye rolling and ‘of course we do that already!’ ” (C3P02 palliative care physician and clinical champion)) or that pain warranted a specific focus rather than symptoms more generally: “ I find it more useful when more than one symptom is targeted ” (C5P06 palliative care physician).
More specific criticism was also levelled at each of the intervention strategies as follows.
In the case of screening, two participants questioned the validity of a 0–10 numerical rating scale (NRS) for different reasons: “ sometimes getting the numbers breaks the flow of the narrative” (C6P04 medical oncologist); “they [patients] would say, ‘no, I’m not in pain but I have a lot of discomfort when I swallow’ - it was in the wording ” C5P02 registered nurse). Even one of the clinical champions felt that screening was redundant where pain was very severe: “ if someone is clearly in a pain crisis, you don’t need to be asking … you kind of know what number - they might tell you it’s 15 [out of 10] ” (C6P02 palliative care physician and clinical champion). Perceptions of the value of screening were also influenced by the degree to which it led to demonstrable improvements in pain care, which was undermined by problems with establishing an efficient process at some centres: “ I think I’ve still probably got stray [pain screening] forms on my desk ” (C3P06 palliative care physician). A lack of understanding among personnel and patients about how screening might lead to better pain outcomes was said to result in “ fatigue ” (C5P03 clinical nurse consultant [clinical nurse consultant]; C1P01 palliative care physician and clinical champion), manifest as a downward spiral of effort in, and value from, screening.
The audit and feedback strategy attracted limited attention from personnel at most centres: “ I don’t think that the audit and feedback were terribly noticeable ” (C4P01 medical oncologist and clinical champion). At the centre where only the palliative care department participated, one participant perceived baseline audit results to be acceptable and therefore demotivating for change: “[ the audit results showed] we were doing a good job even ahead of time … it did sort of make you think – ‘well where do we go from here?’ ” (C6P04 pain medicine physician). At another centre, motivation among personnel to improve on less favourable audit findings was perceived to depend on whether they prioritised pain care to start with: “ people have come up to me and said, ‘Gee, we really did very badly didn’t we?’ … but they’re not necessarily the people who don’t treat pain well - that’s the problem ” (C1P01 palliative care physician and clinical champion).
Participants’ opinion on the value of the online spaced education depended on discipline and seniority, with nurses and junior medical officers reporting benefits “( it gave me a bit more confidence that I was on the right track” (C5P01 nurse practitioner)) but consultant physicians perceiving the knowledge level too “basic” (C6P04 pain medicine physician) or questioning advice from online spaced education that their responses were ‘wrong’: “…some of the multiple answers could have been equally valid” (C504 medical oncologist and clinical champion). Where consultants remained engaged, motivation was said to rely on cultivating “ competition” between colleagues (C602 palliative care physician and clinical champion). Inevitably, the voluntary nature of online spaced education also meant that only motivated personnel engaged to begin with.
All participants who had used the patient self-management resource perceived at least some value. However, its use was limited by barriers relating to role and process considered below.
Among the most commonly voiced barriers was a lack of clarity about which specialties and disciplines should be responsible for pain screening, patient education and management. This was usually described in terms of a ‘lack of time’ for pain care relative to other duties afforded greater priority within their scope of practice. Perspectives on roles and responsibilities are considered separately for each aspect of pain care as follows.
While most centres allocated the clinical task of pain screening to clinic receptionists, there was widespread reflection that this had been suboptimal. The only participating clinic receptionist felt that pain screening fell outside her area of responsibility: “but I’m an administrative person - I don’t have anything to do with pain management ” (C2P03 clinic receptionist). Clinician participants across disciplines similarly perceived that pain screening required clinical expertise to assist patients with reporting their pain and triage for urgent follow-up: “ you need somebody talking to the patients, rather than just handing the form, say ‘fill this in’ ” (C2P04 clinical nurse consultant). One centre that recognised this early on reallocated screening from an administrative to a nursing role, leading to substantial improvements in the completeness and quality of data: “ it made a big difference and certainly improved our ability to recognise people who had pain and allowed access for those people who were in severe pain to medications or at least an assessment … implementation through the clerical staff was not a long-term strategy ” (C1P01 palliative care physician and clinical champion).
There was little consensus on which disciplines should be responsible for supporting patients to use the self-management resource, with medical personnel deferring to nurses and vice-versa. Role allocation was challenged by the diverse components within the resource, with each perceived to fall within a different scope of practice: “ pain is something I always do as an assessment … [but] … I’m not managing the pain … I’ll review and make recommendations and talk about the pain diaries and discussing their diary with their palliative care doctor or their general practitioner. And I would encourage that process. [But] I wouldn’t be the one that’s setting the goals on their daily activities and stuff ” (C5P01 nurse practitioner). Some oncology nursing roles were perceived to focus on chemo- or radiotherapy protocols to the exclusion of supportive care unless symptoms arose from, or impeded, treatment. Meanwhile, oncologists tended to interpret their role as solely focused on prescribing rather than also encompassing patient education: “ junior doctors only [have] 15 minutes to take a history and everything. [They] could enter in meds [into the patient resource] if everything else is done by someone else … part of me knows it’s [patient resource] important, but the other part of me - I just - when will I have time in my clinical practice to do it? ” (C5P05 radiation oncologist).
Some oncologists viewed even pharmacological pain management as peripheral to their scope of practice when consultation time was short, prioritising cancer treatment instead. These participants viewed their role as limited to referring to palliative medicine or pain specialists, especially where pain was believed to have causes other than cancer: “ if the pain is a complex pain where the patient doesn’t have evidence of cancer, and it may be treatment-related, then in those scenarios we tend to divert to the chronic pain team ” (C5P07 radiation oncology advanced trainee). While participants from palliative care and pain medicine welcomed referrals for complex cases, they felt that oncologists sometimes referred for pain they could have easily managed themselves: “ what about some regular paracetamol? … These are things that you’d expect any junior doctors, never mind consultants [to have provided advice on] ” (C5P06 palliative care physician).
Participants from several centres expressed a view that the intervention’s complex nature had proven overwhelming for systems and processes at their centres. At two centres, integration was especially challenged by broader infrastructure shifts and process failures that limited receptiveness to further changes. Participants at several centres emphasised the process-driven nature of oncology services and the challenge of changing established processes: “ they have got a pro forma that they use for chemo-immunotherapy review, and pain is not part of it, and that perhaps needs more of an organisational nuance … why doesn’t pain feature as a clinical outcome as part of the chemotherapy, immunotherapy review?” (C6P01 clinical nurse consultant). Participants emphasised the need to integrate pain care into existing processes to help personnel understand what was expected of them: “…nursing staff were getting them [screening forms] in the patient’s files and going, ‘what am I supposed to do with this?’ ” (C2P04 clinical nurse consultant). Moreover, centres’ focus on cancer treatment meant that pain care struggled to gain traction even when a process could be instituted: “ unless pain is the presenting complaint and is at the forefront it goes into those, sorts of, you know, the ‘other details’ ” (C5P06 palliative care physician). For the palliative care centre, where pain care was already prioritised, there were doubts about how the proposed process improved on those already in place: “ I generally ask pretty detailed questions about pain anyway [so don’t need patients to be screened in the waiting room] ” (C6P04 pain medicine physician).
Suggestions for better integrating the intervention included “in-building” (C3P04 medical oncologist) responsibility for the strategies within new staff roles or introducing the strategies gradually by means of a “ multistep process” (C5P04 medical oncologist, head of department and clinical champion). Features of two strategies were singled out as having positive potential for supporting existing processes of care. The patient resource was said to “ facilitate communication between the oncology teams and the palliative care team ” (C5P05 radiation oncologist) and serve as a “ visual cue ” (C3P02 medical oncologist) to cover educational topics that “ they might have otherwise forgotten ” (C2P01 palliative care physician and clinical champion). Participants also found the spaced education email administration, spacing and repetition “ easy to manage ” (C2P01 palliative care physician and clinical champion) within their daily routines.
Several participants expressed surprise at the prevalence of moderate-severe pain in screening results, and acknowledged that this revealed under-reporting of pain in usual care. Under-reporting was perceived to stem partly from patient expectations that pain from cancer was “ normal ” (C4P03 nurse practitioner) and to be especially common in the context of certain generational or cultural attitudes towards pain and opioids (“ I certainly think there’s a cultural element but there’s also your elderly patients who you know have been through the war and they’re just used to coping with things and you just suck it up … it’s like a badge of honour to be able to say ‘I’m not one of these pill-takers ’” (C3P03 registered nurse [RN])) or when patients were concerned that reporting pain might reduce their fitness for anti-cancer treatment: “[ patients might think that] if I tell them honestly how crappy I am with other symptoms and pain and everything, then they might stop my chemo” (C3P02 palliative care physician). Several participants perceived that under-reporting was also due to patients taking an overly passive role in consultations: “[clinicians assume that] if the patient doesn’t bring it up, it’s not a problem for them and … then the patient [is] thinking ‘the doctor will only talk about important things that are important for me and I won’t mention it because obviously it’s not important’ ” (C3P02 palliative care physician and clinical champion).
The screening component of the intervention was considered to address under-reporting by “ normal[ising] ” pain care, thus encouraging disclosure. The patient resource was also considered helpful for building patient capability to partner with clinicians on pain management by “ encouraging self-efficacy ” (C2P01 palliative care physician and clinical champion) through the tools it provided and its positive message that “ you can get control of your pain ” (C3P02 palliative care physician and clinical champion). It was also perceived to help patients “ keep a record ” (C5P03 clinical nurse consultant) of breakthrough pain and analgesia to discuss in their consultation. However, some participants delineated patient groups who might be less able to use the resource, including those with lower educational levels who struggled to set goals and identify an ‘acceptable’ level of pain balanced against side-effects from pharmacological management. For these patients, it was suggested that too many resources could be overwhelming rather than supportive: “ it’s almost like, the more resources they have, the less resourced there are ” (C5P06 RN). At one centre with an especially diverse demographic, patients were said to require substantial support even to understand the purpose and process of pain screening: “ most [patients] look at you going ‘oh, do I have to do anything?’ … They don’t want to read the [instruction] page which is relatively simple ” (C2P03 clinic receptionist).
All participants perceived the role of clinical champion to be pivotal to the intervention’s success. Champions were perceived to have two major responsibilities: advocating for the intervention among colleagues to boost motivation and providing practical support to build capability.
To be effective advocates, champions were perceived to need support from senior management ( “[leadership of change] it’s got to happen from the top ” (C5P02 RN)) as well as established, cordial relationships with colleagues they could leverage to motivate engagement: “ it also relies on the champion’s personal relationship with the staff which you’re asking to perform these roles and trying to change their management ” (C1P01 palliative care physician and clinical champion). Where champions felt under-supported by management, they relied on moral support from the project team to sustain their advocacy work: “ being the champion, and sometimes being the nagging champion, it actually felt quite nice to have the back-up of other people ” (C1P01 palliative care physician and clinical champion). Both physicians and nurses perceived the champion role might better suit the scope of practice of a junior doctor or senior nurse rather than consultants, based on their willingness to engage and approachability: “ realistically, you’re probably always going to get more engagement with registrars compared to consultants, unless it’s their own trial ” (C5P07 radiation oncologist); “ just give it [the role] to the CNCs [clinical nurse consultants] because as a general rule they’re the best at everything and have the best relationships with the patient ” (C3P04 medical oncologist).
From a practical perspective, clinical champions were expected to provide human resources for establishing and supporting pain screening and patient education: “ you need a body ” (C2P04 clinical nurse consultant). Unfortunately, however, champions across centres reported having limited time protected for the role within their usual duties: “ there just wasn’t the manpower to do that here ” (C3P02 palliative care physician and clinical champion). One suggestion for boosting capacity was to narrow the focus to one clinic and delegate practical tasks to less senior delegates than required for advocacy to render the time commitment more cost-effective: “[ it] might have been better to focus on one clinic and have full-time … junior nurse ” (C5P05 radiation oncologist). This presented an opportunity to train more than one clinical champion to provide better coverage across shifts and safeguard against the risk of losing champions to staff turnover.
While the barriers above meant only modest practice changes could be achieved, champions at half the centres perceived incremental progress had been made through increasing awareness among personnel regarding pain care as a focus for improvement: “ I think just trying to make pain something that people think about was probably one of the better strategies ” (C1P01 palliative care physician and clinical champion); it’s more at the top of our minds to remember, to screen the pain at every visit ” (C2P01 palliative care physician and clinical champion); “ I think it has highlighted those issues for us and we now need to take this on ” (C5P04 medical oncologist, head of department and clinical champion). Both nursing and medical participants at Centre 5 emphasized the need to be persistent in striving for continuous improvement: “ I think to get practice change, even for well-motivated people, I think it just needs to be pushed … they’ve done similar things with hand washing for doctors and it’s finally getting through ” (C504 medical oncologist and clinical champion); “ it would take more than just one of these kind of programs to get people to change ” (C5P03 clinical nurse consultant). Encouragingly, participants at this and one other centre expected some clinicians to continue using the patient education booklet and resource after the project ended: “ I’d just love to continue using these booklets ” (C5P02 RN); “[the] patient-held resource has been useful and has been taken up by people and I think they will continue to use those ” (C6P02 palliative care physician and clinical champion).
This qualitative sub-study of a cluster randomized controlled trial identified centre-level capabilities to be the most influential factors impeding or facilitating guideline implementation strategies for improving pain care for outpatients with cancer. Findings suggest that future initiatives of this kind should: consider centre readiness for change; articulate and deliver on the intervention’s value proposition; define clinician roles and responsibilities; integrate the intervention within existing systems and processes; promote patient partnership; invest in the clinical champion role, drawing from senior nurses and junior doctors, with support from medical leaders and management; and design the initiative around slow incremental change rather than rapid uptake.
Our findings are largely consistent with those from an ethnographic study exploring factors influencing implementation of cancer pain guidelines in Korean hospital cancer units, which identified a ‘lack of receptivity for change’ to be a key barrier [ 28 ]. However, observations from the Korean study suggested that a lack of centre leadership and cultural norms regarding nursing hierarchy were the most important underlying factors, whereas our Australian sample focused more on constraints imposed by centre systems and processes and a lack of clarity regarding disciplinary roles. These factors were consistently emphasized regardless of participants’ discipline and seniority, including by one centre’s head of department. Consistent with these findings, a recent Australian qualitative sub-study of anxiety/depression guideline implementation in oncology centres found greater role flexibility to be a key factor underpinning organisational readiness for change [ 29 ]. This team also provided quantitative evidence consistent with our finding that centres’ readiness for change is associated with personnel’s perception of benefit from guideline implementation [ 30 ]. Future initiatives should work harder to persuade clinicians of the intervention’s rationale and evidence base prior to commencement, given that perceptions of coherence and effectiveness are key dimensions of acceptability required for clinicians to invest time and effort [ 31 ]. Since our Trial was conducted, evidence has emerged for an impact from cancer symptom screening on survival that could be used persuasively [ 32 ]. Furthermore, the spaced education module might be more acceptable if made adjustable to the knowledge levels of a broader range of clinicians.
Other studies on implementation of cancer pain guidelines [ 11 , 13 ] suggest that structured approaches to process change tend to be more successful than less prescriptive approaches of the kind taken in the Stop Cancer PAIN Trial. We provided centres with guideline implementation strategies but no clear guidance on how to integrate these within existing contexts - i.e. implementation of the implementation, or ‘meta-implementation’. It was wrongly assumed that clinical champions could support integration with centre processes based on their knowledge of local context, but this turned out to be unreasonable given champions’ limited time for the role and lack of training in change management. Like most research to date [ 33 , 34 ], our trial focused largely on the advocacy role played by clinical champions, neglecting more practical and time consuming aspects that our interviews identified to be just as important. We join others in calling for more research on the mechanisms by which clinical champions can optimally facilitate change and ways to maximize their efficacy through training and support [ 24 ]. This should include exploration of optimal models by which different aspects of the champion role might be shared between more than one person where no-one is available with all the necessary attributes, as well as ways to ensure sustainability after support from the project team is withdrawn.
Theory-based research suggests that adding complex interventions to complex healthcare systems creates dynamic interplay and feedback loops, making consequences hard to predict [ 35 ]. In the current trial, this was likely exacerbated by our attempt to combine multiple strategies targeting patient, clinician and centre levels. We chose each strategy based on evidence for its stand-alone efficacy, and combined strategies rather than used them singly with the intent of leveraging complementary mechanisms, as recommended by the COM-B Model and US Institute of Medicine [ 36 ]. However, findings from our interviews suggest that interactions between the strategies and local processes separated their spheres of influence, precluding intended synergies. The Stop Cancer PAIN Trial is not alone in having over-estimated the value of combining guideline implementation strategies; a recent systematic review found that 8 other multi-component interventions similarly demonstrated limited effects on guideline adherence and patient outcomes [ 37 ]. Collectively, these findings suggest that future attempts at combining strategies should consider complex systems theory as well as behaviour change frameworks at each of a number of stages [ 38 ]. Alternatively, a more manageable approach for most cancer centres might be to focus on just one component at a time, periodically reviewing progress against SMART goals and, depending on results, supplementing with additional components using plan-do-study cycles [ 39 ].
Given the challenges with integrating screening into centre processes, it seems unlikely that improvements in pain scores during the control phase reported in our primary results article were due to the spontaneous use of screening data in consultations [ 18 ]. Indeed, while routine use of patient-reported outcome measures (PROMs) in oncology has been researched for more than a quarter-century [ 40 ], benefits to patient outcomes have only recently been demonstrated in the context of electronically-administered PROMs (ePROMs) that enable remote self-reporting, real-time feedback to clinicians, and clinician-patient telecommunication [ 12 ]. Further research is needed on how best to support clinician engagement with ePROMs, including training on how to use results in partnership with patients to assist shared decision-making and self-management [ 41 ].
A worrying finding from the current study was that some or all aspects of pain care were perceived to fall between the scopes of practice for oncology clinicians from each discipline. Clinical practice guidelines emphasize the need for pain care to be inter-disciplinary in recognition of the need for comprehensive assessment, non-pharmacological as well as pharmacological management, and patient education and support for self-management [ 42 ]. While the patient self-management resource included in the intervention was perceived to support communication between clinicians and patients, its potential for assisting coordination of care between disciplines was limited where roles and responsibilities were not previously established. Our findings and other research suggest that future initiatives may benefit from ‘process mapping’ with clinicians to identify where clinical workflow and roles might be reconfigured to incorporate the various aspects of pain care in the most efficient ways that do not substantially add to workload [ 41 ].
Patient education has been proven to improve pain outcomes by clinical trials [ 43 , 44 ], and we have argued previously that supporting pain self-management should be core business for all clinicians working in cancer care [ 45 ]. The ‘coaching’ approach needed to empower patients to recognize themselves as ‘experts’ on their pain and equal partners with clinicians in its management is iterative rather than a single event, and is ideally built on established and ongoing therapeutic relationships of trust with a particular team member. However, findings from patient education research more generally suggest that patient education and behaviour change is also optimally supported when key messages are reinforced by differing disciplinary perspectives [ 46 ]. Results from the current study suggest that these principles of pain care need more formal recognition within the scope of practice of oncology clinicians to ensure they are afforded sufficient time alongside anti-cancer treatment and related supportive care. Findings also indicate that clinicians may require training in the person-centred, partnership-oriented aspects of pain care beyond the educational approach used in the Stop Cancer PAIN Trial and other research [ 47 ]. Such training should be repeated regularly to ensure it reaches the majority of personnel at cancer centres, allowing for turnover.
The current study had several limitations. Transferability even within Australia is limited by a focus on metropolitan services in only three out of eight jurisdictions. Data relied on clinician perspectives, and the response rate was less than one quarter of personnel at each centre, with the disciplines and specialties of participants being unrepresentative of centre workforces. Over-sampling of medical compared to nursing personnel likely reflects the fact that all clinical champions were medical consultants, while the predominance of palliative care physicians among medical participants presumably arises from the central focus this specialty has on pain care. Notably, our sample included no perspectives from allied health disciplines, despite the important roles these can play in non-pharmacological pain management. Confirmability was threatened by the potential for cognitive bias among researchers towards a favourable view of the intervention given their long-standing investment as members of the project team. We attempted to offset this by explicitly inviting criticism of the intervention from participants, and having the initial analysis conducted by researchers with no prior involvement in the project. A final limitation concerns reliance on the COM-B Model for analysis rather than an alternative framework or more inductive approach. While the COM-B has been widely used to explore barriers and facilitators across a wide range of healthcare interventions, we applied the model in a somewhat novel way to systems and processes as well as individuals’ behaviour after finding that participants perceived their agency to be majorly constrained by these. An implementation framework such as the integrated-Promoting Action on Research Implementation in Health Service (i-PARIHS) framework (iPARIHS) [ 48 ] or Consolidated Framework for Implementation Research (CFIR) [ 49 ] would have conceived of factors and their relationships in alternative ways that might have proven equally informative [ 50 ].
This qualitative sub-study elucidated important factors influencing the success of guideline implementation strategies at six cancer centres in the Stop Cancer PAIN Trial. Findings underscore the value that a qualitative approach offers for understanding the role of context when evaluating complex interventions [ 51 ]. Ultimately, the Stop Cancer PAIN Trial may have been overly ambitious in the scale of its intervention, especially given limited resources available at each centre. Further research is needed to understand how multi-component guideline implementation strategies can be optimally introduced within the context of local roles, systems and processes.
The qualitative interview datasets generated and analysed during the current study are not publicly available due to the conditions of ethical approval which acknowledge the risk of participant re-identification.
Van Den Beuken-Van MH, Hochstenbach LMJ, Joosten EAJ, Tjan-Heijnen VCG, Janssen DJA. Update on prevalence of pain in patients with cancer: systematic review and meta-analysis. J Pain Symptom Manag. 2016;51(6):1070–90.
Article Google Scholar
Luckett T, Davidson PM, Green A, Boyle F, Stubbs J, Lovell M. Assessment and management of adult cancer pain: a systematic review and synthesis of recent qualitative studies aimed at developing insights for managing barriers and optimizing facilitators within a comprehensive framework of patient care. J Pain Symptom Manag. 2013;46(2):229–53.
Oldenmenger WH, Sillevis Smitt PAE, van Dooren S, Stoter G, van der Rijt CCD. A systematic review on barriers hindering adequate cancer pain management and interventions to reduce them: a critical appraisal. Eur J Cancer. 2009;45(8):1370–80.
Article PubMed Google Scholar
Jacobsen R, Sjogren P, Moldrup C, Christrup L. Physician-related barriers to cancer pain management with opioid analgesics: a systematic review. J Opioid Manag. 2007;3(4):207–14.
Jacobsen R, Moldrup C, Christrup L, Sjogren P. Patient-related barriers to cancer pain management: a systematic exploratory review. Scand J Caring Sci. 2009;23(1):190–208.
Jacobsen R, Liubarskiene Z, Moldrup C, Christrup L, Sjogren P, Samsanaviciene J. Barriers to cancer pain management: a review of empirical research. Medicina (Kaunas). 2009;45:427–33.
Fazeny B, Muhm M, Hauser I, Wenzel C, Mares P, Berzlanovich A, et al. Barriers in cancer pain management. Wiener Klinische Wochenschrift. 2000;112(22):978–81.
CAS PubMed Google Scholar
Makhlouf SM, Pini S, Ahmed S, Bennett MI. Managing pain in people with cancer—a systematic review of the attitudes and knowledge of professionals, patients, caregivers and Public. J Cancer Educ. 2020;35(2):214–40.
Roberto A, Greco MT, Uggeri S, Cavuto S, Deandrea S, Corli O, et al. Living systematic review to assess the analgesic undertreatment in cancer patients. Pain Pract. 2022;22(4):487–96.
Brink-Huis A, van Achterberg T, Schoonhoven L, Brink-Huis A, van Achterberg T, Schoonhoven L. Pain management: a review of organisation models with integrated processes for the management of pain in adult cancer patients. J Clin Nurs. 2008;17(15):1986–2000.
Du Pen SL, Du Pen AR, Polissar N, Hansberry J, Kraybill BM, Stillman M, et al. Implementing guidelines for cancer pain management: results of a randomized controlled clinical trial. J Clin Oncol. 1999;17(1):361–70.
Article CAS PubMed Google Scholar
Govindaraj R, Agar M, Currow D, Luckett T. Assessing patient-reported outcomes in routine cancer clinical care using electronic administration and telehealth technologies: Realist synthesis of potential mechanisms for improving health outcomes. J Med Internet Res. 2023;25:e48483.
Article PubMed PubMed Central Google Scholar
Koesel N, Tocchi C, Burke L, Yap T, Harrison A. Symptom distress: implementation of palliative care guidelines to improve pain, fatigue, and anxiety in patients with advanced cancer. Clin J Oncol Nurs. 2019;23(2):149–55.
PubMed Google Scholar
Dulko D, Hertz E, Julien J, Beck S, Mooney K. Implementation of cancer pain guidelines by acute care nurse practitioners using an audit and feedback strategy. J Am Acad Nurse Pract. 2010;22(1):45–55.
Samuelly-Leichtag G, Adler T, Eisenberg E. Something must be wrong with the implementation of cancer-pain treatment guidelines. A lesson from referrals to a pain clinic. Rambam Maimonides Med J. 2019;10(3):18.
Lopez Lopez R, Camps Herrero C, Khosravi-Shahi P, Guillem Porta V, Carrato Mena A, Garcia-Foncillas J, et al. Oncologist’s knowledge and implementation of guidelines for breakthrough cancer pain in Spain: CONOCE study. Clin Transl Oncol. 2018;20(5):613–8.
Luckett T, Phillips J, Agar M, Lam L, Davidson PM, McCaffrey N, et al. Protocol for a phase III pragmatic stepped wedge cluster randomised controlled trial comparing the effectiveness and cost-effectiveness of screening and guidelines with, versus without, implementation strategies for improving pain in adults with cancer attending outpatient oncology and palliative care services: the Stop Cancer PAIN trial. BMC Health Serv Res. 2018;18(1):1–13.
Lovell MR, Phillips JL, Luckett T, Lam L, Boyle FM, Davidson PM, et al. Effect of cancer pain guideline implementation on pain outcomes among adult outpatients with cancer-related pain: a stepped wedge cluster randomized trial. JAMA Netw Open. 2022;5(2):1–12.
Lovell M, Luckett T, Boyle F, Stubbs J, Phillips J, Davidson PM, et al. Adaptation of international guidelines on assessment and management of cancer pain for the Australian context. Asia-Pac J Clin Oncol. 2015;11(2):170–7.
Phillips JL, Heneka N, Bhattarai P, Fraser C, Shaw T. Effectiveness of the spaced education pedagogy for clinicians’ continuing professional development: a systematic review. Med Educ. 2019;53(9):886–902.
Cancer Council Australia. Overcoming cancer pain: A guide for people with cancer, their families and friends 2013 [Available from: http://www.cancercouncil.com.au/wp-content/uploads/2014/01/Can487-Overcoming-Pain-NSW-Lores2.pdf .
Luckett T, Davidson PM, Green A, Marie N, Birch MR, Stubbs J, et al. Development of a cancer pain self-management resource to address patient, provider, and health system barriers to care. Palliat Support Care. 2019;17(4):472–8.
Michie S, Atkins L, West R. The behaviour change wheel: a guide to designing interventions. London: Silverback Publishing; 2014.
Google Scholar
Santos WJ, Graham ID, Lalonde M, Demery Varin M, Squires JE. The effectiveness of champions in implementing innovations in health care: a systematic review. Implement Sci Commun. 2022;3(1):80.
Ramanadhan S, Revette AC, Lee RM, Aveling EL. Pragmatic approaches to analyzing qualitative data for implementation science: an introduction. Implement Sci Commun. 2021;2(1):70.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.
Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):117.
Kim M, Jeong S, McMillan M, Higgins I. Translation of policy and guidelines into practice: lessons learned from implementation of the cancer pain management guideline. J Hosp Manage Health Policy. 2020;4. https://doi.org/10.21037/jhmhp.2020.02.01 .
Geerligs L, Shepherd HL, Butow P, Shaw J, Masya L, Cuddy J, et al. What factors influence organisational readiness for change? Implementation of the Australian clinical pathway for the screening, assessment and management of anxiety and depression in adult cancer patients (ADAPT CP). Support Care Cancer. 2021;29(6):3235–44.
Faris MM, Shepherd HL, Butow PN, Kelly P, He S, Rankin N, et al. Staff- and service-level factors associated with organisational readiness to implement a clinical pathway for the identification, assessment, and management of anxiety and depression in adults with cancer. BMC Health Serv Res. 2023;23(1):866.
Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17(1):88.
Caminiti C, Maglietta G, Diodati F, Puntoni M, Marcomini B, Lazzarelli S, et al. The effects of patient-reported outcome screening on the survival of people with cancer: a systematic review and meta-analysis. Cancers. 2022;14(21):5470.
Morena AL, Gaias LM, Larkin C. Understanding the role of clinical champions and their impact on clinician behavior change: the need for causal pathway mechanisms. Front Health Serv. 2022;2:896885.
Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: an integrative review of champions in healthcare-related implementation. SAGE Open Med. 2018;6:2050312118773261.
Braithwaite J, Churruca K, Long JC, Ellis LA, Herkes J. When complexity science meets implementation science: a theoretical and empirical analysis of systems change. BMC Med. 2018;16:1–14.
Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust. In: Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, editors. Washington, D.C.: National Academies Press; 2011.
Bora A-M, Piechotta V, Kreuzberger N, Monsef I, Wender A, Follmann M, et al. The effectiveness of clinical guideline implementation strategies in oncology—a systematic review. BMC Health Serv Res. 2023;23(1):347.
McGill E, Er V, Penney T, Egan M, White M, Meier P, et al. Evaluation of public health interventions from a complex systems perspective: a research methods review. Soc Sci Med. 2021;272: 113697.
Institute for Healthcare Improvement. Quality improvement essentials toolkit. 2017.
Trowbridge RDW, Jay SJ, et al. Determining the effectiveness of a clinical-practice intervention in improving the control of pain in outpatient with cancer. Acad Med. 1997;72:798–800.
Howell D, Rosberger Z, Mayer C, Faria R, Hamel M, Snider A, et al. Personalized symptom management: a quality improvement collaborative for implementation of patient reported outcomes (PROs) in ‘real-world’ oncology multisite practices. J Patient Rep Outcomes. 2020;4(1):47.
Australian Adult Cancer Pain Management Guideline Working Party. Autralian clinical pathway for screening, assessment and management of cancer pain in adults Sydney, Australia: Cancer Council Australia. 2013. http://wiki.cancer.org.au/australia/Guidelines:Cancer_pain_management/Flowchart_overview .
Marie N, Luckett T, Davidson P, Lovell M, Lal S. Optimal patient education for cancer pain: a systematic review and theory-based meta-analysis. Support Care Cancer. 2013;21:3529–37.
Abernethy AP, Currow DC, Shelby-James T, Rowett D, May F, Samsa GP, et al. Delivery strategies to optimize resource utilization and performance status for patients with advanced life-limiting illness: results from the palliative care trial [ISRCTN 81117481]. J Pain Symptom Manage. 2013;45(3):488–505.
Lovell MR, Luckett T, Boyle FM, Phillips J, Agar M, Davidson PM. Patient education, coaching, and self-management for cancer pain. J Clin Oncol. 2014;32(16):1712–20.
Luckett T, Roberts MM, Smith T, Swami V, Cho JG, Wheatley JR. Patient perspectives on how to optimise benefits from a breathlessness service for people with COPD. NPJ Prim Care Respir Med. 2020;30(1). https://doi.org/10.1038/s41533-020-0172-4 .
Phillips JL, Heneka N, Lovell M, Lam L, Davidson P, Boyle F, et al. A phase III wait-listed randomised controlled trial of novel targeted inter-professional clinical education intervention to improve cancer patients’ reported pain outcomes (the Cancer Pain Assessment (CPAS) Trial): study protocol. Trials. 2019;20:1–12.
Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. 2016;11(1):33.
Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. 2022;17(1):1–16.
Geerligs L, Rankin NM, Shepherd HL, Butow P. Hospital-based interventions: a systematic review of staff-reported barriers and facilitators to implementation processes. Implement Sci. 2018;13(1):36.
Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of medical research council guidance. BMJ. 2021;374:n2061.
Download references
The authors would like to dedicate this article to the memory of Sally Fielding, who worked as a valued member of the project team throughout the Stop Cancer PAIN Trial. We would also like to acknowledge the contributions of project manager A/Prof Annmarie Hosie, data manager Dr Seong Cheah, and research assistant Layla Edwards.
This research was supported by a grant from the National Breast Cancer Foundation.
Authors and affiliations.
IMPACCT Centre—Improving Palliative, Aged and Chronic Care through Clinical Research and Translation, Faculty of Health, University of Technology Sydney (UTS), Building 10, 235 Jones St, Ultimo, Sydney, NSW, 2007, Australia
Tim Luckett, Meera Agar & Maja Garcia
School of Nursing and Centre for Healthcare Transformation, Queensland University of Technology (QUT), Brisbane, QLD, Australia
Jane Phillips
South West Sydney School of Clinical Medicine, University of New South Wales (UNSW), Sydney, NSW, Australia
The Limbic, Sydney, Australia
Linda Richards
Palliative Care Department, Greenwich Hospital, HammondCare, Sydney, NSW, Australia
Najwa Reynolds & Melanie Lovell
University of Wollongong, Wollongong, NSW, Australia
Patricia Davidson & David Currow
Charles Perkins Centre, School of Medical Sciences, The University of Sydney, Sydney, NSW, Australia
Patricia Ritchie Centre for Cancer Care and Research, The University of Sydney, Sydney, NSW, Australia
Frances Boyle
Northern Medical School, The University of Sydney, Sydney, NSW, Australia
Frances Boyle & Melanie Lovell
Macau University of Science and Technology, Macau, China
Lawrence Lam
Deakin Health Economics, Institute for Health Transformation, School of Health and Social Development, Deakin University, Melbourne, Australia
Nikki McCaffrey
You can also search for this author in PubMed Google Scholar
TL, JP, MA, PMD, TS, DCC, FB, LL, NM and ML contributed to the concept and design of this research. TL, LR, MR, MG and ML contributed to the acquisition, analysis or interpretation of the data. TL and ML contributed to drafting of the manuscript. All authors contributed to revisions of the manuscript and approved the final version.
Correspondence to Tim Luckett .
Ethics approval and consent to participate.
This research was approved by the Southwestern Sydney Local Health District Human Research Ethics Committee (HREC/14/LPOOL/479). All participants gave written informed consent to participate.
Not applicable.
The authors declare no competing interests.
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .
Reprints and permissions
Cite this article.
Luckett, T., Phillips, J., Agar, M. et al. Factors influencing fidelity to guideline implementation strategies for improving pain care at cancer centres: a qualitative sub-study of the Stop Cancer PAIN Trial. BMC Health Serv Res 24 , 969 (2024). https://doi.org/10.1186/s12913-024-11243-1
Download citation
Received : 10 April 2024
Accepted : 25 June 2024
Published : 22 August 2024
DOI : https://doi.org/10.1186/s12913-024-11243-1
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1472-6963
IMAGES
COMMENTS
In assessing validity of qualitative research, the challenge can start from the ontology and epistemology of the issue being studied, e.g. the concept of "individual" is seen differently between humanistic and positive psychologists due to differing philosophical perspectives: Where humanistic psychologists believe "individual" is a ...
However, the increased importance given to qualitative information in the evidence-based paradigm in health care and social policy requires a more precise conceptualization of validity criteria that goes beyond just academic reflection. After all, one can argue that policy verdicts that are based on qualitative information must be legitimized by valid research, just as quantitative effect ...
Although the tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research, there are ongoing debates about whether terms such as validity, reliability and generalisability are appropriate to evaluate qualitative research.2-4 In the broadest context these terms are applicable, with validity referring to the integrity and ...
nts the concept of rigor in qualitative research using a phenomenological study as an exemplar to further illustrate the process. Elaborating on epistemological and theoretical conceptualizations by Lincoln and Guba, strategies congruent with qualitative perspective for ensuring validity to establish the credibility of the study are described. A synthesis of the historical development of ...
Validity and reliability or trustworthiness are fundamental issues in scientific research whether qualitative, quantitative, or mixed research. It is a necessity for researchers to describe which ...
Rather than prescribing what reliability and/or validity should look like, researchers should attend to the overall trustworthiness of qualitative research by more directly addressing issues associated with reliability and/or validity, as aligned with larger issues of ontological, epistemological, and paradigmatic affiliation.
Fundamental Criteria: General Research Quality. Various researchers have put forward criteria for evaluating qualitative research, which have been summarized in Table 3.Also, the criteria outlined in Table 4 effectively deliver the various approaches to evaluate and assess the quality of qualitative work. The entries in Table 4 are based on Tracy's "Eight big‐tent criteria for excellent ...
Following a chapter on objectivity, the authors discuss the role of reliability and validity and the problems that arise when these issues are neglected. They present a paradigm for the qualitative research process that makes it possible to pursue validity without neglecting reliability.
Concerns with the issues of validity in qualitative research have dramatically increased. Traditionally, validity in qualitative research involved determining the degree to which researchers' claims about knowledge corresponded to the reality (or research participants' construction of reality) being studied. The authors note that recent ...
Abstract. Much contemporary dialogue has centered on the difficulty of establishing validity criteria in qualitative research. Developing validity standards in qualitative research is challenging because of the necessity to incorporate rigor and subjectivity as well as creativity into the scientific process.
Issues of trustworthiness in qualitative leisure research, often demonstrated through particular techniques of reliability and/or validity, is often either nonexistent, unsubstantial, or unexplained.
K E Y WO R D S: e research, research validityIntroductionIssues related to validity in qualitative research have been addressed for more t. an half a century (Atkinson et al., 2003). Recently, concerns about valid. ty in qualitative research have increased. This is true internationally (see e.g. Bradbury and Reason, 2001; Seale, 1999), and ...
This article presents the concept of rigor in qualitative research using a phenomenological study as an exemplar to further illustrate the process. Elaborating on epistemological and theoretical conceptualizations by Lincoln and Guba, strategies congruent with qualitative perspective for ensuring validity to establish the credibility of the ...
Processual Validity. In qualitative research, validity does not present a unitary concept. 12,28 Therefore, a processual approach may offer qualitative research more flexibility to adapt projects to different situations, contexts, epistemological paradigms, and personal styles in conducting research. In addition, qualitative methods should not use very strict or "one best way" strategies ...
2. Rolfe G. Validity, trustworthiness and rigour: quality and the idea of qualitative research. J Adv Nurs 2006;53:304-10. 3. Sandelowski M. Rigor or rigor mortis: the problem of rigor in qualitative research revisited. Adv Nurs Sci 1993;16:1-8. 4. Long T, Johnson M. Rigour, reliability and validity in qualitative research. Clin Eff Nurs ...
Describes 5 types of validity commonly used in qualitative research: descriptive validity, interpretive validity, theoretical validity, generalizability, and evaluative validity. Such a typology provides a useful checklist of the kinds of threats to validity that researchers need to consider. While the terms involved in descriptive validity can be either etic or emic, interpretive validity ...
Developing validity standards in qualitative research is challenging because of the necessity to incorporate rigor and subjectivity as well as creativity into the scientific process. This article explores the extant issues related to the science and art of qualitative research and proposes a synthesis of contemporary viewpoints.
Qualitative validity is also too important and rich to be based on only a few fixed measures at the end of the study, such as Cronbach's alpha among the quantitative research methods. 21,22 Every stage of qualitative research is significant, and a processual approach to ensure validity can influence the quality of all stages. 13,14 The ...
Validity. Qualitative research is based on subjective, interpretive and contextual data; whereas quantitative. research attempts to control and/or exclude those elements (Auerbach & Silverstein ...
Reliability in qualitative research refers to the stability of responses to multiple coders of data sets. It can be enhanced by detailed field notes by using recording devices and by transcribing the digital files. However, validity in qualitative research might have different terms than in quantitative research. Lincoln and Guba (1985) used "trustworthiness" of ...
Reliability and validity are equally important to consider in qualitative research. Ways to enhance validity in qualitative research include: Building reliability can include one or more of the following: The most well-known measure of qualitative reliability in education research is inter-rater reliability and consensus coding.
In this article, we argue that reliability and validity remain appropriate concepts for attaining rigor in qualitative research. We argue that qualitative researchers should reclaim responsibility for reliability and validity by implementing verification strategies integral and self-correcting during the conduct of inquiry itself.
Validity in qualitative research can also be checked by a technique known as respondent validation. This technique involves testing initial results with participants to see if they still ring true. Although the research has been interpreted and condensed, participants should still recognize the results as authentic and, at this stage, may even ...
Design. The intervention, methods and results of the Stop Cancer PAIN trial have been described in previous open-access articles [17, 18].The sub-study used a qualitative approach with pragmatic orientation to enable in-depth exploration of factors influencing success from the perspectives of clinicians at each participating centre [].Clinician views canvassed at interview were considered the ...