health needs assessment research

• Open access
• Published: 21 December 2015

Tools and instruments for needs assessment, monitoring and evaluation of health research capacity development activities at the individual and organizational level: a systematic review

• Johanna Huber 1 ,
• Sushil Nepal 1 ,
• Daniel Bauer 1 ,
• Insa Wessels 2 ,
• Martin R Fischer 1 &
• Claudia Kiessling 1 , 3  

Health Research Policy and Systems volume  13 , Article number:  80 ( 2015 ) Cite this article

28k Accesses

24 Citations

1 Altmetric

Metrics details

In the past decades, various frameworks, methods, indicators, and tools have been developed to assess the needs as well as to monitor and evaluate (needs assessment, monitoring and evaluation; “NaME”) health research capacity development (HRCD) activities. This systematic review gives an overview on NaME activities at the individual and organizational level in the past 10 years with a specific focus on methods, tools and instruments. Insight from this review might support researchers and stakeholders in systemizing future efforts in the HRCD field.

A systematic literature search was conducted in PubMed and Google Scholar. Additionally, the personal bibliographies of the authors were scanned. Two researchers independently reviewed the identified abstracts for inclusion according to previously defined eligibility criteria. The included articles were analysed with a focus on both different HRCD activities as well as NaME efforts.

Initially, the search revealed 700 records in PubMed, two additional records in Google Scholar, and 10 abstracts from the personal bibliographies of the authors. Finally, 42 studies were included and analysed in depth. Findings show that the NaME efforts in the field of HRCD are as complex and manifold as the concept of HRCD itself. NaME is predominately focused on outcome evaluation and mainly refers to the individual and team levels.

A substantial need for a coherent and transparent taxonomy of HRCD activities to maximize the benefits of future studies in the field was identified. A coherent overview of the tools used to monitor and evaluate HRCD activities is provided to inform further research in the field.

Peer Review reports

The capacity to cope with new and ill-structured situations is a crucial ability in today’s world. Developing this ability, by shaping empowered citizens, challenges individuals as well as organisations and societies. This process of empowerment is usually referred to as capacity development (CD) [ 1 ]. While this term has been commonly used for years in the field of foreign aid, other societal and political domains (e.g. social work, education and health systems) are increasingly adopting the concept of CD when developing new or existing competencies, structures, and strategies for building resilient individuals and organizations [ 2 ]. Also in the field of health research, an increasing number of activities to strengthen health research competencies and to support organizations can be observed – as demanded by the three United Nations Millennium Development Goals addressing health related issues [ 3 – 6 ]. Several frameworks are already in use that support a structured approach to health research capacity development (HRCD) and address competencies that are specific to health research [ 7 – 9 ]. These frameworks usually incorporate the individual or team, organization or institution, and society levels [ 8 , 10 , 11 ]. One conclusion that can be drawn from the available evidence is that, in such a structured approach to HRCD efforts, meaningful data collection is crucial. First, data collection incorporates the HRCD needs assessment and second, the monitoring and evaluation (NaME) of activities and programs once implemented. Therefore, HRCD activities should address the needs as assessed. Monitoring and evaluation of these activities should reflect the desired outcomes as defined beforehand [ 12 – 15 ]. Bates et al. [ 16 ] indicate how data collection tools and instruments are usually developed for a certain purpose in a certain context. The context specificity of tools and instruments has to be considered and the appropriateness of these must be determined when selecting instruments for any needs assessment for a new project. This article offers a systematic review of tools and instruments for the NaME of HRCD activities at the individual or team and the organizational levels to aid HRCD initiatives in selecting appropriate tools and instruments for data collection within their respective context. For this purpose, a range of studies published between January 1, 2003, and June 30, 2013, were chosen and analysed based on different context parameters such as the level of the CD and the nature of the HRCD activities.

We followed the PRISMA checklist for reporting systematic reviews and meta-analyses [ 17 ]. Inclusion and analysis criteria were defined in advance and documented in a protocol (Tables  1 and 2 ).

Information sources and search strategy

We conducted the systematic literature search in July 2013. The search was done in both the literature database PubMed and the search engine Google Scholar. We applied the three search terms “capacity building” AND “research ” , “capacity development” AND “research”, and “capacity strengthening” AND “research”. We checked the first 200 hits in Google Scholar for each search term. “Health” and “evaluation” were not included in the search terms as a pre-test search had revealed this would exclude relevant literature. Articles from personal bibliographies of the authors were also included.

Inclusion categories and criteria

The inclusion process was structured along the five inclusion categories ‘capacity development’, ‘research’, ‘health profession fields’, ‘monitoring and evaluation’, and ‘level of NaME’. Table  1 gives a detailed overview of all descriptions and operationalisations used.

The category ‘capacity development’ [ 18 ] represents an exemplary definition which serves as a guideline for inclusion but should not to be applied word by word. ‘Research’ was operationalized according to the categories of the ‘research spider’ [ 19 ]. Some process-related research skills as well as communicational and interpersonal skills were added to our operationalisation [ 20 ]. Main health professions were identified and grouped within different fields. NaME was operationalized according to a self-constructed NaME framework of HRCD activities (Fig.  1 ), which summarizes 13 HRCD/NaME frameworks [ 2 , 5 , 8 , 10 – 13 , 15 , 21 – 25 ] and reflects the level of HRCD, common indicators, and the order (from needs assessment to impact evaluation) commonly used in the original frameworks.

Framework for needs assessment, monitoring and evaluation (NaME) of health research capacity development (HRCD) [ 2 , 5 , 8 , 10 – 13 , 15 , 21 – 25 ].

For the categories ‘research’, ‘health profession fields’ and ‘monitoring and evaluation’, at least one of the operationalisations of each category had to be addressed by the study. The category ‘level of NaME’ was operationalized referring to the ESSENCE framework ‘Planning, monitoring and evaluation framework for capacity strengthening in health research’ which describes three CD levels: individual and/or team, organizational, and system levels [ 10 ]. Only publications focussing on NaME on the individual/team and organizational levels were considered for this review.

Additionally, the following eligibility criteria were set: English or German language, publication period from January 1, 2003, to June 30, 2013, intervention, non-intervention and multiple design studies (Fig.  2 ). We excluded grey literature, editorials, comments, congress abstracts, letters, and similar. Articles focussing on institutional networks with external partners were excluded as well.

Categorization of the study designs. The study designs are restricted to the included studies.

Study selection

Two researchers, JH and SN, independently scanned the abstracts identified for inclusion. In case of disagreement, JH and SN discussed the abstracts in question. If consensus could still not be reached, a third reviewer, CK, was consulted. After consensus on inclusion was reached, the full-texts of all included studies were rechecked for inclusion by JH and SN.

Study analysis procedure

We analysed the included articles according to nine aspects defined in Table  2 .

The search in PubMed revealed 700 suitable records (Fig.  3 ). We removed 27 duplicates, resulting in 673 records for inclusion screening. The first 200 hits for each of the three search terms in Google Scholar were considered, resulting in two additional records after removing duplicates. Furthermore, we included articles from the personal bibliographies of the authors, adding 10 more abstracts after checking for duplicates. Of the 685 records identified, 24 did not contain an abstract, but were preliminarily included for the full-text screening. JH and SN scanned the remaining 661 abstracts in terms of the inclusion criteria, thus excluding 616 records; 45 abstracts and the 24 records without abstracts were considered for full-text screening. After the full-text screening, 42 articles were finally included for further analysis; 37 articles originated from PubMed, one from Google Scholar, and four from the personal bibliographies of the authors.

Flowchart of the inclusion process.

These 42 articles were subsequently analysed along nine aspects (Table  2 ). The results are summarized in Table  3 .

Around half of the NaME studies on HRCD activities were conducted in high-income countries (n = 24) [ 26 ]. Six studies took place in lower-middle-income and two in upper-middle-income economies. Participants of one study were from a low-income country [ 27 ]. Two studies were performed in partnerships between a high-income and several low-, lower-middle and upper-middle-income economies. Mayhew et al. [ 28 ] described a partnership study between two upper-middle income countries and Bates et al. [ 29 ] analysed case studies from two lower-middle-income and two low-income economies. Five authors did not specify the country or region of their studies.

The evaluation focus of the studies was predominately on outcome evaluation (n = 23). Besides that, six studies surveyed the current state, three studies assessed requirements, and two studies investigated needs of HRCD activities. The remaining eight studies combined two evaluation aspects: definition of needs and outcome evaluation (n = 4), analysis of current state and outcome evaluation (n = 1), outcome evaluation and impact evaluation (n = 1), and analysis of current state and definition of needs (n = 1). Jamerson et al. [ 30 ] did not define their focus of evaluation.

Nearly half of the studies investigated HRCD on the individual/team level (n = 20); 16 studies were conducted at both the individual/team and organizational levels. The authors of six studies focused on organizational aspects of HRCD.

Almost all studies (n = 38) described and evaluated HRCD activities; 19 of these HRCD activities were training programmes of predefined duration, lasting between some hours or days up to 2 years. Another nine HRCD activities were perpetual or their duration not specified and 10 studies defined and pre-assessed the setting in preparation of an HRCD activity. The authors of four studies did not specify an HRCD activity, focussing on the development or validation of tools, instruments, and frameworks.

The participants of HRCD activities represent a wide range of health professions (e.g. laboratory scientists, physiotherapists, dentists, pharmacists); 10 studies investigated staff with management tasks in health, e.g. hospital managers, clinical research managers. Nurses participated in eight studies with another eight studies looking into ‘research staff’ and ‘scientists’ with no further description. Medical practitioners were studied in five papers. Besides all these, the background of participants was often not specified beyond general terms like ‘health professionals’, ‘ethic committee members’, ‘scholars’, ‘university faculty members’, or ‘allied health professionals’. In a different approach, Suter et al. [ 31 ] analysed reports and Bates et al. [ 29 ] investigated case studies (without specifying the material scrutinized).

A wide variety of study designs was employed by the studies included in the review. We identified 35 single-study and six multi-study approaches. Of the 35 single-study approaches, 10 were designed as intervention (three with control groups) and 25 as non-intervention studies. Four multi-study approaches combined an intervention study with a non-intervention study. Two multi-study approaches combined different non-intervention studies. Jamerson et al. [ 30 ] did not specify their study design.

Many different tools and instruments for NaME were identified and applied in quantitative, qualitative and mixed mode of analysis. No preferred approach was observed. One third of the studies (n = 16) used a combination of tools for quantitative as well as qualitative analysis. In 13 studies, tools like questionnaires and assessment sheets were applied to evaluate and monitor HRCD activities quantitatively. Evaluation tools, such as interviews, focus group discussions, document analyses, or mapping of cases against evaluation frameworks, were identified in 12 studies and commonly analysed in a qualitative approach. In one study, tools for evaluation were not described at all.

Summary of evidence

The aim of our systematic review was to give an overview on tools and instruments for NaME of HRCD activities on the individual and organizational level; 42 included articles demonstrated a large variety of tools and instruments in specific settings. Questionnaires, assessment sheets and interviews (in qualitative settings) were most commonly applied and in part disseminated for further use, development and validation.

Overall, 36 studies were either conducted on the individual/team or on both individual/team and organizational level. Within these studies, a well-balanced mixture of quantitative, qualitative and mixed tools and modes of analysis were applied. Judging from the depth of these studies, it seems as if NaME of HRCD on the individual level is quite well developed. Only six studies focused exclusively on organizational aspects, almost all with qualitative approaches, indicating that HRCD studies at this level are still mainly exploratory. The organizational level is possibly a more complex construct to measure. The fact that 13 out of 19 studies that broach organizational aspects were conducted in high-income countries might reflect the wider possibilities of these research institutions and indicates a need for more attention to NaME on the organizational level in lower-income settings. Results from these exploratory studies on the organizational level should feed into the development of standardized quantitative indicators more regularly. Qualitative approaches could be pursued for complex and specific constructs not easily covered quantitatively.

By not limiting the primary selection of articles for this review to a specific health profession, it was revealed that staff with management tasks in health research, as well as nurses, were the cohorts most frequently targeted by NaME studies. Further research should concentrate on other health professionals to determine communalities and differences of health-research related skill acquisition and development between health professions. These studies could determine whether and which parts of HRCD and NaME can be considered generic across health professions. Further, we will at some point have to ask, who is being left out and who is not getting access to HRCD programs, and why.

The focus of NaME throughout the studies included in this review was on outcome measurement, regardless of whether these were conducted in high-income, upper-middle, lower-middle, or low-income countries. However, there were only few reports of needs assessment from middle- and low-income economies, while high-income countries regularly give account of current states. While this should not be over-interpreted, it still raises the question of whether the needs assessment in the middle- and low-income countries is being done as thoroughly as warranted, but not reported in the articles, or if these countries’ needs might not always be at the very centre of the HRCD’s attention. While the evaluation of HRCD outcomes is, of course, of importance, more attention should be paid to the sustainability of programs and impact evaluation, e.g. parameters of patient care or societal aspects. Only one study, that of Hyder et al. [ 32 ], made use of one such indicator and assessed the impact of a HRCD training by considering “teaching activities after returning to Pakistan”. The development of valid impact indicators of course constitutes a methodological challenge. Some studies reporting impact evaluation on a system level might of course have been missed due to the search parameters applied.

When undertaking the review, three main methodological weaknesses of this research area became apparent. First, there is a need for common definitions and terminologies to better communicate and compare the HRCD efforts. The analysis of the studies showed that there is an inconsistent use of terms, for example, for CD activities (e.g. training, course, or workshop). Similar problems were already identified in the context of educational capacity building by Steinert et al. [ 33 ], who suggest definitions for different training settings which may also be suitable for a more precise description of CD activities. A common taxonomy for the description of health professionals (i.e. the study participants) would be just as desirable. The use of coherent terms would not only enable the accurate replication of studies but also help in determining whether tools and instruments from one setting can be easily transferred to another. A clear and coherent description of study setting and participants is thus an integral step towards scientific transparency. The incoherent categorisation of study types is probably not a new problem. It is, however, amplified by authors who choose very complex approaches to collect data at different NaME levels with deviating terms to describe these approaches [ 28 , 34 – 36 ].

The second weakness of the research area is the varying adherence to reporting standards. While there are standards available for reporting qualitative or quantitative research (e.g. Rossi et al. [ 12 ], Downing [ 37 ], Mays & Pope [ 38 ]), it seems these or similar recommendations were not frequently considered when reporting or reviewing NaME studies. This was particularly the case in studies with a mixed-method mode of analysis, where the need for more standardised reporting became apparent. Frambach et al.’s [ 39 ] “Quality Criteria in Qualitative and Quantitative Research” could provide guidance, especially for studies with mixed-method approaches. Another important aspect of transparent reporting would be the publication of the tools and instruments used in NaME studies. Of the 42 articles scrutinized during this review, only 15 either disclosed the tools and instruments within the article itself in an appendix or volunteered to have them sent to any audience interested. Of all the tools and instruments disclosed, only two were used in two or more studies. Making the tools and instruments available to the HRCD community would not only allow for their adaptation whenever necessary but, more importantly, support their validation and enhancement.

The last point concerns the study designs implemented. The majority of articles are mainly descriptive, non-intervention studies that only allow for low evidence according to Cochrane standards [ 40 ]. While most HRCD studies conducted in high-income economies were of non-interventional nature, those from low- and middle-income countries were a mix of non-intervention, intervention and multi-study approaches, yielding higher levels of evidence. Of all interventional studies, most employed a quasi-experimental design with only one randomized controlled trial [ 23 ]. The studies reporting HRCD on the institutional level were also primarily on a descriptive level. Cook et al. [ 41 ], however, demand going beyond describing what one did (descriptive studies) or whether an intervention worked or not (justification studies). Instead, they call for analysing how and why a program worked or failed (clarification studies). An in-depth analysis of the effectiveness of different HRCD activities is, however, still lacking.

Limitations of the systematic review

This systematic review displays some methodological limitations itself. The issue of deviating terminologies has been raised earlier. In most cases, we adopted the terms used in the studies themselves, e.g. when reporting the authors’ denoted study designs. In very few cases, we changed or completed terms to make the studies more comparable to others. One example is changing the wording from Green et al.’s [ 35 ] “case study approach” into a “multi-study approach” to match Flyvberg’s taxonomy [ 42 ]. Other limitations typical for reviews may also apply. Relevant sources might not have been detected due to the selected search terms, the range of the data sources, the exclusion of grey literature, and the restriction to English and German sources.

A systematic review on studies from the field of HRCD activities was conducted, with 42 studies being fully analysed. The analysis revealed that a variety of terms and definitions used to describe NaME efforts impedes the comparability and transferability of results. Nevertheless, insight from this review can help to inform researchers and other stakeholders in the HRCD community. A coherent overview on tools and instruments for NaME of HRCD was developed and is provided (Table  3 ).

Furthermore, it is time to set standards for NaME in the HRCD community. Researchers and stakeholders should develop a common research agenda to push, systematise and improve the research efforts in the field of NaME of HRCD activities. To do so, a common language and terminology is required. The conceptualizations used for the purpose of these review can inform this development. On the other hand, we have to critically analyse research gaps in terms of generalizable versus context-specific theories, methods, tools, and instruments. To maximize the benefits and to incorporate different research traditions, these undertakings should be done internationally and multi-professionally within the HRCD community.

Lusthaus C, Adrien M-H, Perstinger M. Capacity Development: Definitions, Issues and Implications for Planning, Monitoring and Evaluation. Universalia Occasional Paper. 1999;35:[about 21 p.]. http://preval.org/documentos/2034.pdf . Accessed 2 July 2015.

Labin SN, Duffy JL, Meyers DC, Wandersman A, Lesesne CA. A research synthesis of the evaluation capacity building literature. Am J Eval. 2012;33(3):307–38.

Article   Google Scholar  

Gadsby EW. Research capacity strengthening: donor approaches to improving and assessing its impact in low- and middle-income countries. Int J Health Plann Manage. 2011;26(1):89–106. doi: 10.1002/hpm.1031 .

Article   PubMed   Google Scholar  

Bailey J, Veitch C, Crossland L, Preston R. Developing research capacity building for Aboriginal & Torres Strait Islander health workers in health service settings. Rural Remote Health. 2006;6(4):556.

PubMed   CAS   Google Scholar  

Lansang MA, Dennis R. Building capacity in health research in the developing world. Bull World Health Organ. 2004;82(10):764–70.

PubMed   PubMed Central   Google Scholar  

United Nations. Millennium Development Goals and Beyond. 2015. http://www.un.org/millenniumgoals/ . Accessed 2 July 2015.

Google Scholar  

Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26. doi: 10.1186/1478-4505-9-26 .

Article   PubMed   PubMed Central   Google Scholar  

Cooke J. A framework to evaluate research capacity building in health care. BMC Fam Pract. 2005;6:44. doi: 10.1186/1471-2296-6-44 .

Trostle J. Research capacity building in international health: definitions, evaluations and strategies for success. Soc Sci Med. 1992;35(11):1321–4.

Article   PubMed   CAS   Google Scholar  

ESSENCE (Enhancing Support for Strengthening the Effectiveness of National Capacity Efforts). Planning, monitoring and evaluation framework for capacity strengthening in health research. Geneva: ESSENCE Good practice document series; 2011. http://whqlibdoc.who.int/hq/2011/TDR_essence_11.1_eng.pdf?ua = 1. Accessed 2 July 2015.

Ghaffar A, IJsselmuiden C, Zicker F. Changing mindsets: Research capacity strengthening in low- and middle-income countries. Geneva: Council on Health Research for Development (COHRED), Global Forum for Health Research, INICEF/UNDP/World Bank/WHO Special Programme for Research and Training in Tropical Diseases (TDR); 2008. http://www.cohred.org/downloads/cohred_publications/Changing_Mindsets.pdf . Accessed 30 November 2015.

Rossi PH, Lipsey MW, Freeman HE. Evaluation. A systematic approach. 7th ed. Thousand Oaks: Sage Publications; 2004.

Donabedian A. The quality of care. How can it be assessed? 1988. Arch Pathol Lab Med. 1997;121(11):1145–50.

McLaughlin JA, Jordan GB. Logic models: a tool for telling your programs performance story. Eval Program Plan. 1999;22(1):65–72.

Best A, Terpstra JL, Moor G, Riley B, Norman CD, Glasgow RE. Building knowledge integration systems for evidence-informed decisions. J Health Organ Manage. 2009;23(6):627–41.

Bates I, Boyd A, Smith H, Cole DC. A practical and systematic approach to organisational capacity strengthening for research in the health sector in Africa. Health Res Policy Syst. 2014;12:11. doi: 10.1186/1478-4505-12-11 .

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100. doi: 10.1371/journal.pmed.1000100 . Epub 2009 Jul 21.

Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH. Capacity WORKS: The Management Model for Sustainable Development. Eschborn: GTZ; 2009.

Smith H, Wright D, Morgan S, Dunleavey J. The ‘Research Spider’: a simple method of assessing research experience. Prim Health Care Res Dev. 2002;3:139–40.

Bauer D, Wessels I, Huber J, Fischer MR, Kiessling C. Current State and Needs Assessment for Individual and Organisational Research Capacity Strengthening in Africa: A Case Report from Mbeya (TZ), and Kumasi (GH). Eschborn: Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH; 2013.

Bates I, Akoto AY, Ansong D, Karikari P, Bedu-Addo G, Critchley J, et al. Evaluating health research capacity building: An evidence-based tool. PLoS Med. 2006;3(8), e299.

Golenko X, Pager S, Holden L. A thematic analysis of the role of the organisation in building allied health research capacity: a senior managers’ perspective. BMC Health Serv Res. 2012;12:276. doi: 10.1186/1472-6963-12-276 .

Ijsselmuiden C, Marais DL, Becerra-Posada F, Ghannem H. Africa’s neglected area of human resources for health research - the way forward. S Afr Med J. 2012;102(4):228–33.

Levine R, Russ-Eft D, Burling A, Stephens J, Downey J. Evaluating health services research capacity building programs: Implications for health services and human resource development. Eval Program Plan. 2013;37:1–11. doi: 10.1016/j.evalprogplan.2012.12.002 . Epub 2012 Dec 12.

Mahmood S, Hort K, Ahmed S, Salam M, Cravioto A. Strategies for capacity building for health research in Bangladesh: Role of core funding and a common monitoring and evaluation framework. Health Res Policy Syst. 2011;9:31. doi: 10.1186/1478-4505-9-31 .

The World Bank. Data: Country and Lending Groups. 2015. http://data.worldbank.org/about/country-and-lending-groups . Accessed 2 July 2015.

Njie-Carr V, Kalengé S, Kelley J, Wilson A, Muliira JK, Nabirye RC, et al. Research capacity-building program for clinicians and staff at a community-based HIV clinic in Uganda: a pre/post evaluation. J Assoc Nurses AIDS Care. 2012;23(5):431–41. doi: 10.1016/j.jana.2011.11.002 .

Mayhew SH, Doherty J, Pitayarangsarit S. Developing health systems research capacities through north–south partnership: an evaluation of collaboration with South Africa and Thailand. Health Res Policy Syst. 2008;6:8. doi: 10.1186/1478-4505-6-8 .

Bates I, Taegtmeyer M, Squire BS, Ansong D, Nhlema-Simwaka B, Baba A, et al. Indicators of sustainable capacity building for health research: analysis of four African case studies. Health Res Policy Syst. 2011;9:14. doi: 10.1186/1478-4505-9-14 .

Jamerson PA, Fish AF, Frandsen G. Nursing Student Research Assistant Program: A strategy to enhance nursing research capacity building in a Magnet status pediatric hospital. Appl Nurs Res. 2011;24(2):110–3. doi: 10.1016/j.apnr.2009.08.004 .

Suter E, Lait J, Macdonald L, Wener P, Law R, Khalili H, et al. Strategic approach to building research capacity in inter-professional education and collaboration. Healthc Q. 2011;14(2):54–60.

Hyder AA, Akhter T, Qayyum A. Capacity development for health research in Pakistan: the effects of doctoral training. Health Policy Plan. 2003;18(3):338–43.

Steinert Y, Mann K, Centeno A, Dolmans D, Spencer J, Gelula M, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2006;28(6):497–526.

Ajuwon AJ, Kass N. Outcome of a research ethics training workshop among clinicians and scientists in a Nigerian university. BMC Medical Ethics. 2008;9:1. doi: 10.1186/1472-6939-9-1 .

Green B, Segrott J, Hewitt J. Developing nursing and midwifery research capacity in a university department: case study. J Adv Nurs. 2006;56(3):302–13.

Janssen J, Hale L, Mirfin-Veitch B, Harland T. Building the research capacity of clinical physical therapists using a participatory action research approach. Phys Ther. 2013;93(7):923–34. doi: 10.2522/ptj.20120030 .

Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7.

Mays N, Pope C. Qualitative research in health care. Assessing quality in qualitative research. BMJ. 2000;320(7226):50–2.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Frambach JM, van der Vleuten CP, Durning SJ. AM last page. Quality criteria in qualitative and quantitative research. Acad Med. 2013;88(4):552. doi: 10.1097/ACM.0b013e31828abf7f .

Cochrane Consumer Network. Levels of Evidence. 2015. http://consumers.cochrane.org/levels-evidence#about-cochrane . Accessed 30 November 2015.

Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008;42(2):128–33. doi: 10.1111/j.1365-2923.2007.02974.x . Epub 2008 Jan 8.

Bent Flyvbjerg. Five misunderstandings about case-study research. Qual Inq. 2006;12:219. doi: 10.1177/1077800405284363 .

Ali J, Hyder AA, Kass NE. Research ethics capacity development in Africa: exploring a model for individual success. Dev World Bioeth. 2012;12(2):55–62. doi: 10.1111/j.1471-8847.2012.00331.x .

Barchi FH, Kasimatis-Singleton M, Kasule M, Khulumani P, Merz JF. Building research capacity in Botswana: a randomized trial comparing training methodologies in the Botswana ethics training initiative. BMC Med Educ. 2013;13:14. doi: 10.1186/1472-6920-13-14 .

Bates I, Ansong D, Bedu-Addo G, Agbenyega T, Akoto AY, Nsiah-Asare A, et al. Evaluation of a learner-designed course for teaching health research skills in Ghana. BMC Med Educ. 2007;7:18.

Bullock A, Morris ZS, Atwell C. Collaboration between health services managers and researchers: making a difference? J Health Serv Res Policy. 2012;17 Suppl 2:2–10. doi: 10.1258/jhsrp.2011.011099 .

Kirkpatrick DL. Evaluating training programs. Mumbai: Tata McGraw-Hill Education; 1998.

Kirkpatrick DL. Techniques for evaluating training programs. Training Dev J. 1979;33:78–92.

Cooke J, Nancarrow S, Dyas J, Williams M. An evaluation of the ‘Designated Research Team’ approach to building research capacity in primary care. BMC Fam Pract. 2008;9:37. doi: 10.1186/1471-2296-9-37 .

Corchon S, Portillo MC, Watson R, Saracíbar M. Nursing research capacity building in a Spanish hospital: an intervention study. J Clin Nurs. 2011;20(17–18):2479–89. doi: 10.1111/j.1365-2702.2011.03744.x .

Dodani S, La Porte RE. Ways to strengthen research capacity in developing countries: effectiveness of a research training workshop in Pakistan. Public Health. 2008;122(6):578–87. doi: 10.1016/j.puhe.2007.09.003 .

du Plessis E, Human SP. Reflecting on ‘meaningful research’: a qualitative secondary analysis. Curationis. 2009;32(3):72–9.

Finch E, Cornwell P, Ward EC, McPhail SM. Factors influencing research engagement: research interest, confidence and experience in an Australian speech-language pathology workforce. BMC Health Serv Res. 2013;13:144. doi: 10.1186/1472-6963-13-144 .

Holden L, Pager S, Golenko X, Ware RS, Weare R. Evaluating a team-based approach to research capacity building using a matched-pairs study design. BMC Fam Pract. 2012;13:16. doi: 10.1186/1471-2296-13-16 .

Henderson-Smart DJ, Lumbiganon P, Festin MR, Ho JJ, Mohammad H, McDonald SJ, et al. Optimising reproductive and child health outcomes by building evidence-based research and practice in South East Asia (SEA-ORCHID): study protocol. BMC Med Res Methodol. 2007;7:43.

Holden L, Pager S, Golenko X, Ware RS. Validation of the research capacity and culture (RCC) tool: measuring RCC at individual, team and organisation levels. Aust J Prim Health. 2012;18(1):62–7. doi: 10.1071/PY10081 .

Hyder AA, Harrison RA, Kass N, Maman S. A case study of research ethics capacity development in Africa. Acad Med. 2007;82(7):675–83.

Jones A, Burgess TA, Farmer EA, Fuller J, Stocks NP, Taylor JE, et al. Building research capacity. An exploratory model of GPs’ training needs and barriers to research involvement. Aust Fam Physician. 2003;32(11):957–60.

PubMed   Google Scholar  

Kwon S, Rideout C, Tseng W, Islam N, Cook WK, Ro M, et al. Developing the community empowered research training program: building research capacity for community-initiated and community-driven research. Prog Community Health Partnersh. 2012;6(1):43–52. doi: 10.1353/cpr.2012.0010 .

Lazzarini PA, Geraghty J, Kinnear EM, Butterworth M, Ward D. Research capacity and culture in podiatry: early observations within Queensland Health. J Foot Ankle Res. 2013;6(1):1. doi: 10.1186/1757-1146-6-1 .

McIntyre E, Brun L, Cameron H. Researcher development program of the primary health care research, evaluation and development strategy. Aust J Prim Health. 2011;17(1):114–21. doi: 10.1071/PY10049 .

Minja H, Nsanzabana C, Maure C, Hoffmann A, Rumisha S, Ogundahunsi O, et al. Impact of health research capacity strengthening in low- and middle-income countries: the case of WHO/TDR programmes. PLoS Negl Trop Dis. 2011;5(10):e1351. doi: 10.1371/journal.pntd.0001351 .

Moore J, Crozier K, Kite K. An action research approach for developing research and innovation in nursing and midwifery practice: building research capacity in one NHS foundation trust. Nurse Educ Today. 2012;32(1):39–45. doi: 10.1016/j.nedt.2011.01.014 .

Otiniano AD, Carroll-Scott A, Toy P, Wallace SP. Supporting Latino communities’ natural helpers: a case study of promotoras in a research capacity building course. J Immigr Minor Health. 2012;14(4):657–63. doi: 10.1007/s10903-011-9519-9 .

Pager S, Holden L, Golenko X. Motivators, enablers, and barriers to building allied health research capacity. J Multidiscip Healthc. 2012;5:53–9. doi: 10.2147/JMDH.S27638 .

Perry L, Grange A, Heyman B, Noble P. Stakeholders’ perceptions of a research capacity development project for nurses, midwives and allied health professionals. J Nurs Manag. 2008;16(3):315–26. doi: 10.1111/j.1365-2834.2007.00801.x .

Priest H, Segrott J, Green B, Rout A. Harnessing collaboration to build nursing research capacity: a research team journey. Nurse Educ Today. 2007;27(6):577–87.

Green B, Segrott J, Coleman M, Cooke A. Building the research capacity of an academic department of nursing. Occasional Paper 1a. Swansea: School of Health Sciences, University of Wales; 2005.

Segrott J, Green B, McIvor M. Challenges and strategies in developing nursing research capacity: a review of the literature. Int J Nurs Stud. 2006;43(5):637–51.

Redman-MacLaren M, MacLaren DJ, Harrington H, Asugeni R, Timothy-Harrington R, Kekeubata E, et al. Mutual research capacity strengthening: a qualitative study of two-way partnerships in public health research. Int J Equity Health. 2012;11:79. doi: 10.1186/1475-9276-11-79 .

Ried K, Farmer EA, Weston KM. Setting directions for capacity building in primary health care: a survey of a research network. BMC Fam Pract. 2006;7:8.

Salway S, Piercy H, Chowbey P, Brewins L, Dhoot P. Improving capacity in ethnicity and health research: report of a tailored programme for NHS Public Health practitioners. Prim Health Care Res Dev. 2013;14(4):330–40. doi: 10.1017/S1463423612000357 .

Webster E, Thomas M, Ong N, Cutler L. Rural research capacity building program: capacity building outcomes. Aust J Prim Health. 2011;17(1):107–13. doi: 10.1071/PY10060 .

Wilson LL, Rice M, Jones CT, Joiner C, LaBorde J, McCall K, et al. Enhancing research capacity for global health: evaluation of a distance-based program for international study coordinators. J Contin Educ Health Prof. 2013;33(1):67–75. doi: 10.1002/chp.21167 .

Wootton R. A simple, generalizable method for measuring individual research productivity and its use in the long-term analysis of departmental performance, including between-country comparisons. Health Res Policy Syst. 2013;11:2. doi: 10.1186/1478-4505-11-2 .

Download references

Author information

Authors and affiliations.

Institut für Didaktik und Ausbildungsforschung in der Medizin, Klinikum der Universität München, Ziemssenstraße 1, 80336, Munich, Germany

Johanna Huber, Sushil Nepal, Daniel Bauer, Martin R Fischer & Claudia Kiessling

bologna.lab, Humboldt-Universität zu Berlin, Hausvogteiplatz 5-7, 10117, Berlin, Germany

Insa Wessels

Medizinische Hochschule Brandenburg Theodor Fontane, Fehrbelliner Straße 38, 16816, Neuruppin, Germany

Claudia Kiessling

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johanna Huber .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

JH and SN designed and conducted the systematic review. JH wrote the draft of the systematic review and revised it according to the commentaries of SN, DB, IW, MF, and CK. JH provided the final version of the manuscript. SN additionally critically reviewed the manuscript and substantially contributed to the final version of the manuscript. DB critically reviewed both the design of the systematic review as well as the manuscript. He was involved in the development of meaningful inclusion criteria. DB contributed substantially to the final version of the manuscript. IW critically reviewed the design of the study and made important suggestions for improvement. She also critically reviewed the manuscript and contributed substantially to the final version of the manuscript. MF critically reviewed the design of the study and the manuscript. He suggested important improvements for the design of the study and substantially contributed to the final version of the manuscript. CK made substantial contributions to the design, conduction and review of the study, and was the third reviewer during the inclusion process of the identified studies. She critically reviewed the manuscript and delivered important improvements for the final version of the manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Huber, J., Nepal, S., Bauer, D. et al. Tools and instruments for needs assessment, monitoring and evaluation of health research capacity development activities at the individual and organizational level: a systematic review. Health Res Policy Sys 13 , 80 (2015). https://doi.org/10.1186/s12961-015-0070-3

Download citation

Received : 23 July 2015

Accepted : 07 December 2015

Published : 21 December 2015

DOI : https://doi.org/10.1186/s12961-015-0070-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health research capacity development
• Individual level
• Monitoring and evaluation
• Needs assessment
• Organizational level

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

A guiding framework for needs assessment evaluations to embed digital platforms in partnership with Indigenous communities

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft

Affiliation School of Occupational and Public Health, Toronto Metropolitan University, Toronto, ON, Canada

Roles Data curation, Formal analysis, Investigation, Software, Visualization, Writing – original draft

Affiliation School of Public Health Sciences, University of Waterloo, Waterloo, ON, Canada

Roles Conceptualization, Investigation, Project administration, Resources, Supervision, Writing – review & editing

Affiliation Île-à-la-Crosse School Division, The Northern Village of Île-à-la-Crosse, Île-à-la-Crosse, SK, Canada

Roles Conceptualization, Investigation, Resources, Supervision

Roles Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – review & editing

* E-mail: [email protected]

Affiliations DEPtH Lab, Faculty of Health Sciences, Western University, London, ON, Canada, Department of Epidemiology and Biostatistics, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada, Lawson Health Research Institute, London, Ontario, Canada

• Jasmin Bhawra, 
• M. Claire Buchan, 
• Brenda Green, 
• Kelly Skinner, 
• Tarun Reddy Katapally

• Published: December 22, 2022
• https://doi.org/10.1371/journal.pone.0279282
• Reader Comments

Introduction

In community-based research projects, needs assessments are one of the first steps to identify community priorities. Access-related issues often pose significant barriers to participation in research and evaluation for rural and remote communities, particularly Indigenous communities, which also have a complex relationship with academia due to a history of exploitation. To bridge this gap, work with Indigenous communities requires consistent and meaningful engagement. The prominence of digital devices (i.e., smartphones) offers an unparalleled opportunity for ethical and equitable engagement between researchers and communities across jurisdictions, particularly in remote communities.

This paper presents a framework to guide needs assessments which embed digital platforms in partnership with Indigenous communities. Guided by this framework, a qualitative needs assessment was conducted with a subarctic Métis community in Saskatchewan, Canada. This project is governed by an Advisory Council comprised of Knowledge Keepers, Elders, and youth in the community. An environmental scan of relevant programs, three key informant interviews, and two focus groups (n = 4 in each) were conducted to systematically identify community priorities.

Through discussions with the community, four priorities were identified: (1) the Coronavirus pandemic, (2) climate change impacts on the environment, (3) mental health and wellbeing, and (4) food security and sovereignty. Given the timing of the needs assessment, the community identified the Coronavirus pandemic as a key priority requiring digital initiatives.

Recommendations for community-based needs assessments to conceptualize and implement digital infrastructure are put forward, with an emphasis on self-governance and data sovereignty.

Citation: Bhawra J, Buchan MC, Green B, Skinner K, Katapally TR (2022) A guiding framework for needs assessment evaluations to embed digital platforms in partnership with Indigenous communities. PLoS ONE 17(12): e0279282. https://doi.org/10.1371/journal.pone.0279282

Editor: Stephane Shepherd, Swinburne University of Technology, AUSTRALIA

Received: June 1, 2022; Accepted: December 2, 2022; Published: December 22, 2022

Copyright: © 2022 Bhawra et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data are co-owned by the community and all data requests should be approved by the Citizen Scientist Advisory Council and the University of Regina Research Office. Citizen Scientist Advisory Council Contact: Mr. Duane Favel, Mayor of Ile-a-lacrosse, email: [email protected] ; [email protected] University of Regina Research Office contact: Ara Steininger, Research Compliance Officer; E-mail: [email protected] . Those interested can access the data in the same manner as the authors.

Funding: TRK received funding from the Canadian Institutes of Health Research (CIHR) and the Canada Research Chairs Program to conduct this research. The funding organization had no role to play in any part of the study implementation of manuscript generation.

Competing interests: The authors have declared that no competing interests exist.

Community engagement has been the cornerstone of participatory action research in a range of disciplines. Every community has a unique culture and identity, hence community members are the experts regarding their diverse histories, priorities, and growth [ 1 – 3 ]. As a result, the successful uptake, implementation, and longevity of community-based research initiatives largely depends on meaningful community engagement [ 4 – 9 ]. There is a considerable body of evidence establishing the need for ethical community-research partnerships which empower citizens and ensure relevant and sustainable solutions [ 1 – 3 , 10 ]. For groups that have been marginalized or disadvantaged, community-engaged research that prioritizes citizens’ control in the research process can provide a platform to amplify citizens’ voices and ensure necessary representation in decision-making [ 11 ]. Such initiatives must be developed in alignment with a community’s cultural framework, expectations, and vision [ 12 ] to support continuous and meaningful engagement throughout the project. In particular, when partnering with Indigenous communities, a Two-Eyed Seeing approach can provide valuable perspective to combine the strengths of Indigenous and Western Knowledges, including culturally relevant methods, technologies, and tools [ 13 – 15 ].

Many communities have a complicated relationship with research as a result of colonialism, and the trauma of exploitation and discrimination has continued to limit the participation of some communities in academic partnerships [ 16 ]. Indigenous Peoples in Canada experience a disproportionate number of health, economic, and social inequalities compared to non-Indigenous Canadians [ 17 ]. Many of these health (e.g., elevated risk of chronic and communicable diseases) [ 18 – 21 ]), socioeconomic (e.g., elevated levels of unemployment and poverty) [ 19 , 22 – 24 ], and social (e.g., racism and discrimination) [ 19 , 22 – 24 ]) inequities can be traced back to the long-term impacts of assimilation, colonization, residential schools, and a lack of access to healthcare [ 19 , 20 , 22 – 24 ]. To bridge this gap, and more importantly, to work towards Truth and Reconciliation [ 25 ], work with Indigenous Peoples must be community-driven, and community-academia relationship building is essential before exploring co-conceptualization of initiatives [ 26 ].

One of the first steps in building a relationship is to learn more about community priorities by conducting a needs assessment [ 27 , 28 ]. A needs assessment is a research and evaluation method for identifying areas for improvement or gaps in current policies, programs, and services [ 29 ]. When conducted in partnership with a specific community, needs assessments can identify priorities and be used to develop innovative solutions, while leveraging the existing knowledge and systems that communities have in place [ 30 ]. Needs assessments pave the path for understanding the value and applicability of research for community members, incorporating key perspectives, and building authentic partnerships with communities to support effective translation of research into practice.

For rural, remote, and northern communities within Canada, issues related to access (e.g., geographic location, transportation, methods of communication, etc.) pose significant barriers to participation in research and related initiatives [ 31 ]. Digital devices, and in particular, the extensive usage of smartphones [ 32 ] offers a new opportunity to ethically and equitably engage citizens [ 33 ]. Digital platforms (also referred to as digital tools) are applications and software programs accessible through digital devices. Digital platforms can be used for a variety of purposes, ranging from project management, to healthcare delivery or mass communication [ 34 ]. Digital infrastructure–the larger systems which support access and use of these digital platforms, including internet, satellites, cellular networks, and data storage centres [ 34 ]. The Coronavirus (COVID-19) pandemic has catalyzed the expansion of digital technology, infrastructure and the use of digital devices in delivering essential services (e.g., healthcare) and programs to communities [ 35 , 36 ].

While digital platforms have been used in Indigenous communities for numerous initiatives, including environmental mapping initiatives (e.g., research and monitoring, land use planning, and wildlife and harvest studies) [ 37 , 38 ] and telehealth [ 39 ], there has largely been isolated app development without a corresponding investment in digital infrastructure. This approach limits the sustainability of digital initiatives, and importantly does not acknowledge an Indigenous world view of holistic solutions [ 39 ].

Thus given the increasing prominence of digital devices [ 39 , 40 ], it is critical to evaluate the conceptualization, implementation, and knowledge dissemination of digital platforms. To date, there is little guidance on how to evaluate digital platforms, particularly in partnership with rural and remote communities [ 41 ]. A review of recent literature on community-based needs assessments uncovered numerous resources for conducting evaluations of digital platforms, however, a key gap is the lack of practical guidance for conducting needs assessments in close collaboration with communities in ways that acknowledge existing needs, resources, supports and infrastructure that also incorporates the potential role of digital platforms in addressing community priorities.

This paper aims to provide researchers and evaluators with a framework (step-by-step guide) to conduct needs assessments for digital platforms in collaboration with Indigenous communities. To achieve this goal, a novel needs assessment framework was developed using a Two-Eyed Seeing approach [ 13 – 15 ] to enable the identification of community priorities, barriers and supports, as well as existing digital infrastructure to successfully implement digital solutions. To demonstrate the application of this framework, a community-engaged needs assessment conducted with a subarctic Indigenous community in Canada is described and discussed in detail.

Framework design and development

This project commenced with the design and development of a new framework to guide community-based needs assessments in the digital age.

Needs assessments

Needs assessments are a type of formative evaluation and are often considered a form of strategic or program planning, even more than they are considered a type of evaluation. Needs assessments can occur both before and during an evaluation or program implementation; however, needs assessments are most effective when they are conducted before a new initiative begins or before a decision is made about what to do (e.g., how to make program changes) [ 29 ]. Typically, a needs assessment includes: 1) collecting information about a community; 2) determining what needs are already being met; and 3) determining what needs are not being met and what resources are available to meet those needs [ 42 ].

Framework development

Based on existing literature, community consultation, and drawing expertise from our team of evaluation experts who have over a decade of experience working with Indigenous communities on a range of research and evaluation projects, a novel framework was developed to guide community-based needs assessments focused on the application of digital platforms.

This framework (see Fig 1 ) is driven by core questions necessary to identify community priorities that can be addressed by developing and implementing digital platforms. Through team discussion and community consultation, five key topic areas for the assessment of community needs were identified: i) current supports; ii) desired supports; iii) barriers; iv) community engagement; and v) digital access and connectivity. A series of general questions across the five needs assessment topic areas were developed. Thereafter, a set of sub-questions were embedded in each key topic area.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0279282.g001

The Guiding Framework outlines an approach for conducting community needs assessments which can be adapted across communities and jurisdictions. This framework offers a flexible template that can be used iteratively and applied to various community-engaged needs assessments in a range of areas, including but not limited to community health and wellness projects. The questions assigned to each topic area can be used to guide needs assessments of any priority identified by community stakeholders as suitable for addressing with digital platforms.

Needs assessment methods

The Guiding Framework was implemented in collaboration with a subarctic Indigenous community in Canada, and was used to identify key community priorities, barriers, supports, and existing digital infrastructure which could inform the design and implementation of tailored digital platforms.

Using an environmental scan of relevant documents and qualitative focus groups and interviews, a needs assessment was conducted with the Northern Village of Île-à-la-Crosse, Saskatchewan, Canada between February and May 2020.

This project is governed by a Citizen Scientist Advisory Council which included researchers, Knowledge Keepers, Elders, and youth from Île-à-la-Crosse. The study PI (TRK) and Co-Investigator (JB) developed a relationship with key decision-makers in Île-à-la-Crosse in 2020. Through their guidance and several community visits, the decision-makers introduced the research team to Elders, youth, and other community members to gain a better understanding of current priorities and needs in Île-à-la-Crosse. The research team developed relationships with these community members and invited them to join the Council to formally capture feedback and plan ongoing projects to promote health and wellbeing in the community. The Council represents the needs and interests of the community, and guides the project development, implementation, and evaluation. Council members were provided with Can $150 (US $119.30) as honoraria for each meeting to respect their time, knowledge, and contributions.

Written consent was obtained from all focus group participants and verbal consent was obtained from all key informants participating in interviews. This study received ethics clearance from the research ethics boards of the University of Regina and the University of Saskatchewan through a synchronized review protocol (REB# 2017–29).

Established in 1776, Île-à-la-Crosse is a northern subarctic community with road access in northwest Saskatchewan. Sakitawak, the Cree name for Île-à-la-Crosse, means “where the rivers meet,” hence the community was an historically important meeting point for the fur trade in the 1800s [ 43 , 44 ] The community lies on a peninsula on the Churchill River, near the intersections with the Beaver River and Canoe River systems. Île-à-la-Crosse has a rich history dating back to the fur trade. Due to its strategic location, Montreal-based fur traders established the first trading point in Île-à-la-Crosse in 1776, making the community Saskatchewan’s oldest continually inhabited community next to Cumberland House [ 45 ]. In 1821, Île-à-la-Crosse became the headquarters for the Hudson’s Bay Company’s operations in the territory. In 1860, the first convent was established bringing Western culture, medical services, and education to the community.

Île-à-la-Crosse has a population of roughly 1,300 people [ 19 ]. Consistent with Indigenous populations across Canada, the average age of the community is 32.7 years, roughly 10 years younger than the Canadian non-Indigenous average [ 19 ]. Census data report that just under half (44%) of the community’s population is under the age of 25, 46.3% are aged 25–64, and 9.3% aged 65 and over [ 19 ]. Members of the community predominantly identify as Métis (77%), with some identifying as First Nations (18%), multiple Indigenous responses (1.2%), and non-Indigenous (2.7%) [ 19 ]. Many community members are employed in a traditional manner utilizing resources of the land (e.g., hunting, fishing, trapping), others in a less traditional manner (e.g., lumbering, tourism, wild rice harvesting), and some are employed through the hospital and schools. The community currently has one elementary school with approximately 200 students from preschool to Grade 6, and one high school serving Grades 7–12 with adult educational programming. Île-à-la-Crosse has a regional hospital with Emergency Services, which includes a health services centre with a total of 29 beds. Other infrastructure of the community includes a Royal Canadian Mounted Police (RCMP) station, a village office, volunteer fire brigade, and a catholic church [ 46 ].

Needs assessment approach

Île-à-la-Crosse shared their vision of integrating digital technology and infrastructure as part of its growth, thus the needs assessment was identified as an appropriate method to provide the formative information necessary to understand what the needs are, including who (i.e., players, partners), and what (i.e., information sources) would need to be involved, what opportunities exist to address the needs, and setting priorities for action with key community stakeholders [ 47 ]. As a starting point and rationale for this needs assessment, the community of Île-à-la-Crosse values the potential of technology for improving health communication, information reach, access to resources, and care, and was interested in identifying priorities to begin building digital infrastructure. Given the timing of the COVID-19 pandemic, being responsive to community health needs were key priorities that they wanted to start addressing using a digital platform. This needs assessment facilitated and enabled new conversations around key priorities and next steps.

The evaluation approach was culturally-responsive and included empowerment principles [ 48 – 50 ]. Empowerment evaluation intends to foster self-determination. The empowerment approach [ 50 ] involved community members–represented through the Citizen Scientist Advisory Council–engaging in co-production of the evaluation design and implementation by establishing key objectives for the evaluation, informing evaluation questions, building relevant and culturally responsive indicators, developing focus group guides, leading recruitment and data collection, and interpreting results [ 51 ]. In this way, the approach incorporated local community and Indigenous Knowledges as well as Western knowledge, in a similar approach to Two-Eyed Seeing [ 13 – 15 ]. Using these needs assessment evaluation results, the community will identify emerging needs and potential application issues, and work with the researchers to continue shaping project development and implementation.

Two-Eyed Seeing to embed digital platforms

Two-Eyed Seeing as described by Elder Albert Marshall [ 13 , 14 ], refers to learning to see with the strengths of Indigenous and Western Knowledges. Our engagement and overall approach to working with the community of Île-à-la-Crosse takes a Two-Eyed Seeing lens, from co-conceptualization of solutions, which starts with understanding the needs of the community. All needs are a result of direct Indigenous Knowledge that was provided by the Advisory Council. Indigenous Knowledge is not limited to the knowledge of Elders and Traditional Knowledge Keepers; however, they play a critical role in guiding that knowledge through by providing historical, geographic, and cultural context. Moreover, the Knowledge Keepers can be key decision-makers in the community, and in our case, they were key informants who participated in this needs assessment. Every aspect of needs assessment was dependent on the Advisory Council and Key informants providing the Indigenous Knowledge that the research team needed to tailor digital solutions. As a result, Two-Eyed Seeing approach informed all aspects of the research process.

As we are working to develop, and bring digital platforms and technologies (i.e., Western methods) to address key community priorities, Indigenous Knowledge is central to the overall project. Indigenous Elders, decision-makers, and Advisory Council members are bringing both their historical and lived experience to inform project goals, key priority areas, target groups, and methods. Île-à-la-Crosse is a predominantly Metis community, which differs in culture from other Indigenous communities in Canada—First Nations and Inuit communities. Ceremony is not a key part of community functioning; thus, specific cultural ceremonies were not conducted upon advice of the Advisory Council. Instead, the knowledge of historical issues, challenges, and success stories in the community is considered Indigenous Knowledge for this needs assessment, and more importantly, this Indigenous Knowledge informed the focus areas and next steps for this project. Overall, the spirit of collaboration and co-creation which combined Western research methods/technology with Indigenous Knowledge and expertise is considered Two-Eyed Seeing in this project. This lens was taken at all phases, from the engagement stage to Advisory Council meetings, to planning and executing the needs assessment and next steps.

Data collection

In order to obtain an in-depth understanding of the key priorities and supports within the community of Île-à-la-Crosse, this needs assessment used a qualitative approach. An environmental scan was conducted in February 2020 of current school and community policies and programs. Published reports, meeting memos, community social media accounts, and the Île-à-la-Crosse website were reviewed for existing policies and programs. The Citizen Scientist Advisory Council identified appropriate data sources for the document review and corroborated which programs and initiatives were currently active in the community.

Qualitative data were collected from key decision-makers and other members within the community. A purposeful convenience sampling approach was employed to identify members of the community who could serve on the Council and participate in focus group discussions. Key decision makers and existing Council members recommended other community members who could join the focus group discussions to provide detailed and relevant information on community priorities, digital infrastructure, supports, and challenges. Two focus groups were conducted by members of the research team in Île-à-la-Crosse with the Council in May 2020. Focus group participants were asked to describe community priorities, supports, and barriers, as well as experience and comfort with digital platforms. Each focus group had four participants, were two-hours in length, and followed an unstructured approach. Three key informant interviews were conducted in Île-à-la-Crosse between February and April 2020. One-hour interviews were conducted one-on-one and followed a semi-structured interview format. The focus groups and key informant interviews were led by the study PI, TRK, and Co-Investigator, JB, who have extensive training and experience with qualitative research methods, particularly in partnership with Indigenous communities. Focus groups and key informant interviews were conducted virtually using Zoom [ 52 ]. The key informant interviews and focus groups were audio-recorded and transcribed. All data were aggregated, anonymized, and securely stored in a cloud server. Data are owned by the community. Both the Council and the research team have equal access to the data.

Data analysis

All documents identified through the environmental scan were reviewed for key themes. A list of existing school and community programs was compiled and organized by theme (i.e., education-focused, nutrition-focused, health-focused, etc.). Follow-up conversations with key informants verified the continued planning and provision of these programs.

Following the 6-step method by Braun and Clarke (2006), a thematic analysis was conducted to systematically identify key topic areas and patterns across discussions [ 53 ]. A shortlist of themes was created for the key informant interviews and focus groups, respectively. A manual open coding process was conducted by two reviewers who reached consensus on the final coding manual and themes. Separate analyses were conducted for key informant interviews and focus group discussions; however, findings were synthesized to identify key themes and sub-themes in key priorities for the community, community supports and barriers, as well as digital connectivity and infrastructure needs.

Needs assessment findings

The needs assessment guiding framework informed specific discussions of key issues in the community of Île-à-la-Crosse. Key informant interviews and focus group discussions commenced by asking about priorities–“what are the key areas of focus for the community?” In all conversations–including a document review of initiatives in Île-à-la-Crosse–health was highlighted as a current priority; hence, questions in the guiding framework were tailored to fit a needs assessment focused on community health. The following five overarching evaluation questions were used to guide the evaluation: i) What are the prominent health issues facing residents of Île-à-la-Crosse?; ii) What supports are currently available to help residents address prominent health issues in the community?; iii) What types of barriers do community members face to accessing services to manage their health?; iv) How is health-related information currently shared in the community?; and v) To what extent are health services and information currently managed digitally/electronically? The evaluation questions were kept broad to capture a range of perspectives. An evaluation matrix linking the proposed evaluation questions to their respective sub-questions, indicators, and data collection tools is outlined in Table 1 .

https://doi.org/10.1371/journal.pone.0279282.t001

Feedback on each needs assessment topic area is summarized in the sections below. Sample quotes supporting each of the key topic areas is provided in Table 2 .

https://doi.org/10.1371/journal.pone.0279282.t002

Key priorities

Four priorities were identified through the focus groups, key informant interviews, and document review ( Fig 2 ). Given the timing of the discussion, the primary issue of concern was the COVID-19 pandemic. Many community members were worried about contracting the virus, and the risk it posed to Elders in the community. Of greater concern, however, was how COVID-19 exacerbated many existing health concerns including diabetes and hypertension in the community. For example, routine procedures were postponed and community members with other health conditions were not receiving routine healthcare during the height of the pandemic. The St. Joseph’s Hospital and Health Centre services Île-à-la-Crosse and bordering communities, hence maintaining capacity for COVID-19 patients was a priority. COVID-19 exposed existing barriers in the healthcare system which are described in greater detail in the barriers to community health section.

https://doi.org/10.1371/journal.pone.0279282.g002

Another priority discussed by many community members was climate change and the environment. Community members noted that changes in wildlife patterns, land use, and early winter ice road thaw were areas of concern, particularly due to the impact these factors have on traditional food acquisition practices (i.e., hunting) and food access. For instance, the geographic location of Île-à-la-Crosse is surrounded by a lake, and the main highway which connects the community to the land has experienced increased flooding in the past few years.

In addition to posing immediate danger to community members, food security and sovereignty are also closely linked to road access. While the community produces some of its own food through the local fishery and greenhouses, Île-à-la-Crosse is still dependent on a food supply from the south (i.e., Saskatoon). During COVID-19, food access was further restricted due to limited transport and delivery of food products, which increased the risk of food insecurity for community members. Food insecurity was believed to be of bigger concern for Elders in the community compared to younger members. Younger community members expressed having the ability to source their own food in a variety of capacities (e.g., fishing in the lake), whereas Elders rely more heavily on community resources and support (e.g., grocery stores, friends, and family).

Community members also discussed issues surrounding mental health and wellbeing. This topic was of particular concern for youth and Elders in the community. Community members discussed the importance of identifying covert racism (vs. discrimination) that exists within health services that exacerbated mental health issues and care, as well as developing coping strategies, resilience, and supports to prevent mental health crises. Key informants emphasized the need to minimize the stigma around mental health and focus on holistic wellbeing as they work to develop strategies to improve community wellness.

Community health supports

Île-à-la-Crosse has been working on developing supports to improve community health through various initiatives. A document review identified a community-specific wellness model which has informed program development and planning over the past few years. The key components of the Île-à-la-Crosse wellness model are: i) healthy parenting; ii) healthy youth; iii) healthy communities; iv) Elders; v) healing towards wellness; and vi) food sovereignty. The Elders Lodge in the community provides support for holistic wellbeing by promoting intergenerational knowledge transmission, guidance to youth and community members, as well as land-based activities which improve bonding, cultural awareness, and mental and spiritual well-being among community members. The Elders Lodge hosts both drop-in and organized events.

Several initiatives have been developed to support food sovereignty in the community, including a greenhouse program where fruits and vegetables are grown and shared locally. This program is run in partnership with the school to increase food knowledge and skills among youth. In addition, after-school programs including traditional food education (i.e., cooking classes) and land-based activities (i.e., berry picking) led by Elders support the goals of the wellness model. The community is currently working on developing additional programs dedicated to improving mental wellness among adults, youth, and Elders.

Barriers to community health

When key informants were asked to identify barriers to community health, they described delays in access to timely health information. For example, daily COVID-19 tests conducted at the regional health centre in Île-à-la-Crosse were relayed to the provincial health authority; however, information about the total number of COVID-19 cases could take up to one week to be sent back to the community. This time lag restricted community decision-makers’ ability to enact timely policy (i.e., contact tracing) and rapidly respond to managing cases.

A second barrier that was raised by community members was a delay in access to timely healthcare. The Île-à-la-Crosse hospital is a regional health service centre serving the community as well as surrounding areas. Community members noted that the load often exceeded the capacity of the single hospital, and some patients and procedures were relocated to hospitals and clinics in the larger city of Saskatoon, Saskatchewan. This was reported to be challenging for many community members as it was associated with longer wait times, long commutes, and sometimes required time off work. Many of these challenges were exacerbated during the COVID-19 pandemic. As a result of the pandemic, many medical centres and hospitals postponed routine and elective medical procedures in an attempt to accommodate the overwhelming influx of patients who contracted COVID-19. In addition, community members were advised to avoid spending time in health centres to limit risk of exposure to the virus. These COVID-related changes further delayed access to timely healthcare for many community members of Île-à-la-Crosse.

Several community members reported experiencing institutional racism in healthcare and social service settings outside of Île-à-la-Crosse. This was particularly exacerbated during the COVID-19 movement restrictions, where community members faced significant difficulties in accessing services and care in larger urban centres, and experienced further discrimination due to the stigma of COVID-19-related rumours about communities in the north.

Lastly, community members discussed a lack of awareness about some health topics, including where and how to access reliable health information. Some community members attributed this lack of awareness to a general distrust in government health information due to a history of colonialism and exploitation in Canada, which likely contributed to increased misinformation about COVID-19 risk and spread.

Health communication

The primary modes of communication within Île-à-la-Crosse are radio and social media. These platforms were used throughout the pandemic to communicate health information about COVID-19 case counts and trends. Community members also reported obtaining health information from healthcare practitioners (i.e., for those already visiting a healthcare provider), Elders, and the internet. Key informants indicated an interest in improving digital infrastructure to enable sharing of timely and accurate health information with community members and minimize misinformation. Key informants also reported room for improvement in the community’s digital health infrastructure, particularly in improving timely communication with community members, and to inform decision-making in crisis situations.

Digital infrastructure and connectivity

Île-à-la-Crosse has its own cell tower which offers reliable access to cellular data. The community also has access to internet via the provincial internet provider–SaskTel, as well as a local internet provider—Île-à-la-Crosse Communications Society Inc. Key informants and community members confirmed that most individuals above 13 years of age have access to smartphones, and that these mobile devices are the primary mode of internet access. However, it was unclear whether everyone who owns smartphones also has consistent data plans or home internet connections. Key informants described the great potential of digital devices like smartphones to increase the speed and accuracy of information sharing. Discussions with both key informants and community members suggested the need for a community-specific app or platform which could provide timely health information that was tailored to the community’s needs.

Community members noted that expanding digital infrastructure had to be paired with efforts to improve digital literacy–particularly as it relates to data security, privacy, and online misinformation. A separate initiative was discussed which could work to improve digital literacy among youth and Elders, as this would improve both the uptake of digital health platforms, as well as their usefulness and application. Key informants discussed the importance of building digital infrastructure that would enable data sovereignty, self-governance, and determination. The key informants, who are also primary decision-makers in the community, described opportunities for ethical development of digital platforms that would ensure that data is owned by the community.

Needs assessments are commonly the first step in understanding specific community needs, [ 27 , 28 ]; however, few evaluation frameworks provide practical guidance on how to engage communities in needs assessments [ 41 ]. This paper provides a step-by-step guide for conducting needs assessments in collaboration with communities in the digital age. Using the series of questions outlined in the Guiding Framework, researchers and evaluators can gain an in-depth understanding of a community’s priorities, needs, existing capacity, and relevant solutions.

The Guiding Framework was critical to establishing a partnership with the community of Île-à-la-Crosse, as it enabled the research team to obtain detailed insight into their priorities–in this case, community health–as well as community capacity. Taking a Two-Eyed Seeing approach [ 15 ], conversations with the community highlighted strengths of Western digital technology and the diversity of Indigenous Knowledges for addressing priorities [ 13 ]. This approach was also important to establishing trust and respect for the variety of perspectives that could be used to address community priorities. The resulting partnership also enabled the conceptualization of tangible action items that were aligned with current and future priorities–a key factor in the sustainability and feasibility of community-based initiatives [ 4 – 8 , 54 ].

Challenges and opportunities for using digital platforms for priorities identified by needs assessment

Many rural and remote communities face similar challenges and share common priorities with Île-à-la-Crosse. For example, resource and service access, including food and other essential supplies, healthcare, and internet connectivity are issues faced by many rural and remote communities across Canada [ 55 – 60 ]. Key informants and community members from our partner community corroborated these access issues, particularly in relation to public health. Given the potential for digital technology to bridge access gaps, it has become pertinent to invest in digital infrastructure and platform development.

Research has shown that in many rural and remote communities, smartphone ownership is not the limiting factor–it is internet inequity, which is defined as differential internet access based on wealth, location (urban, rural, or remote), gender, age, or ethnicity [ 61 ]. The United Nations has declared internet access a human right [ 10 ], which makes it imperative to develop digital infrastructure such as internet connectivity to improve digital accessibility. Île-à-la-Crosse has its own cell tower which offers reliable access to cellular data. The community of Île-à-la-Crosse also has access to consistent and dedicated internet service through a provincial internet provider and local internet provider. The needs assessment showed that the universality of smartphone ownership combined with good internet connectivity lays the foundation for the development of tailored, culturally appropriate digital health platforms in communities like Île-à-la-Crosse.

In particular, the needs assessment revealed that smartphone apps, which most citizens are well-versed with, can be used to provide local services and access to resources. For example, a locally developed app can connect the Mayor’s office with community members in real-time to provide updates on COVID-19 outbreaks. Apps also have the potential to connect communities to resources within and outside of the community [ 35 , 57 ]. For example, advanced artificial intelligence algorithms can be used to anticipate community needs prior to urgent crises like COVID-19, environmental disasters, or food crises [ 35 , 62 – 65 ]. To date, the issue has not been the lack of technology or ability to bridge this gap for rural and remote communities. Instead, larger systemic inequities have limited our ability to co-create local solutions for global problems by decentralizing technology that is widely available [ 35 , 66 ], which highlights upstream inequities in developing digital platforms.

Recommendations for inclusive digital needs assessments

Given the widespread adoption of digital technology, digital platforms can provide rich data to identify and address community crises [ 2 , 3 , 35 ]. Importantly, co-created digital platforms can be used to share knowledge in real-time with community members and other stakeholders to enable remote engagement, which is especially important during crisis situations such as a pandemic [ 2 , 3 , 35 ]. As we implement creative digital platforms in varied programs or research projects, we must also integrate this digital perspective into the evaluation process. Research and evaluation literature has well established approaches to needs assessment evaluations [ 29 , 42 , 67 ]; however, in the 21st century, we need to account for the use and application of digital platforms in community-focused initiatives. To identify how and where digital platforms can play a role in addressing community priorities, we propose several recommendations for inclusive community-based needs assessments.

First, at the crux of all community-based needs assessments is relationships. A relationship built on respect, reciprocity, mutual understanding, and prioritizing the needs and vision of communities is essential for sustainable impact. The First Nations OCAP® principles [ 68 ] informed conversations between the research team and community about data ownership and control. These principles include ownership of knowledge and data, control over all aspects of research, access to information about one’s own community, and possession or control of data [ 68 ]. The OCAP® principles ensure First Nations and other Indigenous Peoples the right to their own information, and also reflect commitments to use and share information in a way that maximizes the benefit to a community, while minimizing harm. Some communities may choose to lead a project, or work closely in collaboration with experts for specific projects. Irrespective of the project dynamics, needs assessments rely on detailed information and context about a community for a project to succeed.

Second, it is important for researchers and evaluators to gain an understanding of the current digital infrastructure and connectivity in the community. The needs assessment framework ( Fig 1 ) includes relevant questions for identifying data and WIFI access in a community, penetration of digital devices, and existing digital infrastructure. Even for community-based initiatives that are not focused on a digital platforms, digital technologies will inevitably be a part of the solution, a barrier, or both. Hence the digital landscape has become part of the context that we must capture and understand in a needs assessment to better design and develop programming, policies, and other initiatives.

Third, it is important to ask the question of where and how a digital tool or platform could help. Are there gaps that digital platforms can help address or fill? In rural and remote communities, in particular, digital platforms can provide access to real-time information and services not otherwise available. For example, Telehealth [ 69 , 70 ] in the Canadian north offers citizens access to essential healthcare services, including video appointments with medical specialists. Prior to Telehealth, many residents would need to fly into bigger cities in the nearest province to access health care [ 55 ].

Lastly, an understanding of the broader context which affects a community’s ability to adopt digital platforms is critical to the success of digital initiatives. This includes, but is not limited to, capturing data on socioeconomic status and the accessibility of internet-connected digital devices. Digital platforms should help to bridge the divide in resource, service, and information access–not widen the gap. For some communities, this may require working on building digital infrastructure and obtaining dedicated funds to expand access prior to implementing digital initiatives. In addition, digital literacy cannot be taken for granted. Digital literacy refers to individuals’ ability to not only use digital devices, but according to Eshet-Alkalai [ 71 ], “includes a large variety of complex cognitive, motor, sociological, and emotional skills, which users need in order to function effectively in digital environments.” In its simplest form, digital literacy may include the ability to navigate digital platforms, download apps, and communicate electronically. Other more specific skills include ability to read and understand instructions, terms and services, as well as data privacy and security statements [ 72 – 74 ] As part of a needs assessment, identifying digital literacy within a community is an important step to safe, ethical, and relevant digital tool development.

Considering the challenges, immense potential, and learnings from applying the Guiding Framework, a tailored digital platform was conceptualized called Sakitawak Health.

Development of Sakitawak Health

Sakitawak Health is a culturally-responsive digital epidemiological platform to monitor, mitigate, and manage COVID-19 outbreaks. The needs assessment concluded that digital platforms can be used for emerging or other existing population health crises within Île-à-la-Crosse and potentially other Indigenous communities. Moreover, to co-create digital platforms, the Île-à-la-Crosse Citizen Scientist Advisory Council identified key features to embed in CO-Away, including free virtual care for citizens via a smartphone app at the frontend, and access to anonymized community data on the backend for decision-makers.

The app will provide three key precision medicine services that are specific to each citizen: 1) continuous risk assessment of COVID-19 infection; 2) evidence-based public health communication; and 3) citizen reporting of food availability, access to public services, and COVID-19 symptoms and test results. These culturally-responsive features have been co-created with Métis decision-makers in Île-à-la-Crosse based on imminent community needs and preferences. CO-Away will enable real-time data collection through continuous citizen engagement to inform municipal jurisdictional policies.

There are three guiding principles for developing Sakitawak Health: I) Citizen empowerment and data ownership: Active engagement is enabled through app features such as visualizing community risk. More importantly, the community owns the data to ensure data sovereignty; II) Privacy: Utilizing a cutting-edge methodology called federated machine learning, we will develop artificial intelligence algorithms that stores sensitive data such as participant location on mobile devices itself (i.e., sensitive data are not stored in external servers); III) Security and scalability: The backend server will be located in Cloud in Canada, which allows for horizontal and vertical scalability (i.e., the potential for developing multiple frontend apps and decision-making dashboards).

Recognizing the importance of data sovereignty and Indigenous self-governance

Data sovereignty and social justice are important aspects of community-based work, particularly for communities that have experienced discrimination or systemic inequities [ 2 , 75 ]. Data sovereignty refers to meaningful control and ownership of one’s data [ 76 ]. For Indigenous communities in Canada, self-determination and self-governance are of paramount importance given the colonial history of oppression, trauma, and disenfranchisement [ 77 ], and data sovereignty and ownership of digital platforms can promote that independence. In conducting digital community-based needs assessments, the application of a Two-Eyed Seeing lens enables us to leverage strengths of both Indigenous and Western Ways of Knowing to help focus on key priorities and develop solutions.

The engagement and overall approach to working with the community of Île-à-la-Crosse applied a Two-Eyed Seeing lens. In the needs assessment with Île-à-la-Crosse, Two-Eyed Seeing involved incorporation of Métis Knowledge during team engagements, which ensured that any digital platforms developed would incorporate Indigenous Knowledge to promote data sovereignty. All priorities identified within this manuscript are a result of direct Indigenous Knowledge that was provided by the Council. Indigenous Knowledge is not limited to the knowledge of Elders and Traditional Knowledge Keepers; however, they play a critical role in guiding that knowledge through by providing historical, geographic, and cultural context. Discussions with Île-à-la-Crosse about data sovereignty centered around citizen ownership of data, community access, and ensuring data privacy and security. The ultimate goal of this approach to data sovereignty is to facilitate decreased dependence on external systems and use digital solutions for Indigenous self-determination and self-governance.

The needs assessment represents the first phase of a larger evaluation strategy to develop and implement culturally appropriate digital platforms for community health. Phase 1 involved identifying core health priorities and desired supports in the community of Île-à-la-Crosse. Based on the needs assessment findings, Phase 2 of this project will involve the development of tailored digital health platforms and programming to support digital literacy. As part of Phase 2, digital literacy programs and tailored digital health platforms will be pilot tested and adapted prior to their implementation. In Phase 3, a process evaluation will be conducted to assess the reach, uptake, and use of digital health platforms and digital literacy programming. Integrated knowledge translation will be conducted during all phases to ensure continuous feedback, communication, and knowledge sharing with all relevant stakeholder groups.

Conclusions

Needs assessments can facilitate important conversations in community-based research and evaluation to learn about key priorities, challenges, and opportunities for growth. The Guiding Framework for Community-Based Needs Assessments to Embed Digital Platforms details a step-by-step approach to begin a conversation with communities to better understand their needs, and to tailor research and evaluation projects focused on embedding digital platforms. In Île-à-la-Crosse, the needs assessment framework has propelled the launch of a timely, community-engaged digital initiative to address key priorities, starting with COVID-19. Overall, tailored platforms can help bridge existing gaps in resource, program, and service access in Indigenous communities, irrespective of their location across the world.

Supporting information

https://doi.org/10.1371/journal.pone.0279282.s001

Acknowledgments

The authors would like to acknowledge the contributions of community members of Île-à-la-Crosse. The Elders, youth, and key decision-makers who are part of the Île-à-la-Crosse Citizen Scientist Advisory Council have been invaluable in providing support, guidance, and cultural training to the research team. The authors also acknowledge the support of the Canadian Internet Registration Authority in advancing the uptake of digital health applications.

• View Article
• Google Scholar
• 2. Smith LT. Decolonizing Methodologies: Research and Indigenous Peoples. London; New York; Dunedin: Zed Books Ltd; University of Otago Press; Distributed in the USA exclusively by St Martin’s Press; 2021. https://doi.org/10.4324/9781315597843-10 .
• 7. Narayan D. Empowerment and Poverty Reduction: A Sourcebook. Washington, D.C.; The World Bank; 2002.
• 8. Cairncross L, Morrell C, Drake J, Brownhill S. Tenants Managing: An Evaluation of Tenant Management Organisations in England. London: Office of the Deputy Prime Minister; 2002.
• 9. Pratchett L, Durose C, Lowndes V, Smith G, Stoker G, Wales C. Empowering communities to influence local decision making: systematic review of the evidence. London: 2009.
• 10. La Rue F. Report of the Special Rapporteur on the Promotion and Protection of the Right to Freedom of Opinion and Expression, Frank La Rue: addendum. 2011.
• PubMed/NCBI
• 19. Statistics Canada. Île-à-la-Crosse, NV [Census subdivision], Saskatchewan and Saskatchewan [Province] (table). Ottawa: 2017. https://doi.org/Statistics Canada Catalogue no. 98-316-X2016001.
• 20. Waldram JB, Herring A, Young TK. Aboriginal Health in Canada: Historical, Cultural, and Epidemiological Perspectives. 2nd editio. Toronto: University of Toronto Press; n.d.
• 22. Central Urban Métis Federation Inc., Kinistin Saulteaux Nation, Saskatoon Health Region. Strengthening the Circle: Partnering for Improved Health for Aboriginal People. Saskatoon, SK: 2010.
• 28. Pavlish CP, Pharris MD. Community-based collaborative action research : a nursing approach. Sudbury, MA: Jones & Bartlett Learning; 2012.
• 42. Stufflebeam DL, Shinkfield AJ. Evaluation theory, models, and applications. 1st ed. San Francisco, CA: Jossey-Bass; 2007.
• 43. McLennan D. Sakitawak Development’s Facebook Page n.d.
• 44. Cameco Corp. Patuanak—English River First Nation—Community Profiles—Community. Cameco North Saskatchewan 2020.
• 45. McLennan D. The Encyclopedia of Saskatchewan: Île-à-la-Crosse n.d. https://esask.uregina.ca/entry/ile-a-la-crosse.jsp .
• 52. Zoom Video Communications Inc. Security guide 2016.
• 68. The First Nations Information Governance Centre. The First Nations Principles of OCAP TM Training Course 2021:1–6. https://fnigc.ca/ocap-training .
• 69. Telehealth 2021. North Heal 2021. https://www.northernhealth.ca/services/digital-health/telehealth .
• 75. Kukutai T, Taylor J, editors. Indigenous Data Sovereignty: Toward an Agenda. The Australian National University Press; 2016. https://doi.org/10.1109/IPDPSW.2014.214 .

• Request Info
• Departments
• Community Service
• Patient Care
• Give to SLU Medicine
• Search & Directory

Department of Health and Clinical Outcomes Research

Assessment and analysis are vital to establishing an equitable, efficient and affordable health care system. At the Saint Louis University Department of Health and Clinical Outcomes Research, we conduct — and translate — research into practices that improve health outcomes across patient populations.

Developing national cancer survivorship standards to inform quality of care in the United States using a consensus approach

• Open access
• Published: 13 May 2024

Cite this article

You have full access to this open access article

• Michelle A. Mollica 1 ,
• Gina McWhirter 2 ,
• Emily Tonorezos 1 ,
• Joshua Fenderson 3 , 4 ,
• David R. Freyer 5 , 6 ,
• Michael Jefford 7 , 8 , 9 ,
• Christopher J. Luevano 10 ,
• Timothy Mullett 11 ,
• Shelley Fuld Nasso 12 ,
• Ethan Schilling 13 ,
• Vida Almario Passero 14 , 15 , 16 &

the National Cancer Survivorship Standards Subject Matter Expert Group

717 Accesses

27 Altmetric

Explore all metrics

To develop United States (US) standards for survivorship care that informs (1) essential health system policy and process components and (2) evaluation of the quality of survivorship care.

The National Cancer Institute and the Department of Veterans Affairs led a review to identify indicators of quality cancer survivorship care in the domains of health system policy, process, and evaluation/assessment. A series of three virtual consensus meetings with survivorship care and research experts and advocates was conducted to rate the importance of the indicators and refine the top indicators. The final set of standards was developed, including ten indicators in each domain.

Prioritized items were survivor-focused, including processes to both assess and manage physical, psychological, and social issues, and evaluation of patient outcomes and experiences. Specific indicators focused on developing a business model for sustaining survivorship care and collecting relevant business metrics (e.g., healthcare utilization, downstream revenue) to show value of survivorship care to health systems.

Conclusions

The National Standards for Cancer Survivorship Care can be used by health systems to guide development of new survivorship care programs or services or to assess alignment and enhance services in existing survivorship programs. Given the variety of settings providing care to survivors, it is necessary for health systems to adapt these standards based on factors including age-specific needs, cancer types, treatments received, and health system resources.

Implications for Cancer Survivors

With over 18 million cancer survivors in the United States, many of whom experience varied symptoms and unmet needs, it is essential for health systems to have a comprehensive strategy to provide ongoing care. The US National Standards for Survivorship Care should serve as a blueprint for what survivors and their families can anticipate after a cancer diagnosis to address their needs.

Similar content being viewed by others

Survivorship objectives in comprehensive cancer control plans: a systematic review

A descriptive study of policy and system-level interventions to address cancer survivorship issues across six United States health systems

Survivorship care plan outcomes for primary care physicians, cancer survivors, and systems: a scoping review.

Avoid common mistakes on your manuscript.

Introduction

A cancer survivor is any individual from the point of diagnosis through the balance of life [ 1 ]. There are over 18 million cancer survivors in the United States [ 2 ], and with advances in diagnostic and treatment capabilities and the aging population, this number is expected to grow. People with cancer have unique survivorship needs, including physical and psychological symptoms both during and after their treatment, risk for recurrence and subsequent cancers, and social needs. As a result, most survivors require long-term follow-up care.

Survivorship care is multifaceted, and recommendations have included surveillance for recurrence and new cancers, prevention and management of physical and psychosocial symptoms, and promoting healthy behaviors [ 3 , 4 ]. While survivorship guidelines exist [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ], the delivery of survivorship care, including what care is delivered, to whom it is delivered, and who delivers the care, varies greatly based on factors including care setting, geographical area, and personal resources. Survivorship care is often fragmented, depending on survivors to seek care from multiple providers without a coordinated system. This is further exacerbated by differing philosophies concerning when survivorship care should be delivered (e.g., post-treatment for those treated with curative intent versus post-diagnosis for anyone with a cancer diagnosis). Survivorship care for many people in the United States is suboptimal, leaving survivors with persistent symptoms, unmet needs, and lack of access to comprehensive care.

There have been several previous efforts to define survivorship care. The LIVESTRONG Essential Elements of Survivorship Care were developed in 2011 with the goal of building consensus in the survivorship community around how best to address the needs of post-treatment survivors [ 14 ]. The American College of Surgeons’ Commission on Cancer (CoC) Survivorship Standard 4.8, recently updated in 2019, defined requirements for CoC accredited programs [ 15 ]. The updated survivorship standards require a survivorship coordinator, a survivorship program documenting a minimum of three services offered each year to support patients, and a focus on enhancing existing and developing new services. This revised standard was an update from the 2016 survivorship standard that required documentation of a survivorship care plan for patients with early-stage cancer treated with curative intent [ 16 ]. In addition, the Quality of Survivorship Care Framework was developed to define the key components of quality survivorship care that are applicable to diverse populations of adult cancer survivors and was intended to inform clinical care, research, and policy [ 3 ].

Given that people with cancer are treated in diverse settings, including cancer centers, academic medical centers, and community sites, there is a need for developing a comprehensive set of national standards for health systems to provide quality survivorship care. The overall goal of this project was to build upon existing efforts to develop national standards for survivorship care that can be utilized by all healthcare systems to assess the quality of existing survivorship care and guide the development of new programs and services. Standards of care represent recommendations for health systems that apply to the patients they serve. Specifically, we sought to define standards for (1) essential health system policy and process components of survivorship care programs and (2) the evaluation of the quality of survivorship care.

The Biden Cancer Moonshot, President Biden’s whole of government response to accelerate progress against cancer and end cancer as we know it, established a goal to develop standards for survivorship care. This project was led by the National Cancer Institute and United States Department of Veterans Affairs, in collaboration with several other Health and Human Services Agencies. Methods were adapted from a previous effort in Australia, where an online modified reactive Delphi survey was completed, followed by a consensus meeting of survivorship experts to inform the Victorian Quality Cancer Survivorship Framework [ 17 ].

Key definitions

For the purpose of this project, we defined a cancer survivor as any individual from the time of diagnosis through the balance of life, diagnosed at any age or stage. We also adapted definitions from Lisy et al. for health system policy, health system process, and evaluation/assessment [ 17 ]. Health system policies were defined as principles and procedures guiding an organization’s capacity and structure to provide survivorship care; health system processes were an organization’s capacity to deliver care through its embedded practices and procedures; and evaluation/assessment were how to measure the impacts of survivorship care within an organization.

Identification of possible indicators

A list of potential indicators in the three domains of health system policy, process, and evaluation/assessment were identified through a review of survivorship and cancer-specific guidelines, the CoC survivorship standard [ 15 ], existing survivorship quality frameworks [ 3 , 17 ], US cancer control plans [ 18 ], and relevant literature. These resources were gathered based on the recommendations of the Task Force and subject matter experts.

Subject matter expert consensus meetings

In 2023, three virtual meetings with survivorship subject matter experts were held to prioritize the most important and feasible indicators to include in the standards. The three meetings were iterative and invited subject matter experts included leading national and international experts in clinical survivorship care, survivorship research, implementation science, health policy, and survivor advocates. Subject matter experts were chosen based on their knowledge of the evidence related to survivorship care and/or their experience in providing care, informing health policy, and/or conducting survivorship care delivery research. We utilized a snowball approach to identify experts and accepted additional recommendations from invited experts, with the overall goal of collectively representing diverse perspectives and experiences related to survivorship. A total of 35 experts participated in the meetings. Additionally, these meetings were open for public viewing and attendees were able to submit comments and questions for consideration and comment.

Meeting 1 focused on providing background to the project, an open discussion among the experts, and individual polling where experts rated the importance of each possible survivorship indicator and identified other indicators for consideration in the next round. Importance was defined using the definition from Lisy and colleagues, as “a core component in achieving survivorship care and can be used to measure the quality of survivorship care” [ 17 ]. For the first meeting, experts were asked only to consider the importance of each indicator rather than also considering the feasibility of implementing and collecting this information. Experts could also suggest edits to the indicators. Questions from Meetings 1–3 can be found in the supplementary information (Appendix A ).

Responses from the Meeting 1 poll were aggregated to identify those rated most important and those rated least important. Based on those results and suggestions from experts and public viewers on edits and additional indicators, an updated list of 15–20 indicators in each domain (policy, process, evaluation/assessment) was developed. Meeting 2 was then held one week later, where results from Meeting 1 were shared, including the indicators rated most important and those rated least important. Following was an open discussion of the results among the experts, including a discussion of feasibility. Experts were then asked to select the top 10 most important and feasible (to implement and/or collect) indicators within each domain; they could also suggest edits to the indicators.

Responses from Meeting 2 were then aggregated to identify the top 10 rated most important and feasible indicators in each domain. Results were shared with experts during Meeting 3, followed by an open discussion of the results. A final poll was conducted where experts were asked to suggest edits to the top 10 indicators in domain and to identify indicators that did not make the top 10 but should be considered for inclusion in the final standards.

Based on suggested edits and additions during the Meeting 3 poll and through refinement by the co-chairs, a final set of standards was developed that includes 10 indicators in health system policy, processes, and evaluation/assessment.

Meeting 1 results

The poll for Meeting 1 included 18 indicators for health system policy, 33 indicators for processes, and 20 indicators for evaluation/assessment. Based on polling results, the policy indicators rated highest importance were a policy requiring establishment of a survivorship program, outlining a team of multidisciplinary health professionals included in the survivorship program, collection of data on survivors’ experience of care and patient-reported outcomes, stratifying survivors to appropriate models of care, provision of support services to survivors based on needs, consideration of approach and timing of transitions in survivorship care, training for healthcare providers, and designation of an organizational survivorship care leader. The policy indicators rated lowest importance were a policy for documenting survivorship care reporting requirements to a government agency, public reporting and dissemination of survivorship outcomes, documenting a minimum of three services offered each year to support patients and survivors, and providing access to prescription produce programs using existing systems/programs. The process indicators rated highest importance were assessment of emotional and psychological effects of cancer and its treatment, physical effects during and following cancer treatment, risk of recurrence or new cancers, practical and social effects (e.g., financial challenges), lifestyle behaviors, and provided with treatment, referrals, and advice to manage physical, emotional, and social effects. The process indicators rated lowest importance were providing the opportunity for participating in research including clinical trials, providing support or referrals for other medical or chronic conditions that are non-cancer related, providing access to advice on vaccinations, providing a meeting to plan survivorship care at the time of diagnosis, providing medically tailored food and nutrition services, providing information and access to complementary health services to support overall health and well-being, and providing a consultation with palliative care. For evaluation , indicators rated highest importance were survivors’ patient-reported outcomes, quality of life, patient-reported experiences of care, return to work, and functional capacity. The evaluation indicators rated lowest importance were overall cost of survivorship care to the health system, number of survivors provided with a survivorship care plan, health professionals’ view of survivorship care, survivors’ hospital admissions, number of referrals made for survivors, and number of primary care providers who are sent a survivorship care plan.

Meeting 2 results

The poll for Meeting 2 included the top 15 indicators for health system policy, 20 indicators for processes, and 20 indicators for evaluation/assessment. The indicators that were ranked in the top 10 for policy included a policy that requires establishment or existence of a survivorship program either on-site or by referral; that describes framework for the provision of survivorship care informed by relevant survivorship guidelines (e.g., ASCO, NCCN, ACS), on stratifying survivors to appropriate models of care; that designates an organizational survivorship care lead who evaluates compliance with standards, has senior role in healthcare system, and includes succession plan for the role, outlining team of multidisciplinary health professionals included in survivorship program; and that considers approach and timing of transitions in survivorship care (e.g., pediatric to adult, acute to primary care, oncology team to survivorship team), for the provision of support services to survivors with special needs and from diverse backgrounds (e.g., navigators, interpreters), for training healthcare providers to deliver survivorship care, for collection of data on survivors’ experience of survivorship care and patient-reported outcomes, and for outlining business case/plan with funding allocated for survivorship care (to include budget). Policy indicators not ranked in the top 10 included outlining the provision of needs assessment tools for survivors at certain time points post-treatment; requiring survivorship-focused information available in other languages or different formats for low-literacy readers; outlining the role of survivors in design, evaluation, and reporting of progress; documenting survivorship care reporting requirements to relevant organizational executive committee; and collecting data on caregivers’ experiences of survivorship care.

For processes , the indicators that were ranked in the top 10 were that cancer survivors were provided access to a survivorship program which addresses the needs of cancer survivors either on-site or by referral; assessed for physical effects during and following cancer treatment, including monitoring for late effects and chronic conditions, and provided with treatment and/or referrals; assessed for emotional and psychological effects of cancer and its treatment and provided with treatment and/or referrals; assessed for practical and social effects of cancer and its treatment (e.g., relationship difficulties, financial challenges, education and employment/return to work) and provided with resources and/or referrals, provided with recommendations regarding surveillance for recurrent or new cancers; assessed for their risk of recurrence or new cancers, including family history and genetic testing; assessed for lifestyle behaviors with recommended management and/or provided with appropriate referral (e.g., smoking cessation, promoting physical activity); provided with access to allied health services (e.g., nutrition, physical therapy, sexual health, fertility services, rehabilitation, dental and podiatry services); provided with access to specialty care services to manage potential late effects (e.g., cardiology); assessed for financial hardship/toxicity and provided with resources and support; and provided with care planning conversations including coordination of care with primary care provider and/or other multidisciplinary health professionals involved in their care. The process indicators not ranked in the top 10 were providing care consistent with their goals, providing access to care to manage fertility and reproductive concerns, providing access to age-specific survivorship care, providing access to primary care services, providing access to age- and gender-appropriate cancer screening or referrals to appropriate screening services, and providing access to tobacco cessation services.

In the domain of evaluation/assessment, indicators ranked in the top 10 were survivors’ and caregivers’ patient-reported outcomes, including quality of life, functional capacity, survival rates (1, 5, and 10 years), experiences of care, return to work, rate of recurrence and new cancers, number and characteristics of survivors lost to follow-up, number of survivors with subsequent chronic condition, rate of survivor service referrals and completions, and relevant business metrics to show return on investment of survivorship care to the healthcare system. Evaluation indicators that were not ranked in the top 10 were collecting data on the number of health professionals trained to provide survivorship care, the number of survivors who have their needs assessed at certain times post-treatment, overall cost of care to survivors and caregivers, survivors’ emergency care and urgent care utilization, number of survivors stratified to different models of care, and oncology providers’ view of the role of nurses and advanced practice providers in survivorship care.

Meeting 3 results

The poll for Meeting 3 included an updated list of the top 10 highest-rated indicators in each domain, as well as a list of the indicators that did not make the top 10. Experts were asked to suggest edits, including suggestions to combine indicators, as well as identify lower-rated indicators to consider for inclusion. Lower-rated indicators that were identified by at least 40% of experts to consider for inclusion were then either combined with other indicators or added to the final set. For health system policy, this included a policy outlining the role of survivors in design, evaluation, and reporting of progress. For processes, this included providing care that is consistent with goals, and consideration of age-specific care. For evaluation/assessment, this included the number of health professionals trained to provide survivorship care.

Final cancer survivorship standards

The National Cancer Standards for Survivorship Care are presented below and in Table  1 and include the top 10 indicators in each of the three domains of health system policy, process, and evaluation/assessment.

Health system policy

The organization has a policy that includes:

Establishment or existence of a survivorship program either on-site, through telehealth, or by referral

A framework for the provision of survivorship care informed by survivor stakeholders and relevant survivorship guidelines (e.g., American Society of Clinical Oncology, National Comprehensive Cancer Network, Children’s Oncology Group)

A description of multidisciplinary care, including each team member’s specific roles and responsibilities and workflow(s) for referrals to team members

An overview of how to stratify and refer survivors to appropriate models of care based on age, treatments, and risk factors

Description of the approach and timing of transitions in survivorship care and shared care (e.g., pediatric to adult providers and settings, oncology team to survivorship team and/or primary care) and efforts to prevent/mitigate loss to follow-up care

An outline for the provision of information for support services (e.g., navigators, social work, interpreters) for survivors based on their needs (including but not limited to health, insurance, and financial literacy, disability status), including survivors from diverse and underserved backgrounds

Identification of an executive-level survivorship care lead (with succession plan) whose role is to ensure compliance with standards, with reporting to an appropriate executive committee

Collection of longitudinal data on survivors’ experience of survivorship care and patient-reported outcomes

Requirements and methods for training healthcare providers (either on-site or through an external training program) to deliver survivorship care within their scope of practice

A business case/plan, including budget, with funding allocated for survivorship care

Health system processes

Cancer survivors are…

Provided with access and referral to a survivorship program that addresses the needs of cancer survivors either on-site, through telehealth, or by referral

Assessed at multiple points in their follow-up care for physical effects during and following cancer treatment, including monitoring for late effects and chronic conditions, and provided with treatment and/or referrals

Provided with access to appropriate specialty care services to manage potential late effects (e.g., cardiovascular issues) either on-site, through telehealth, or by referral

Assessed at multiple points in their follow-up care for emotional and psychological effects of cancer and its treatment and provided with treatment and/or referrals

Assessed for practical and social effects of cancer and its treatment (e.g., social risks, health-related social needs, education and employment/return to work or school) and provided with resources and/or referrals

Assessed for their risk of recurrence or new cancers, including family history and genetic testing, and provided with recommendations and referrals regarding surveillance for recurrence or new cancers

Assessed for lifestyle behaviors and provided with recommended strategies for management and appropriate referrals or education as needed (e.g., smoking cessation, diet/nutrition counseling, promoting physical activity)

Provided with access and referrals to appropriate supportive health services (e.g., nutrition, occupational and physical therapy, rehabilitation, sexual health, fertility services, dental and podiatry services)

Assessed for financial hardship/toxicity and concerns regarding insurance coverage, and provided with resources and support as needed

Engaged in the care planning process including discussion of shared goals of care, advanced care planning, and coordination of care with providers and services (e.g., primary care provider, other health professionals, and community-based services) as needed

Health system evaluation/assessment

The organization has a process to collect data on…

Survivors’ patient-reported outcomes, including quality of life, and experiences of survivorship care

Survivors’ functional capacity

Survivors' return to previous participation in paid and unpaid work/ school/ productive activities of living

Survival rates (1, 5, and 10 years) from the time of diagnosis

Rate of recurrence

Rate of subsequent cancers

Number and relevant characteristics (demographics, clinical factors) of survivors lost to follow-up

Caregivers’ experiences and unmet needs

Number of health professionals trained to provide survivorship care

Relevant business metrics to show return on investment of survivorship care to the healthcare system (e.g., healthcare utilization, rate of referrals and completion, downstream revenue)

Efforts to advance survivorship care have largely been focused on development of evidence-based guidelines and defining the key components of quality survivorship care. Survivorship care services vary greatly among cancer centers and in the community [ 19 ]. Given the growing population of survivors treated in a variety of care settings, it is essential to define a standard for health systems to care for survivors. This current effort aimed to address this gap by developing national standards to define and prioritize key health system policy, process, and evaluation/assessment indicators. While evidence-based guidelines inform provider practices [ 5 ], and the Nekhlyudov framework identifies key components to survivorship care [ 3 ], the standards presented herein build on this previous work. They are intended to be utilized to assess survivorship programs within a health system or organization to address the comprehensive needs of cancer survivors during and after treatment.

The methods for this project were adapted from the Victorian Quality Cancer Survivorship Framework [ 17 ], and the resulting indicators differed in several ways. First, consensus meeting discussions included the need to de-emphasize survivorship care plan documentation, given limited evidence on improving survivor outcomes [ 20 ]. In addition, these standards incorporate all modalities to offer survivorship care services, including telehealth. Experts also identified the need to emphasize support for care transitions across the continuum from diagnosis forward, to include a policy requiring training of healthcare professionals to deliver survivorship care, and subsequently to assess the number of providers trained. Though it was recognized that assessment of survival would be challenging, experts also recognized the need to include and aspire to collect long-term survival data after diagnosis with cancer (1, 5, and 10 years). Finally, US standards include a policy to develop a business case/plan with funding allocated for survivorship care, as well as relevant business metrics to show return on investment for survivorship care. Experts in the consensus meeting stressed the need for a sustainable business model for delivering survivorship care services that is evaluated longitudinally using appropriate metrics including (but not limited to) overall healthcare utilization, rate of referrals and completion, and downstream revenue to the organization or healthcare system. If organizations are to provide quality care for the growing number of survivors, it will be critical to show financial impacts for the healthcare system. An additional process indicator focused specifically on assessing and mitigating survivors’ financial hardship and concerns regarding insurance coverage. This is not surprising given the high proportion of cancer survivors who reporting experiencing financial challenges in the United States [ 21 ].

Consensus meetings also discussed considerations that health systems should take when implementing these standards. In the area of health system policy, experts and advocates noted that these indicators could be combined into one survivorship policy that informs care system-wide. One advantage of separating these indicators, however, is that key informants or stakeholders could be included in development or writing the individual policies. Furthermore, the impact of separate policy changes could be evaluated individually. Additionally, several experts noted that process indicators related to physical, psychological, and social impacts of cancer and its treatment should go beyond only assessment for late effects and should include management and specialty referral, as indicated. Experts and advocates also noted that in the area of evaluation/assessment, validated measures should be utilized whenever possible. While this process was not intended to endorse specific measures or tools, participants emphasized an expectation that validated, patient-centered measures would be used. Finally, health systems should ensure consent has been obtained from survivors and caregivers before assessment.

The final set of standards represents input from survivorship experts and advocates and can be implemented in a variety of settings. A key next step in this work is to implement the standards within healthcare systems that are developing new survivorship programs or have existing programs or services. Healthcare systems that provide care for people after a cancer diagnosis, including but not limited to cancer centers, may use these standards to assess organizational alignment and enhance their survivorship care services. After aligning with the standards, there will be a need to evaluate for feasibility, potential for sustainability, and impact on survivor outcomes. It is important to note that use of these standards by health systems is voluntary, and components of care may or may not be covered by public or private health insurance.

There are several key considerations when implementing these survivorship standards to inform survivorship program development or assess alignment with the indicators. First, while these standards were intended to inform health systems caring for cancer survivors diagnosed at any age, with any cancer, and at any stage, there is a need to tailor care services based on specific factors, including age, setting, and specific cancer types and treatment. For example, while we utilized the NCI recognized definition of a cancer survivor from the time of diagnosis through the balance of life [ 1 ], the standards could also be applied to post-treatment survivors. In addition, survivors of pediatric cancers diagnosed between birth and 15 years may have markedly different needs from survivors of adolescent and young adult (AYA) or older adult survivors [ 22 , 23 ]. For patients diagnosed as young children, survivorship care and research are already highly developed with effective, existing clinical models that constitute “standard of care” for this population. Indeed, in many ways pediatric survivorship care and research have inspired adult-focused efforts. But in pediatric cancer survivorship, particularly for well-established treatments, the evidence base for late effects and their trajectory is well-established and comprehensive. For most childhood cancer survivors, recommended survivorship care for late effects monitoring and management is generally annual, lifelong follow-up, including transition to adult-focused care during young adulthood [ 24 ]. There is still significant work to be done to improve transition services and outcomes [ 25 ], which is one area where the standards could be very informative. Existing guidelines for childhood cancer survivorship care should continue to be utilized [ 13 ].

For survivors of AYA cancers (diagnosed between 15 and 39 years old), health systems and providers must pay close attention to the unique needs of this population. While the components of their survivorship care may be similar to older adults, AYAs are particularly vulnerable to adverse impacts of cancer on education, career development, work, financial status, and psychosocial needs. Fertility is a particular medical concern for this cancer population. Although a separate, parallel set of survivorship standards for survivors of AYA cancer may not be necessary, it is crucial these standards be applied in a manner responsive to their needs. For older adult survivors, geriatric assessment and focused provider training could be incorporated to address the unique considerations of older adults with a history of cancer [ 26 ]. Overall, the standards should be used as a guide for health systems to adapt based on the known needs of populations served.

Though the results of this work represent national standards for survivorship care, these standards can also be utilized to inform survivorship research. The NCI has supported key efforts in survivorship care, including funding opportunities focused on addressing primary care for cancer survivors [ 27 ] and optimizing survivorship care for survivors transitioning between oncology and non-oncology providers [ 28 ]. A challenge in delivering and evaluating survivorship care, however, is that there has not been an accepted national standard. This project represents consensus agreement of national experts on essential policy, process, and evaluation components to survivorship care that health systems should utilize based on the available evidence. Rigorous evaluation of the implementation and outcomes of these standards will be critical to show continued value to follow-up care for people after a cancer diagnosis [ 29 ]. Additionally, the evaluation/assessment indicators of these standards may be used as meaningful endpoints for survivorship care interventions to show impact on survivor and health system outcomes.

These standards represent a key foundation for improving the delivery of survivorship care across the United States; however, there are limitations to this work. First, though a robust review was conducted to identify potential indicators, it is possible that specific literature may have been inadvertently missed. In addition, the consensus meetings represented a diverse panel of experts and survivor advocates who provided feedback and input. It is possible, however, that some perspectives were not represented in the expert group. One important perspective missing is that of a healthcare business administrator, who will be essential in converting these standards to implementation of survivorship programs. A key next step in this work will be collaboration with healthcare administrators and payors to translate these recommendations into action. The selection of experts also may lead to limitations in the prioritization of indicators. Future empirical support is needed to provide evidence of the outcomes of implementing the standards.

Data availability

No datasets were generated or analyzed during the current study.

This study has not been previously presented.

National Cancer Institute. Office of cancer survivorship: definitions. 2020; Available from: https://cancercontrol.cancer.gov/ocs/statistics#definitions .

Miller, K.D., et al. Cancer treatment and survivorship statistics, 2022. CA Cancer J Clin. 2022;72(5):409–436. https://doi.org/10.3322/caac.21731 .

Nekhlyudov L, et al. Developing a quality of cancer survivorship care framework: implications for clinical care, research and policy. J Natl Cancer Inst. 2019;111(11):1120–30.

Article   PubMed   PubMed Central   Google Scholar  

From Cancer Patient to Cancer Survivor: Lost in Transition. Washington, DC: The National Academies Press. 2006.  https://doi.org/10.17226/11468 .

Sanft T, et al. NCCN Guidelines® Insights: survivorship, version 1.2023. J Natl Compr Canc Netw. 2023;21(8):792–803.

Article   PubMed   Google Scholar  

Cohen EE, et al. American Cancer Society head and neck cancer survivorship care guideline. CA Cancer J Clin. 2016;66(3):203–39.

Nekhlyudov L, et al. Head and Neck cancer survivorship care guideline: American Society of Clinical Oncology clinical practice guideline endorsement of the American Cancer Society Guideline. J Clin Oncol. 2017;35(14):1606–21.

Skolarus T, et al. American Cancer Society prostate cancer survivorship care guidelines. CA: A Cancer Journal for Clinicians. 2014;64:225–49.

PubMed   Google Scholar  

El-Shami K, et al. American Cancer Society colorectal cancer survivorship care guidelines. CA Cancer J Clin. 2015;65(6):428–55.

Runowicz C, et al. American Cancer Society/American Society of Clinical Oncology breast cancer survivorship care guideline. J Clin Oncol. 2016;34:611–35.

Article   CAS   PubMed   Google Scholar  

American Society of Clinical Oncology, American Society of Clinical Oncology clinical practice survivorship guidelines and adaptations: summary of recommendations. Alexandria. VA: American Society of Clinical Oncology; 2014.

Google Scholar  

Children’s Oncology Group, Long-term follow-up guidelines for survivors of childhood, adolescent, and young adult cancer. 2013, Children’s Oncology Group: Monrovia, CA.

Children’s Oncology Group. Long-term follow-up guidelines for survivors of childhood, adolescent, and young adult cancers Version 6.0. 2023 [cited 2024 January 30]; Available from: http://www.survivorshipguidelines.org/ .

Rechis, R., et al. The essential elements of survivorship care: a LIVESTRONG brief. 2011 [cited 2018; Available from: https://www.livestrong.org/sites/default/files/what-we-do/reports/EssentialElementsBrief.pdf .

Blaes AH, et al. Survivorship care plans and the commission on cancer standards: the increasing need for better strategies to improve the outcome for survivors of cancer. JCO Oncology Practice. 2020;16(8):447–50.

Shulman LN, et al. Treatment summaries and survivorship care plans: the approach by the commission on cancer to increase use. J Oncol Pract. 2015;11(1):40–1.

Lisy, K., et al. How do we define and measure optimal care for cancer survivors? An online modified reactive Delphi study. Cancers (Basel), 2021;13(10).

Mollica MA, et al. Survivorship objectives in comprehensive cancer control plans: a systematic review. J Cancer Surviv. 2020;14(2):235–43.

Anampa-Guzmán, A., et al. Online description of services provided in adult survivorship programs across U.S. accredited cancer centers . J Cancer Surviv. 2024;18(1):79–83. https://doi.org/10.1007/s11764-023-01361-w .

Jacobsen PB, et al. Systematic review of the impact of cancer survivorship care plans on health outcomes and health care delivery. J Clin Oncol. 2018;36(20):2088–100.

Abrams HR, et al. Financial toxicity in cancer care: origins, impact, and solutions. Translational Behavioral Medicine. 2021;11(11):2043–54.

Siembida EJ, et al. Measuring health-related quality of life in adolescent and young adult cancer survivors with the National Institutes of Health Patient-Reported Outcomes Measurement Information System(®): comparing adolescent, emerging adult, and young adult survivor perspectives. Psychooncology. 2021;30(3):303–11.

Anderson C, et al. Noncancer mortality among adolescents and young adults with cancer. Cancer. 2019;125(12):2107–14.

Tonorezos ES, et al. Models of care for survivors of childhood cancer from across the globe: advancing survivorship care in the next decade. J Clin Oncol. 2018;36(21):2223–30.

Marchak JG, et al. Transition practices for survivors of childhood cancer: a report from the Children’s Oncology Group. J Cancer Surviv. 2023;17(2):342–50.

Dale W, et al. How is geriatric assessment used in clinical practice for older adults with cancer? A survey of cancer providers by the American Society of Clinical Oncology. JCO Oncol Pract. 2021;17(6):336–44.

National Cancer Institute. RFA-CA-23–025: addressing the primary care needs of cancer survivors (U01 clinical trial required). 2023 [cited 2024 January 19]; Available from: https://grants.nih.gov/grants/guide/rfa-files/rfa-ca-23-025.html .

National Cancer Institute. RFA-CA-19–035: Optimizing the management and outcomes for cancer survivors transitioning to follow-up care. 2020 [cited 2024 January 19]; Available from: https://cancercontrol.cancer.gov/research-emphasis/contribution-to-the-cancer-moonshot/optimizing-management-outcomes .

Halpern M, et al. Myths and presumptions about cancer survivorship. J Clin Oncol. 2024;42(2):134–9.

Download references

Acknowledgements

National Survivorship Standards Subject Matter Expert Group

Open access funding provided by the National Institutes of Health.

Author information

Authors and affiliations.

Division of Cancer Control and Population Sciences, National Cancer Institute, 9609 Medical Center Drive, MSC 9712, Room 3E440, Bethesda, MD, 20892-9762, USA

Michelle A. Mollica & Emily Tonorezos

Department of Veterans Affairs, National Oncology Program, Washington, DC, USA

Gina McWhirter

Hematology/Oncology Service, Brooke Army Medical Center, Defense Health Agency, San Antonio, TX, USA

Joshua Fenderson

Uniformed Services University of Health Sciences, Bethesda, MD, USA

Keck School of Medicine, University of Southern California, Los Angeles, CA, USA

David R. Freyer

Children’s Hospital Los Angeles and USC Norris Comprehensive Cancer Center, Los Angeles, CA, USA

Department of Health Services Research, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia

Michael Jefford

Australian Cancer Survivorship Centre, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia

Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, VIC, Australia

Office of The Assistant Secretary of Defense for Health Affairs, Department of Defense, Washington, DC, USA

Christopher J. Luevano

Department of Surgery, College of Medicine, University of Kentucky, Lexington, KY, USA

Timothy Mullett

National Coalition for Cancer Survivorship, Silver Spring, MD, USA

Shelley Fuld Nasso

Cancer Survivorship Advocate, Carolina Pediatric Therapy, Asheville, NC, USA

Ethan Schilling

Division of Hematology/Oncology, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA

Vida Almario Passero

Section of Hematology/Oncology, VA Pittsburgh Healthcare System, Pittsburgh, PA, USA

VA National TeleOncology, Durham, NC, USA

You can also search for this author in PubMed   Google Scholar

• Catherine Alfano
• , Precilla Belin
• , Anne Blaes
• , Hillary Cavanagh
• , Lanie Francis
• , David R. Freyer
• , Danielle Friedman
• , Shelley Fuld Nasso
• , Patricia Ganz
• , Batsheva Honig
• , Shawna Hudson
• , Linda Jacobs
• , Michael Jefford
• , Nancy Keating
• , Anne Kirchoff
• , Michelle Kirschner
• , Ron Kline
• , Jessica MacIntyre
• , Molly Maher
• , Deborah Mayer
• , Janette Merrill
• , Timothy Mullett
• , Larissa Nekhlyudov
• , Frank Penedo
• , Mackenzi Pergolotti
• , Michael Roth
• , Tara Sanft
• , Alyssa Schatz
• , Ethan Schilling
• , Kathryn Schmitz
• , Lisa Schwartz
•  & Emily Tonorezos

Contributions

MAM, GM, ET, and VAP were responsible for research conceptualization. MAM wrote the main manuscript text in collaboration with all authors. All authors reviewed the study design, results, and the manuscript.

Corresponding author

Correspondence to Michelle A. Mollica .

Ethics declarations

Consent to participate.

No human subject participants were included in this project. Subject matter experts implied consent to participate in the consensus process.

Conflict of interest

The authors declare no competing interests.

Disclaimers

The article was prepared as part of some of the authors’ (MAM, ET) official duties as employees of the US Federal Government. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the National Cancer Institute.

The view(s) expressed herein are those of the author(s) and do not reflect the official policy or position of the Defense Health Agency, Department of Defense, or the US Government. Mention of trade names, commercial products, or organizations does not imply endorsement by Department of Defense or the US Government. The views expressed herein are those of the authors and do not reflect the official policy or position of the Department of Veterans Affairs or the US Government.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 25 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Mollica, M.A., McWhirter, G., Tonorezos, E. et al. Developing national cancer survivorship standards to inform quality of care in the United States using a consensus approach. J Cancer Surviv (2024). https://doi.org/10.1007/s11764-024-01602-6

Download citation

Received : 04 April 2024

Accepted : 11 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1007/s11764-024-01602-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health system
• Survivorship
• Care quality
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 16 May 2024

Predictors for interest to participate in digital mental health therapy: a cross-sectional survey of individuals with anxiety and depression

• Tharidu Gunawardena 1 ,
• Marilyn M. Bartholmae 1 , 2 ,
• Matvey V. Karpov 1 ,
• Rohan Dod 1 ,
• Kripa Ahuja 1 ,
• Aishwarya Rajendran 1 ,
• Mayuri Kathrotia 1 &
• Sunita Dodani 1 , 3  

BMC Digital Health volume  2 , Article number:  21 ( 2024 ) Cite this article

68 Accesses

Metrics details

Due to a multitude of factors, the onset of the COVID-19 pandemic resulted in a significant increase in mental health issues within society, including depression and anxiety. Due to the increased trend of mental health disorders in society, digital mental health therapies are more useful than ever. With the emergence of programs utilizing Internet Cognitive Behavioral Therapy (iCBT), mental health resources are easily accessible and can be widely implemented to those in need.

The aim of this study was to identify predictors for interest to participate in SilverCloud digital mental health therapy among individuals with mild to severe anxiety and/or depression based on preliminary findings from the COVIDsmart study.

COVIDSmart participants who had moderate to severe anxiety and/or depression based on the PHQ-9 and GAD-7 scores, and who consented to be contacted for future studies, were invited to complete a needs assessment survey via Research Electronic Data Capture (REDCap). This assessment used self-reported measures including medical diagnoses, mental health services received, reasons for anxiety and/or depression, the use of coping strategies, suicidal ideology using the Ask Suicide Questions tool, and whether they would be interested in receiving free digital mental therapy. Descriptive statistics were used to report participants’ demographics and a logistic regression was used to find predictors for interest in participation in SilverCloud. SAS 9.4 was used and p values < 0.05 were considered significant.

Out of the original 782 COVIDsmart participants, 634 consented to be contacted for future studies, 280 were subsequently invited to complete the SilverCloud needs assessment, and 120 individuals completed it. The largest demographic among these participants were females (70.83%) who identified as White (80.83%). The mean age was 48.74 years ( SD  = 14.66). Results revealed that having a mental health comorbidity significantly predicted the likelihood of interest in participating in the SilverCloud digital mental health program ( p  = 0.027).

Conclusions

In this study, mental illness comorbidities predicted the interest to participate in digital mental therapy. Fragmented healthcare and perceptions of unmet care needs are likely contributor factors. Further research with a diverse sample of participants is necessary for generalizability. Findings may have important implications for healthcare best practices.

Peer Review reports

Introduction

Repeatedly, it has been evident that periods of widespread human catastrophe or disaster often result in large-scale detrimental impacts on mental health. During the COVID-19 pandemic, there were not only new cases of mental health disorders but also an exacerbation of existing mental health conditions [ 1 ]. According to a briefing released by the World Health Organization in March 2022, during the first year of the COVID-19 pandemic, the prevalence of major depressive disorders and anxiety disorders had increased significantly [ 2 ]. There were multiple factors leading to a significant spike in mental health disorders. Common stressors were social isolationism, perpetual fear of COVID-19 infection, loss of employment, death of a loved one or friend and uncertainty about the future [ 3 ]. Despite many of the COVID-19 restrictions and lockdowns being relaxed since March of 2020, the damaging adverse mental health effects of the pandemic seem to have persisted throughout the United States (U.S). According to a CDC report, the prevalence of anxiety and depressive disorders has been three and four times higher, respectively, compared to pre-pandemic levels within the U.S.[ 4 ]. In 2021, mental health screenings were taken by over 5.4 million people, indicating a near 500% increase since 2019 and a 103% increase since 2020 [ 5 ].

In the state of Virginia, the COVIDsmart study was initiated to assess the effects of COVID-19 on health, behavioral and economic status of individuals in Virginia. The study gathered data from March 2021 to November 2021 through longitudinal surveys. The surveys comprised the Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 (GAD-7), a condensed version of the UCLA Loneliness Scale, and a condensed version of the Social Network Index [ 6 ]. 782 Virginia residents, spanning from 18 to 87 years, registered in the COVIDsmart study. The COVIDsmart sample consisted of mostly White (81.5%), middle aged (mean age 51), middle income (51% with income of $100,000 or higher), college educated (62.6%) females (78.6%) [ 6 , 7 ]. The COVIDsmart preliminary results revealed that 21% of participants had moderate to severe anxiety and 25% had moderate to severe depression [ 7 ]. It was evident that there was a significant presence of mental disorders among Virginia residents after the COVID-19 pandemic. This prompted us to believe that there was a considerable number of individuals that required mental health services, but simply did not receive adequate counseling or therapy, leading to unmet health needs.

Based on these preliminary findings, a significant percentage of the participants of the COVIDsmart study had experienced some level of depression or anxiety, therefore the SilverCloud study was conceived and was funded by Eastern Virginia Medical School (EVMS) to alleviate the high rates of anxiety and depression among Virginians. Designed to alleviate depression and anxiety, SilverCloud is a digital mental health program that uses Internet Cognitive Behavioral Therapy (iCBT). CBT has been a highly effective form of psychological therapy for over 50 years. As patient requirements have increased, mental healthcare delivery has evolved significantly in the past decade with the introduction of new tools. Psychological interventions, specifically CBT, have become easier to implement in the form of iCBT [ 8 ]. The benefits of iCBT include cost effectiveness and global use [ 9 ]. Additionally, patients can utilize iCBT programs from the comfort of their homes, without the need for any in-person interaction. This can be particularly useful for those who have busy schedules or reside in more remote locations.

Although the mental health digital landscape is full of CBT-based apps and other digital interventions, meta-analyses and systematic reviews suggest that iCBT targeting anxiety and depression is not always effective. For example, a 2020 study by Etzelmueller et al., provided evidence for efficacy of iCBT interventions via randomized clinical trials [ 10 ] while Liu et al., 2022, suggested some iCBT-based modules are effective in treating anxiety and depression but other modules are not. Implementation science highlighting specialized needs, resources, and training to operate digital services is necessary for the success of digital therapeutics in healthcare [ 11 ]. SilverCloud Health has achieved global reach by using implementation science strategies and is the global leader in developing and delivering evidence-based mental health interventions across healthcare systems, payors such as insurance companies and government plans, providers, employer organizations, and charity/advocacy agencies [ 12 ]. SilverCloud delivers a vast array of personalized interventions to meet the specific mental health needs of individuals. Compared to other traditional digital platforms, SilverCloud offers self-guided programs, digital coaching, progress tracking tools and mobile accessibility in a secure environment. SilverCloud programs for anxiety and depression are recommended based on PHQ-9 and GAD-7 scores. These programs encourage the development of awareness and understanding of emotions, while increasing daily enjoyable activities and motivation. CBT is used to teach healthy thinking, feelings, behaviors, and to make positive changes to overcome anxiety and/or depression. SilverCloud programs include 10 modules: Getting Started, Understanding Depression and Anxiety, Noticing Feelings, Boosting Behavior, Spotting Thoughts, Challenging Thoughts, Managing Worry, Core Beliefs (unlockable by Coach), Facing Your Fears (unlockable by coach), and Bringing It All Together. SilverCloud coaches are licensed mental health professionals who provide weekly support to motivate users to engage in the SilverCloud programs, provide daily and weekly goals, and help users apply their learnings into everyday life [ 13 ]. The preliminary randomized controlled trials, which utilized SilverCloud, indicate that this therapy is effective as a stand-alone treatment for mild to moderate anxiety and depression [ 14 ].

The objective of this study was to determine the factors that can predict interest in participating in the SilverCloud program by COVIDsmart participants. Additionally, a secondary objective was to recognize the particular traits that enhance the probability of people seeking digital mental health care. This data could have significant clinical implications as it would enable healthcare professionals to gain better insights into the mental health needs of Virginians. With a greater understanding of the characteristics of these individuals in local communities, clinicians can customize digital interventions that cater to specific groups. This approach could enhance the rates of mental health screening and treatment accessibility among vulnerable populations, leading to better patient outcomes. Additionally, utilizing this information may enable digital mental health professionals to provide tailored and suitable treatment to their patients in the future. The implementation of these strategies could potentially have a notable effect in alleviating the continuous mental health emergency at a local, national, and conceivably worldwide scale.

Study design and recruitment

In order to conduct the SilverCloud study, the participants were recruited from the original COVIDSmart study. There were several recruitment strategies implemented in the COVIDSmart study, particularly to increase the likelihood of including ethnic minorities, individuals from rural areas, and individuals from lower socioeconomic backgrounds. The strategies implemented included online articles, employer e-newsletters, purchased email lists, social media posts, television, radio, paper flyers, and digital flyers. As an incentive, electronic gift cards were used to garner participation in the study. The data collection was conducted via an online HIPAA compliant platform designed by Vibrent Health Inc. The study was a joint effort with EVMS-Sentara Healthcare Analytics and Delivery Science Institute (HADSI), George Mason University, and Vibrent Health. 782 residents of Virginia enrolled in the COVIDSmart registry from March to May 2021.

The SilverCloud study was created to address the COVIDSmart study’s preliminary results showing 21% of participants had moderate to severe anxiety and 25% had moderate to severe depression [ 15 ]. Figure 1 shows the demographics for COVIDSmart participants.

Demographics of COVIDsmart participants, N =771

Out of the 782 original COVIDsmart participants, 634 of them consented to be contacted for future studies. 280 of the COVIDSmart participants who had moderate to severe anxiety and/or depression based on the PHQ-9 and GAD-7 scores, and who consented to be contacted for future studies, were invited to complete a needs assessment questionnaire to further determine eligibility to the SilverCloud study. 120 individuals completed the SilverCloud needs assessment. The needs assessment (Appendix 1 ) evaluated self-reported measures such as medical diagnoses, mental health services received, reasons for anxiety and/or depression, the use of coping strategies, suicidal ideology (using the Ask Suicide Questions (ASQ) survey), and whether they would be interested in receiving free digital mental therapy.

Thus, the purpose of this survey was twofold: to determine their eligibility for the 8-week SilverCloud study and to assess the factors contributing to their elevated levels of anxiety and/or depression during the COVID-19 pandemic (Fig. 2 ).

Procedure for participant recruitment and enrollment

COVIDsmart participants who had an acute suicide screening based on the ASQ survey or self-reported substance abuse and dependence were not eligible to participate in SilverCloud. However, they were provided with a list of resources they could reach for immediate assistance (Appendix 2 ). This study was approved by the Eastern Virginia Medical School Institutional Review Board (IRB# 21–07-FB-0185).

Data collection

The SilverCloud needs assessment survey was created using Research Electronic Data Capture (REDCap). REDCap is a secure web-based application used to create forms and manage databases in order to support data capture and surveys for research. REDCap data is stored securely at EVMS on private, protected servers, and meets requirements for the Health Insurance Portability and Accountability Act for collection of personal health information. The needs assessment was sent via REDcap to identified potential participants.

Statistical analysis

Descriptive statistics were used to analyze the demographics of participants. To find predictors for interest to participate in SilverCloud, we used a logistic regression including the independent variables: receiving mental health care (yes or no), having coping strategies (yes or no), severity categories for GAD-7 and PHQ-9 (moderate to severe), mental illness comorbidity (yes or no), current medications for anxiety and/or depression (yes or no), race, sex, and age. The dependent variable was interest to enroll in SilverCloud (yes or no). We conducted the analysis using SAS 9.4, and p values < 0.05 were considered significant.

A total of 120 participants who were part of the COVIDsmart program completed a needs assessment for SilverCloud. Table 1 shows that the majority of SilverCloud participants were female (70.83%) and White (80.83%). The mean age of participants was 48.74 years ( SD  = 14.66, Table  2 ).

The study found that having a co-existing mental health condition was a significant predictor of the likelihood of showing interest in participating in the digital mental health SilverCloud program (p = 0.027). Among participants with moderate to severe anxiety and/or depression who were enrolled in COVIDsmart, those with a comorbidity of mental illness were 378% more inclined to be interested in SilverCloud as compared to those without a mental illness comorbidity. The mental health comorbidities reported by SilverCloud participants had a wide range, but the most common comorbidity was Posttraumatic Stress Disorder (PTSD) (28.6%), followed by eating disorder (21.4%) and Attention Deficit Hyperactivity Disorder (ADHD) (14.3%), as presented in Table  3 . These reported comorbidities were based off of self-reported survey data, and were not verified diagnoses.

However, factors such as race, gender, age, taking medication for anxiety or depression, anxiety severity, depression severity, use of coping strategies, and receiving mental health services did not have any predictive value in terms of interest in participating in SilverCloud ( p  ≥ 0.05), as shown in Table  4 .

Principle results

This study showed that the only significant factor that led COVIDsmart participants to be interested in participating in SilverCloud for depression and anxiety treatment was a mental illness comorbidity. The most common comorbidities reported included PTSD, eating disorders, and ADHD. Individuals with comorbid mental illnesses experience a cumulative medical burden, often requiring multiple physical and mental health services [ 15 , 16 ]. These individuals are more likely to experience unintegrated and uncoordinated healthcare delivery [ 17 , 18 ]. Furthermore, individuals with multiple mental health comorbidities are more likely to have perceived unmet needs for care [ 19 , 20 ]. Thus, we hypothesize that individuals who experience fragmented healthcare and perceived unmet care needs may be more willing to accept mental health treatment for new conditions.

In addition, participants who had high levels of anxiety and/or depression, but no prior experience with mental health counseling, may be more apprehensive to receive treatment due to societal stigma about treatment for mental disorders. Participants with mental disorders with high levels of experienced stigma have lower rates of recovery since they are less likely to receive mental health services for their condition [ 21 ]. In this study, it seemed as if participants with multiple mental conditions were more likely to accept digital mental health treatment. The digital therapeutic setting may remove the stigma barrier for individuals with comorbid mental illnesses.

The PHQ-9 and GAD-7 scores did not have a significant predictive value for participating in mental health therapy. Unfortunately, it is unclear as to why this is. We hypothesize that since PHQ-9 and GAD-7 scores are only indicators for depression and anxiety alone, respectively, it is possible that despite a high level of depression and/or anxiety, the mental burden on these participants may not be as taxing as it is for participants with multiple mental health comorbidities, therefore, these participants were less likely to enroll in digital mental health therapy compared to participants with multiple comorbid mental illness. The individuals who chose to enroll in digital mental therapy may require integrated healthcare approaches to address their physical/mental multi-comorbidity healthcare needs. Additionally, participants who were already receiving mental health care services as well as those who were using coping strategies such as exercising and drinking less alcohol, were not likely to be interested in digital mental therapy.

The majority of participants in the COVIDsmart study were Non-Hispanic Whites and females, which explains why a similar demographic makeup was observed in the SilverCloud study. However, the SilverCloud study had a slightly higher percentage of female and Non-Hispanic White participants compared to the COVIDsmart study. Additionally, the greatest percentage of COVIDsmart participants fell into the 46–55 age range (Fig. 1 ). This corresponds accordingly with the mean age of SilverCloud participants being 48.74 years old (Table  2 ). Despite this, neither race nor gender nor age were significant predictors of participation in digital mental health therapy. It is not clear why these particular demographics were more likely to participate in both studies. Non-Hispanic Whites are known to have higher rates of mental disorders compared to people of color, although this may be due to various factors such as discriminatory medical practices, negative cultural attitudes towards mental illness, lack of insurance, and language barriers that minorities face [ 22 , 23 ].

Future directions

Our findings suggest healthcare workers should especially turn their attention towards individuals who are already suffering from coexisting mental conditions. It is very possible that the introduction of future large-scale stressors can result in new cases or the exacerbation of milder forms of depression and anxiety in these patients. Mental health counselors, psychiatrists, and therapists should regularly screen patients with existing mental health conditions with the PHQ-9 and GAD-7 for any new developments regarding depression and anxiety. Moreover, future research is needed to further evaluate the role of healthcare fragmentation and perceptions of unmet medical care needs in the enrollment of digital mental therapies. Findings could have important implications for healthcare best practices.

Future research should expand the scope and demographics of this study. While the cause of the increase in the percentage of highly-educated female and Non-Hispanic White participants in the SilverCloud study remains unknown, it is important to consider this metric on a global scale. Large nationally representative US surveys have found that mental health of Black, Hispanic, and Asian respondents worsened relative to White respondents during the pandemic, including significant increases in depression and anxiety among racialized minorities compared to White people [ 24 ]. These surveys also showed that White respondents were the most likely to receive professional mental health care before and during the pandemic, while minority respondents demonstrated higher levels of unmet mental health care needs during the pandemic than White respondents [ 24 ]. The findings of our study showed a similar trend, wherein there was an increase in the number of White participants seeking mental health care. Although our study was limited in size compared to a nationwide study, the outcomes are comparable. Securing a diverse socio-economic and racial participation is important for generalizability of future studies.

Limitations

The study has several limitations. Firstly, it only included participants from the COVIDsmart study, which means that the sample was restricted to residents of Virginia. Secondly, research fatigue from the six-month longitudinal COVIDsmart study could have deterred some participants from taking part in the SilverCloud study, thus resulting in a smaller sample size. Thirdly, the socioeconomic and racial diversity of the SilverCloud study participants did not accurately reflect the Virginia population, as there was an overrepresentation of highly educated White females and an underrepresentation of minority groups, including those living in rural areas. The lack of participant diversity and limited scope of this study may significantly hamper its generalizability to broader populations.

Fourthly, because the SilverCloud program was only accessible online, participants without internet access in Virginia may have been less likely to participate. In the future, alternative treatment options such as traditional in-person face-to-face therapy could be considered, although this would require additional resources and higher costs. Fifth, the mental illness comorbidity was self-reported. We included the variable comorbidity (yes or no) in the logistic regression model to determine whether having a mental illness comorbidity would influence the outcome: interest to participate in SilverCloud. Sixth, high risk respondents to the needs assessment could only be provided with available resources to receive help, rather than being provided with an on demand consultation with an available clinician.

Individuals with mental illness comorbidities may have a higher tendency to seek and participate in digital mental health programs amid the COVID-19 pandemic. Digital CBT-based programs can potentially improve the accessibility of care for this group. However, programs should prepare to address the needs of patients with mental illness comorbidities as they may require multidisciplinary healthcare services. Mental health professionals and researchers need to gain a deeper understanding of the unique needs of this population.

Availability of data and materials

All data generated or analyzed during this study are included in this published article [and its supplementary information files].

Abbreviations

Attention Deficit Hyperactive Disorder

Eastern Virginia Medical School

Internet cognitive behavioral therapy

Posttraumatic Stress Disorder

Research Electronic Data Capture

Jones EAK, Mitra AK, Bhuiyan AR. Impact of COVID-19 on mental health in adolescents: a systematic review. Int J Environ Res Public Health. 2021;18(5):2470. https://doi.org/10.3390/ijerph18052470 . PMID: 33802278; PMCID: PMC7967607.

Article   CAS   PubMed   PubMed Central   Google Scholar  

World Health Organization. Covid-19 pandemic triggers 25% increase in prevalence of anxiety and depression worldwide. World Health Organization. 2022. Retrieved February 1, 2023, from https://www.who.int/news/item/02-03-2022-covid-19-pandemic-triggers-25-increase-in-prevalence-of-anxiety-and-depression-worldwide

Panchal N, Kamal R, Cox C, & Garfield R. The implications of COVID-19 for mental health and substance use. KFF. 2021. Retrieved February 3, 2023, from https://www.kff.org/coronavirus-covid-19/issue-brief/the-implications-of-covid-19-for-mental-health-and-substance-use/

Czeisler MÉ, Lane RI, Petrosky E, et al. Mental health, substance use, and suicidal ideation during the COVID-19 pandemic — United States, June 24–30, 2020. MMWR Morb Mortal Wkly Rep. 2020;69:1049–57. https://doi.org/10.15585/mmwr.mm6932a1 .

Mental health and covid-19: Two years after the pandemic, mental health concerns continue to increase. Mental Health America. (n.d.). Retrieved February 3, 2023, from https://mhanational.org/mental-health-and-covid-19-two-years-after-pandemic

Bartholmae MM, Roess AA, Renshaw KD, Levy BL, Karpov MV, Sill JM, Dodani S. Evaluation of recruitment strategies on inclusiveness of populations at risk for health disparities in the statewide remote online COVID smart registry. Virginia J Publ Health. 2022;7(1):11–45 ( https://commons.lib.jmu.edu/vjph/vol7/iss1/5/ ).

Google Scholar  

Schilling J, Klein D, Bartholmae MM, et al. COVID smart: A digital health initiative for remote data collection and study of COVID’s impact on the state of Virginia. J Med Internet Res. 2023. https://doi.org/10.2196/37550 .

Article   PubMed   PubMed Central   Google Scholar  

Karyotaki E, et al. Internet-based cognitive behavioral therapy for depression: a systematic review and individual patient data network meta-analysis. JAMA Psychiat. 2021;78(4):361–71. https://doi.org/10.1001/jamapsychiatry.2020.4364 . PMID: 33471111; PMCID: PMC8027916.

Article   Google Scholar  

Kumar V, Sattar Y, Bseiso A, Khan S, Rutkofsky IH. The effectiveness of internet-based cognitive behavioral therapy in treatment of psychiatric disorders. Cureus. 2017;9(8):e1626. https://doi.org/10.7759/cureus.1626 . PMID: 29098136; PMCID: PMC5659300.

Etzelmueller A, Vis C, Karyotaki E, Baumeister H, Titov N, Berking M, Cuijpers P, Riper H, & Ebert DD. Effects of internet-based cognitive behavioral therapy in routine care for adults in treatment for depression and anxiety: Systematic review and meta-analysis. Med Inter Res 2020;22(8). https://doi.org/10.2196/18100

Liu T, Xu J, Cheng H, Zhang Y, Wang S, Lin L, Tian L. Effects of internet-based cognitive behavioral therapy on anxiety and depression symptoms in cancer patients: a meta-analysis. Gen Hosp Psychiatry. 2022;79:135–45. https://doi.org/10.1016/j.genhosppsych.2022.09.003 .

Article   PubMed   Google Scholar  

Richards D, Enrique A, Palacios J, et al. SilverCloud health: online mental health and wellbeing platform. In: Sverdlov O, van Dam J, eds. Digital Therapeutics Strategic, Scientific, Developmental, and Regulatory Aspects . 1st ed. CRS Press. 2022; 307–329. https://doi.org/10.1201/9781003017288-15

SilverCloud Program Guide. Anxiety depression, and co-morbid anxiety & depression. (n. d.). Retrieved June 22, 2023, from https://www.silvercloudhealth.com/programs/mental-health-wellbeing

Richards D, Enrique A, Eilert N, Franklin M, Palacios J, Duffy D, Earley C, Chapman J, Jell G, Sollesse S, Timulak L. Author correction: A pragmatic randomized waitlist-controlled effectiveness and cost-effectiveness trial of digital interventions for depression and anxiety. Npj Digit Med 2020;3(1). https://doi.org/10.1038/s41746-020-0298-3

Bartholmae MM, Karpov Sill JM, Karpov MV, Sunita D. The association between depression, anxiety, and COVID-19 symptoms. J Mental Health Clin Psychol. 2023;7(2):1–8. https://doi.org/10.29245/2578-2959/2023/2.1277 .

Doherty AM, Gaughran F. The interface of physical and mental health. Soc Psychiatry Psychiatr Epidemiol. 2014;49(5):673–82. https://doi.org/10.1007/s00127-014-0847-7 .

Pizzol D, Trott M, Butler L, et al. Relationship between severe mental illness and physical multimorbidity: a meta-analysis and call for action. BMJ Ment Health. 2023;26:e300870. Retrieved from: https://mentalhealth.bmj.com/content/26/1/e300870 .

Walker ER, Druss BG. Cumulative burden of comorbid mental disorders, substance use disorders, chronic medical conditions, and poverty on health among adults in the USA. Psychol Health Med. 2017;22(6):727–35. https://doi.org/10.1080/13548506.2016.1227855 Epub 2016 Sep 3. PMID: 27593083; PMCID: PMC5564203. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5564203/ .

Forman-Hoffman VL, Batts KR, Hedden SL, Spagnola K, Bose J. Comorbid mental disorders among adults in the mental health surveillance survey. Ann Epidemiol. 2018;28(7):468–74. Retrieved from: https://www.sciencedirect.com/science/article/pii/S104727971730889X .

Chen L-Y, Crum RM, Martins SS, Kaufmann CN, Strain EC, Mojtabai R. Service use and barriers to mental health care among adults with major depression and comorbid substance dependence. Psychiatric Services. 2013;64(9):863–70. Retrieved from https://ps.psychiatryonline.org/doi/10.1176/appi.ps.201200289 .

Oexle N, Müller M, Kawohl W, Xu Z, Viering S, Wyss C, Vetter S, Rüsch N. Self-stigma as a barrier to recovery: A longitudinal study. Eur Arch Psychiatry Clin Neurosci. 2017;268(2):209–12. https://doi.org/10.1007/s00406-017-0773-2 .

Panchal N, Saunders H, & Ndugga N. Five key findings on Mental Health and Substance Use Disorders by race/ethnicity. KFF. 2022. Retrieved March 5, 2023, from https://www.kff.org/racial-equity-and-health-policy/issue-brief/five-key-findings-on-mental-health-and-substance-use-disorders-by-race-ethnicity/#:~:text=Overall%20rates%20of%20mental%20illness,to%20accessing%20mental%20health%20care .

Racial disparities in mental health treatment. SC-UMT. 2021. Retrieved March 5, 2023, from https://online.simmons.edu/blog/racial-disparities-in-mental-health-treatment/

Thomeer MB, Moody MD, Yahirun J. Racial and ethnic disparities in mental health and mental health care during the COVID-19 pandemic. J Racial Ethn Health Disparities. 2022:1–16. https://doi.org/10.1007/s40615-022-01284-9 . Epub ahead of print. PMID: 35318615; PMCID: PMC8939391.

Download references

Acknowledgements

We thank all of the participants who took part in the SilverCloud study. A preprint version of this manuscript has been published in JMIR Publications and is provided as #15 in the list of references.

SilverCloud did not provide any funding for this project. All funding was provided internally by Eastern Virginia Medical School.

Author information

Authors and affiliations.

EVMS-Sentara Healthcare Analytics and Delivery Science Institute, Norfolk, VA, USA

Tharidu Gunawardena, Marilyn M. Bartholmae, Matvey V. Karpov, Rohan Dod, Kripa Ahuja, Aishwarya Rajendran, Mayuri Kathrotia & Sunita Dodani

Department of Psychiatry and Behavioral Sciences, Eastern Virginia Medical School, Norfolk, VA, USA

Marilyn M. Bartholmae

Center 4 Health Research, University of Illinois College of Medicine Peoria, Peoria, IL, USA

Sunita Dodani

You can also search for this author in PubMed   Google Scholar

Contributions

MB, MK, and SD contributed to the study design, data collection, and data analysis. TG, RD, KA, AR, MK, MB, MK, and SD wrote and revised the main manuscript. All authors contributed to the article and approved the submitted version.

Corresponding author

Correspondence to Tharidu Gunawardena .

Ethics declarations

Ethics approval and consent to participate.

The study was conducted in accordance with the ethical principles laid out in the Declaration of Helsinki. The participants gave informed consent to participate in the study. The study protocol was approved by the Eastern Virginia Medical School institutional review board (IRB# 21–07-FB-0185) and was conducted in compliance with all applicable laws and regulations.

Consent for publication

Not applicable.

Competing interests

None declared.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Gunawardena, T., Bartholmae, M.M., Karpov, M.V. et al. Predictors for interest to participate in digital mental health therapy: a cross-sectional survey of individuals with anxiety and depression. BMC Digit Health 2 , 21 (2024). https://doi.org/10.1186/s44247-024-00080-1

Download citation

Received : 24 September 2023

Accepted : 28 February 2024

Published : 16 May 2024

DOI : https://doi.org/10.1186/s44247-024-00080-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• SilverCloud
• Digital mental health therapy

BMC Digital Health

ISSN: 2731-684X

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Community health survey to shape future public health initiatives underway across WNC

by Rex Hodge

HAYWOOD COUNTY, N.C. (WLOS) — In the coming weeks, Haywood County residents may get a call asking about health behaviors and experiences.

Haywood County Health and Human Services is one of 18 Western North Carolina counties partnering with research firm PRC from Nebraska for a community health assessment survey designed to identify important health needs.

Calls are underway for the anonymous survey, estimated to take about 20 minutes to complete, in which randomly picked households will get a call from WNC Health Network .

It covers various health factors like demographics, socioeconomics, health status, behaviors, clinical care, physical environment and quality of life. The information gathered will help the health department prioritize health challenges in the next three years.

TAKE THE SURVEY: RESIDENTS' INPUT TO HELP MOLD BUNCOMBE COUNTY'S HEALTH STRATEGIES

“I would think we’d see some things about maybe food resources or housing resources," Haywood County Health Director Dr. Sarah Banks said Wednesday, May 15. "The economy is a little hard for people right now. Mental health is a huge initiative. I anticipate that being there again."

She said that was a clear priority from the last survey during COVID.

“I think a lot of people experienced isolation during Covid, so you would think that maybe that’s where mental health and the substance use initiative came from.”

Banks said survey data from the calls -- or from an online survey -- can reinforce the need for mental health programs like the statewide 988 crisis lifeline. More data will aid in targeting.

“How do we reach a veteran population? How do we reach an adolescent population?" Banks said.

'NOT SUSTAINABLE:' PRIMARY CARE DOCTORS BURNED OUT, LEAVING SHORTAGE IN MOUNTAINS

Information will guide substance use disorder programs, like medication-assisted treatment.

“We started seeing those clients in-house again,” Banks said.

Haywood County plays a vital role in the regional community health assessment strategy, led by the WNC Healthy Impact Initiative -- a partnership involving hospitals, health departments and others committed to improving community health in the area. Banks said the goal is to build a healthy and resilient community.

Survey insights also help to identify where funding is needed.

“If there are gaps in funding, that’s where a lot of our grant applications come in,” Banks said.

The collective funding equips stakeholders with invaluable insights to guide targeted public health interventions to help residents in North Carolina's western counties.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Family Community Med
• v.15(1); Jan-Apr 2008

HEALTH NEEDS ASSESSMENT

Ibrahim a. bani.

Department of Family and Community Medicine, Faculty of Medicine, University of Jazan, Jazan, Saudi Arabia

This paper takes a public health approach to briefly examine: (i) the concept of community health care need assessment; (ii) the roles of academic institutions in health needs assessment; (iii) Jazan study to address the health care needs in Jazan region, Saudi Arabia. The methods included an analysis of the literature, distillation of experience from the recently Jazan Health Need Assessment Survey, and WHO reports. The most important perceived health problems in Jazan region are shortage of health care providers, increased prevalence of communicable diseases and poor environmental health. The academic institutions, Ministry of Health and other health care institutions need to work together to look for innovative approaches, especially to increase the awareness of the society on public health issues, and give more support to increase national and regional funding for community based studies.

The findings of the assessment of the health needs of Jazan presented in this review could be utilized as a baseline and reference information for policy formulation, subsequent planning and cost effective intervention programs. It could also be utilized for the curriculum development or review for a community oriented medical schools.

INTRODUCTION

“Health needs assessment” is a process of determining the health and health care needs of any given population or sub-group in an area. It is a complex task requiring epidemiological expertise, the ability to work across organizational boundaries as well as an understanding of, and an ability to engage with all appropriate population groups. The “health needs assessment” process needs to take account of the diversity within these populations. Health improvement programs provide opportunities to engage in such assessments in the national and regional population.

Research has shown that preventive programs can improve community health when properly implemented. 1 There are examples of successful prevention programs in local communities. However, many still have significant challenges, which demonstrate a gap between science and practice. Though in the United States, there are such common strategies (training–programs), to address this issue, outcomes are still unsatisfactory. Building the capacity of the community to implement high quality prevention can help communities achieve positive health outcomes, thereby narrowing the gap between theory and practice. While there is ample research on the efficacy of evidence-based programs, there is little on how to improve community capacity to improve the quality of prevention. What is being proposed is a new model of research: one based on Community Science that improves the latest theoretical understanding of community capacity and evaluates technologies designed to enhance it.

In most developing countries, the evolution of health services has been dominated by Western models of health care 2 . These have rarely taken into account how local people explain illness, seek advice, or use traditional healing methods. The emphasis has been on hospitals and curative care rather than on trying to address local health needs equitably and effectively. Since the Alma Ata declaration on primary health care, more attention has been given to increasing coverage of basic services and preventing common diseases. However, the bias in resource allocation towards secondary care and urban areas remains.

Health care needs are changing and new challenges arising from chronic diseases and HIV infection must be faced. Better coverage of preventive and essential healthcare services has led to greater emphasis on improving the quality of health care to ensure the efficient and judicious use of scarce resources. For example, infant mortality has fallen dramatically in the past two decades through such interventions as oral rehydration for diarrhea and immunization programs. With fewer children dying, there has been greater emphasis on the need to tackle the causes of infant and child morbidity. The size of families can be reduced with the improvement of the availability of family planning.

If health services are to respond to the changing health needs of their local populations, planners and managers would need useful and timely information about the health status of these populations. Some of this information can come from routine data sources or may be collected from large, one-off population studies. Some information can be obtained from community surveys.

COMMUNITY HEALTH NEED ASSESSMENT

Community appraisals describe approaches to needs assessments that emphasize the involvement of local people. A confusing number of terms describe similar methods: rapid evaluation methods, rapid appraisal methods, rapid community surveys, rapid rural appraisal, relaxed rural appraisal, participatory rural appraisal 3 . The development of rapid appraisal methods in the 1980s came in recognition of the time consuming and rigid nature of traditional epidemiological and questionnaire surveys. Experience with these appraisal methods showed that when they were perfectly executed, they provided valuable, reliable, and timely information on health status, knowledge, attitudes, and behaviors. More recently, emphasis has been placed on encouraging people to participate in their own appraisal (for example, participatory rural appraisal) 4 .

The hardest part of any needs assessment is translating the results into policies and practices to elicit and (or) initiate beneficial change. The involvement of health care workers in techniques such as rapid or rural appraisal will encourage changes at an individual level. Local workshops can provide opportunities to review the lessons learnt with other health care workers. If this change is going to be sustainable and adaptable, then the appraisal should be a continuous process with ongoing feedback. Implementation of strategic changes can be facilitated if the policy-makers themselves are active in the process. Active collaboration between communities and researchers is critical in developing appropriate public health research strategies that address community concerns. 5

Partners for Healthy Communities conducted interviews with community members from the ethnically diverse neighborhoods of Central and Southeast Seattle. The results suggest that effective community-researcher collaborations require a paradigm shift from traditional practices to an approach that involves: acknowledging community contributions, recruiting and training minority people to participate in research teams, improving communication, sharing power, and valuing respect and diversity.

ACADEMIC APPROACH TO COMMUNITY HEALTH

Academic institutions have always found it a challenge to persuade community members to participate in academic research projects 6 . Starting an open dialogue is usually the critical first step. To begin this dialogue with community members in Dayton, Ohio, in 1999, staff from Wright State University decided to organize a community forum, “The History of Health in Dayton.” The forum was intended as the first project of a new research organization, Alliance for Research in Community Health (ARCH), established with federal funding from the Health Resources and Services Administration in 1998. ARCH was created as a bridge between the Department of Family Medicine of Wright State University School of Medicine and the Center for Healthy Communities, a health advocacy and service organization committed to health profession education. ARCH's mission is to improve the health of citizens of Dayton through research involving community participation. Through ARCH, community members help researchers define priorities, resolve ethical issues, refine procedures, and interpret results.

Guidelines for participatory research, proposed by the National Primary Care Research Group in 1998 adopted by the alliance, emphasize the importance of open dialogue among researchers, subjects, academics, and community members. The initial response to the forum was enthusiastic, with a majority of community residents expressing interest in attending future presentations.

Barker (1999) described the different ways in which academics and community groups may work together, including academic/practice/ community partnerships. Several principles of practice for engaging in these research partnerships are presented followed by a description of how these principles have been put into operation in a family violence prevention program 7 . The principles presented are: (1) Identification of the best processes/models to be used based on the nature of the issue and the intended outcome; (2) Acknowledgement of most of the differences between community input and active community involvement; (3) Development of relationships based on mutual trust and respect; (4) Acknowledgment and honoring of the different agendas of partners; (5) Consideration of multi-disciplinary approaches; (6) Use of evaluation strategies that are consistent with the overall approach taken in the academic/practice/ community partnership; and (7) Awareness of partnership maturation and associated transition periods. The limitations of these principles and their application in various settings are discussed.

While many members of the public are deeply interested in and supportive of the three traditional missions of academic medicine--education, research, and clinical care, they also want to know what academic health centers (AHCs) are doing to improve the overall health of their communities 8 . Much is already being done toward this goal, but improving communities’ health in a measurable way requires a far broader agenda. AHCs must bring together the approaches of medicine and public health, and need to form partnerships with many other players. This agenda must proceed despite all the other challenges that AHCs face. The author reviewed illustrative and emerging national, state, and local efforts, both public and private; in both medicine and public health, in partnerships with individuals and institutions in the larger community. He also highlights the physician's role in assisting stakeholders’ efforts to deal with health threats from the environment, and offers advice on how such efforts should proceed. He closes by emphasizing the importance of community-based research in learning about the health status, problems, and resources of particular communities; and presents a set of principles for such community-based research.

Lillie-Blanton and Hoffman (1995) examined strategies and methodological issues for researchers to consider when conducting community-based research within a racial/ethnic minority group. 9 Members of minority communities have considerable skepticism about the health care system and researchers who work under its auspices. To facilitate quality research, it is necessary to build a mutually beneficial partnership between the community and researchers. Suggested strategies for accomplishing this, such as searching for information on the social and political forces shaping the community and developing the community's capacity to undertake research of this type, are described. Methodological issues include the importance of community input in defining the minority population group and its leadership, the benefits and limitations of conducting comparative analysis, and the need for measurement tools and techniques that are culturally and socially appropriate. Minority and non-minority researchers must make a concerted effort to understand and have respect for a community whose culture, values, and beliefs may differ.

PRIMARY HEALTH CARE IN SAUDI ARABIA

A recent study reported that nearly a quarter of the children in Riyadh contracted diarrhea during the two weeks preceding the data collection, giving about six episodes of diarrhea per child per year 10 . Diarrhea was more common in children over 6 months of age, in children who had no vaccination or follow-up cards, and in those who were being cared for by friends and neighbors as their mothers were working outside home. The mothers of the affected children were young, married before 25 years of age, with 2–6 years of normal schooling. During diarrheal episodes, about 25% of mothers stopped or reduced breast-feeding, 11.3% reduced the volume of fluids given to their children, and 22.7% of the children were fed less solid/semi-solid foods. Mothers used oral rehydration salt in more than 40% of diarrheal episodes and unprescribed antibiotics were used in 17% of cases. The mothers who were not taking appropriate action included young mothers with low level of education and those working outside the home.

In response to the need for comprehensive, cost-effective and cost-benefit services, there have been major changes in the health care system of Saudi Arabia since the early 1980s. This followed government's commitment to the Alma Ata declaration in 1980. Currently, there are nearly 1800 governmental PHC centers distributed evenly throughout the country, 1707 of which belong to the Ministry of Health (MOH) 11 . The remainder are run by various health care providers, including universities, the military, the National Guard and the Security Forces. Entry into the health care system is through PHC centers. More than 180 secondary and tertiary care hospitals serve as referral units and each group of PHC centers is attached to a hospital.

HEALTH NEED ASSESSMENT: PRIMARY HEALTH CARE APPROACH

A number of themes emerged as important to the impact of health needs assessments on policy and planning 12 . These included careful design, strict methodology, decisive leadership, good communication, involvement and the ownership of the work from relevant stakeholders, support from senior decision-makers, appreciation of political dynamics, and engagement with local priorities, availability of resources and, finally, an element of chance. These themes can be categorized broadly into contextual factors, and quality or robustness of the work. Although this study has demonstrated that there are conditions under which needs assessment are more likely to be effective in terms of its influence on policy and planning, it is clear that it is not central in the decision-making process of the health service, for it remains vulnerable to a range of factors over which those responsible for implementation of the decisions have little or no control.

It has been reported that quality of life was unrelated to satisfaction of services, but was strongly associated with unmet needs of mental and physical health, and of rehabilitation. 13 The quality of life decreased as needs increased. Needs are also strongly related to diagnosis and cognitive functioning. Furthermore, more intensive care settings were provided as needs increased.

Primary health care can meet the needs of the population in an equitable way only if these needs are known. The WHO Regional Office for Europe convened a working group to examine the consequences of the assessment of the health needs of the population at the district level for primary care. 14 The group discussed a useful model for community-oriented primary care (COPC), which involved the delivery of programs tailored to community needs. The group considered needs assessment as the basis for allocating resources, prioritizing the needs of community health programs, planning and evaluating these programs; in the third area, they stressed the need to develop further the model for a community-oriented planning and evaluation cycle (COPEC).

The impact of community needs assessments was used in South Australia where the data was collected from regional health planning officers. 15 The needs assessments were found to vary from the regional to the locally driven. Approaches ensured local involvement, but the process was slower and more arduous for the planner. The use of community health needs assessment was useful, but for greater impact these should not be broad, but focused on feasible changes that the health services could support. Other priority-setting techniques, such as marginal analysis, should be used to determine where it could also be used to determine where maximum health gains are possible.

An area of controversy of need assessment is that which asks the question “whose need”? Several researchers stress the importance of collecting both quantitative and qualitative data from a variety of sources to ensure that community needs are examined from a variety of perspectives 16 , 17 . Other studies have described four types of need that should be considered in needs assessment. These include comparative, normative, expressed and felt. 18 , 19 Indicators of normative needs can be seen as the “vision of health” or benchmarks or targets that are described by experts, task forces, commissions, etc. Indicators of comparative needs include information about the determinants of health for the population as they compare with benchmarks of other populations and other areas of health. Expressed needs are described as information about demands or as ‘wants to put into action’ related to health gathered from key informants, survivors, advocacy groups, government directives, etc. Felt needs are wants or attitudes related to personnel or the community's visions of health, e.g., “we want to feel safe walking alone at night” The proposed framework is based on indicators of needs for each of these dimensions.

Doctors, sociologists, philosophers, and economists all have different views on what needs are. 20 – 22 In recognition of the scarcity of resources available to meet these needs, health needs are often differentiated as needs, demands and supply. Need in health care is commonly defined as the capacity to benefit. If health needs are to be identified then an effective intervention should be available to meet these needs and improve health. There will be no benefit from an intervention that is not effective or if there are no resources. Demand is what patients ask for; it is the need that most doctors encounter. General Practitioners play a key role as gatekeepers in controlling this demand, and waiting lists become a surrogate marker with an influence on this demand. Demand for a service from patients can depend on characteristic of the patient or on the media interest in that service. Supply is the health care provided. This depends on health interests of the professionals, the priorities of politicians, and the amount of money available. The demand and supply of needs overlap. This relationship should be an important consideration when assessing health needs.

A lead article which introduced the community survey concept, detailed reasons for a community survey, and outlined mythological framework for completing such a study. 23 The article defined a community survey as “a survey of population researched by health services provider within a defined geographic area such as hospital service area”. The most critical step in conducting a community survey is for the manager to specify what types of information are needed and how that will be used. To assess health needs of a rural community, it is also important to include a cross-section of health care providers in the “information identification process” to avoid focusing only on services offered by major health care providers (i.e. the local hospital).

Recently, a study reported the awareness of General Practitioners (GP) and their experiences of needs assessment. 24 Most GPs were unfamiliar with the concept of needs assessment, and there was no evidence that needs assessment had influenced commissioning decisions. Most of the GPs argued that it was not a core activity. Besides, they lacked training in the relevant skills. The motivation and attitude of the majority of the GPs is a barrier to needs assessment in primary care. GPs require more resources and training if they are to bear this responsibility.

Over the past 20 years, governments throughout Western Europe and North America have encouraged patients to contribute to the planning and development of health care services. 25 In England and Wales, the involvement of patients is central to current efforts to improve the quality of health care. Underlying these changes, is the belief that involving patients leads to more accessible and acceptable services and the improvement of health and quality of life of patients.

Rapid appraisal can be used to involve the public in the identification of local health needs and supplement more informal methods of assessing needs. 26 Rapid Appraisal is best used in homogenous communities. The process of rapid appraisal can give structured orientation to new workers in the community. Rapid appraisal can be adapted to introduce medical students to the concept of community diagnosis as a natural adjunct to individual clinical diagnosis.

Palmer identified the need for reproductive health care in a community affected by conflict in Southern Sudan. 27 The study comprised interviews with key informants, in-depth interviews, and group discussions. Secondary data were collected. Reproductive health in general, and sexually transmitted diseases in particular were important issues for these communities. Perceptions of reproductive issues varied between service providers and community leaders.

To improve the health of any population or subgroup of that population requires coordinated efforts of many partners in health. These are the health authorities, the local authorities, local business, the voluntary sector, the pharmaceutical industry, and organized groups of the society. 28 Improving health is far more complex and long-term than the provision of health care, as some of the root causes of ill health (poverty, housing, lifestyle, employment and crime) are beyond the control of health services.

JAZAN HEALTH NEED ASSESSMENT

Occupying an area of about 16000 km 2 and with a population of about one million (1421 census), the Jazan region is in the south-western region of Saudi Arabia. It has three geographically distinct zones: the mountain zone which is 2000-2500m above sea level with >300mm of rain/year, the hill zone, 400-600m above sea level with <300mm of rain/year, and the coastal zone that is <400m above sea level with very little rain. The region is intersected by perennial streams.

As a result of its special climate and topography, poor sanitation, inadequate water supply and a middle to low socio-economic status of some communities in the mountainous areas, water borne and water associated diseases such as malaria, schistosomiasis, leishmaniasis, hepatitis, typhoid etc. are prevalent. Tuberculosis is highly endemic. Health statistics from the area indicate high rates of morbidity, mortality & low health care coverage compared to other regions of Saudi Arabia.

The people of Jazan are mostly livestock farmers. The close association and proximity of man and animal has resulted in endemicity of such zoonotic diseases as brucellosis. Jazan city is only 70km from the Yemen border. The 1999 epidemic of the Rift Valley Fever occurred at the border areas as a result of the movements of the border population and the introduction of animals from neighboring countries.

The recent Ministry of Health annual report 29 has shown that the prevalence of infectious diseases in Jazan region as follows: TB 147/2322 (6.3%), Bilharzias 18.35%, Malaria 67.48 %, Hepatitis A 100\2250 (4.4%), and measles 26/617 (4.2%). There are 135 Primary Health Care Centers (PHCC), and 13 hospitals, only two of which are specialized.

Academic public health plays an important role in teaching and research, as well as supporting service departments of public health at national and local levels. 30 The academic-service is a continuum and is essential in providing high quality public health delivery for the nation.

Concerns with health conditions of the population have become a part of a wider concern with the direction of development of human resources. All social and economic development is underpinned by the development of the country's human resources to their full potential. Thus, the training of Saudi personnel in various health fields is important for the success of the Kingdom's plans for health development. The Jazan College of Medicine was therefore, established in 2001.

The expected mission of Jazan Medical College is to raise the standard of health in this area by training health personnel who would be involved in community programs to combat health problems, and work with other health agents in the region in preventive and curative programs.

The Jazan Medical College conducted a comprehensive health survey of Jazan 31 which involved three techniques: key informant interviews, focus group interviews and a household survey.

The key findings were as follows: (1) the most important perceived health problems in Jazan were the shortage of health care providers, and an increased prevalence of communicable diseases and poor environmental health, (2) high level of awareness of communicable diseases with weakness in prevention, (3) most frequently reported chronic conditions: hypertension, bronchial asthma, diabetes mellitus and joint diseases, (4) diarrhea in around 15.6 percent of children, and malaria treatment of 35.9% at the health facility, (5) an estimated 33% of the rural and 37.7% of the urban are current cigarette smokers, and the 61.7% current Khat users in rural compared to 45.7% in the urban areas. The overall Khat use is 48.7%. The data and information generated from this study will be utilized as a baseline and reference information for policy formulation, subsequent planning and cost effective intervention programs, and will be important in the development of the curriculum of Jazan Medical College. It will also help decision makers in the Jazan region in the planning of any future health program.

Though a combination of quantitative and qualitative approaches is recommended in health service research, it has advantages and disadvantages. The incorporation of the view point of the general population through development in the health service has the potential of improving the relevance and impact of research and the quality of subsequent services provided. 32 , 33

A very recent study documented important lessons learned from the use of community key informants in needs assessment surveys to identify health problems and needs. The study suggested the means to control and prevent major health problems in the region of Jazan. The study also emphasized the role of the Jazan medical college in providing skilled manpower and expertise to improve the health care service delivery in the region of Jazan. 34

CONCLUSIONS AND RECOMMENDATIONS

The historical development of health services has been dominated by Western models of health care and health beliefs. 35 These have rarely taken into account how local people explain illness or seek advice. However, the assessment of health needs is not simply a process of listening to patients or reliance on personal experience and anecdotes. It is a systematic method of identifying unmet need and making changes to fill this need. In public health, it is the need of the population, which perhaps could be that of a district or perhaps a section of the population such as women of childbearing age, which have to be assessed.

In conclusion: the academic institutions, Ministry of Health, and other health care institutions should cooperate in looking for innovative approaches to increase awareness of the broader social and public health issues, and increase funding from both regional and national sources to support community based studies.