quantitative research design google scholar

CONCEPTUAL ANALYSIS article

Quantitative and qualitative approaches to generalization and replication–a representationalist view.

• Foundations of Education, University of Bamberg, Bamberg, Germany

In this paper, we provide a re-interpretation of qualitative and quantitative modeling from a representationalist perspective. In this view, both approaches attempt to construct abstract representations of empirical relational structures. Whereas quantitative research uses variable-based models that abstract from individual cases, qualitative research favors case-based models that abstract from individual characteristics. Variable-based models are usually stated in the form of quantified sentences (scientific laws). This syntactic structure implies that sentences about individual cases are derived using deductive reasoning. In contrast, case-based models are usually stated using context-dependent existential sentences (qualitative statements). This syntactic structure implies that sentences about other cases are justifiable by inductive reasoning. We apply this representationalist perspective to the problems of generalization and replication. Using the analytical framework of modal logic, we argue that the modes of reasoning are often not only applied to the context that has been studied empirically, but also on a between-contexts level. Consequently, quantitative researchers mostly adhere to a top-down strategy of generalization, whereas qualitative researchers usually follow a bottom-up strategy of generalization. Depending on which strategy is employed, the role of replication attempts is very different. In deductive reasoning, replication attempts serve as empirical tests of the underlying theory. Therefore, failed replications imply a faulty theory. From an inductive perspective, however, replication attempts serve to explore the scope of the theory. Consequently, failed replications do not question the theory per se , but help to shape its boundary conditions. We conclude that quantitative research may benefit from a bottom-up generalization strategy as it is employed in most qualitative research programs. Inductive reasoning forces us to think about the boundary conditions of our theories and provides a framework for generalization beyond statistical testing. In this perspective, failed replications are just as informative as successful replications, because they help to explore the scope of our theories.

Introduction

Qualitative and quantitative research strategies have long been treated as opposing paradigms. In recent years, there have been attempts to integrate both strategies. These “mixed methods” approaches treat qualitative and quantitative methodologies as complementary, rather than opposing, strategies ( Creswell, 2015 ). However, whilst acknowledging that both strategies have their benefits, this “integration” remains purely pragmatic. Hence, mixed methods methodology does not provide a conceptual unification of the two approaches.

Lacking a common methodological background, qualitative and quantitative research methodologies have developed rather distinct standards with regard to the aims and scope of empirical science ( Freeman et al., 2007 ). These different standards affect the way researchers handle contradictory empirical findings. For example, many empirical findings in psychology have failed to replicate in recent years ( Klein et al., 2014 ; Open Science, Collaboration, 2015 ). This “replication crisis” has been discussed on statistical, theoretical and social grounds and continues to have a wide impact on quantitative research practices like, for example, open science initiatives, pre-registered studies and a re-evaluation of statistical significance testing ( Everett and Earp, 2015 ; Maxwell et al., 2015 ; Shrout and Rodgers, 2018 ; Trafimow, 2018 ; Wiggins and Chrisopherson, 2019 ).

However, qualitative research seems to be hardly affected by this discussion. In this paper, we argue that the latter is a direct consequence of how the concept of generalizability is conceived in the two approaches. Whereas most of quantitative psychology is committed to a top-down strategy of generalization based on the idea of random sampling from an abstract population, qualitative studies usually rely on a bottom-up strategy of generalization that is grounded in the successive exploration of the field by means of theoretically sampled cases.

Here, we show that a common methodological framework for qualitative and quantitative research methodologies is possible. We accomplish this by introducing a formal description of quantitative and qualitative models from a representationalist perspective: both approaches can be reconstructed as special kinds of representations for empirical relational structures. We then use this framework to analyze the generalization strategies used in the two approaches. These turn out to be logically independent of the type of model. This has wide implications for psychological research. First, a top-down generalization strategy is compatible with a qualitative modeling approach. This implies that mainstream psychology may benefit from qualitative methods when a numerical representation turns out to be difficult or impossible, without the need to commit to a “qualitative” philosophy of science. Second, quantitative research may exploit the bottom-up generalization strategy that is inherent to many qualitative approaches. This offers a new perspective on unsuccessful replications by treating them not as scientific failures, but as a valuable source of information about the scope of a theory.

The Quantitative Strategy–Numbers and Functions

Quantitative science is about finding valid mathematical representations for empirical phenomena. In most cases, these mathematical representations have the form of functional relations between a set of variables. One major challenge of quantitative modeling consists in constructing valid measures for these variables. Formally, to measure a variable means to construct a numerical representation of the underlying empirical relational structure ( Krantz et al., 1971 ). For example, take the behaviors of a group of students in a classroom: “to listen,” “to take notes,” and “to ask critical questions.” One may now ask whether is possible to assign numbers to the students, such that the relations between the assigned numbers are of the same kind as the relations between the values of an underlying variable, like e.g., “engagement.” The observed behaviors in the classroom constitute an empirical relational structure, in the sense that for every student-behavior tuple, one can observe whether it is true or not. These observations can be represented in a person × behavior matrix 1 (compare Figure 1 ). Given this relational structure satisfies certain conditions (i.e., the axioms of a measurement model), one can assign numbers to the students and the behaviors, such that the relations between the numbers resemble the corresponding numerical relations. For example, if there is a unique ordering in the empirical observations with regard to which person shows which behavior, the assigned numbers have to constitute a corresponding unique ordering, as well. Such an ordering coincides with the person × behavior matrix forming a triangle shaped relation and is formally represented by a Guttman scale ( Guttman, 1944 ). There are various measurement models available for different empirical structures ( Suppes et al., 1971 ). In the case of probabilistic relations, Item-Response models may be considered as a special kind of measurement model ( Borsboom, 2005 ).

Figure 1 . Constructing a numerical representation from an empirical relational structure; Due to the unique ordering of persons with regard to behaviors (indicated by the triangular shape of the relation), it is possible to construct a Guttman scale by assigning a number to each of the individuals, representing the number of relevant behaviors shown by the individual. The resulting variable (“engagement”) can then be described by means of statistical analyses, like, e.g., plotting the frequency distribution.

Although essential, measurement is only the first step of quantitative modeling. Consider a slightly richer empirical structure, where we observe three additional behaviors: “to doodle,” “to chat,” and “to play.” Like above, one may ask, whether there is a unique ordering of the students with regard to these behaviors that can be represented by an underlying variable (i.e., whether the matrix forms a Guttman scale). If this is the case, we may assign corresponding numbers to the students and call this variable “distraction.” In our example, such a representation is possible. We can thus assign two numbers to each student, one representing his or her “engagement” and one representing his or her “distraction” (compare Figure 2 ). These measurements can now be used to construct a quantitative model by relating the two variables by a mathematical function. In the simplest case, this may be a linear function. This functional relation constitutes a quantitative model of the empirical relational structure under study (like, e.g., linear regression). Given the model equation and the rules for assigning the numbers (i.e., the instrumentations of the two variables), the set of admissible empirical structures is limited from all possible structures to a rather small subset. This constitutes the empirical content of the model 2 ( Popper, 1935 ).

Figure 2 . Constructing a numerical model from an empirical relational structure; Since there are two distinct classes of behaviors that each form a Guttman scale, it is possible to assign two numbers to each individual, correspondingly. The resulting variables (“engagement” and “distraction”) can then be related by a mathematical function, which is indicated by the scatterplot and red line on the right hand side.

The Qualitative Strategy–Categories and Typologies

The predominant type of analysis in qualitative research consists in category formation. By constructing descriptive systems for empirical phenomena, it is possible to analyze the underlying empirical structure at a higher level of abstraction. The resulting categories (or types) constitute a conceptual frame for the interpretation of the observations. Qualitative researchers differ considerably in the way they collect and analyze data ( Miles et al., 2014 ). However, despite the diverse research strategies followed by different qualitative methodologies, from a formal perspective, most approaches build on some kind of categorization of cases that share some common features. The process of category formation is essential in many qualitative methodologies, like, for example, qualitative content analysis, thematic analysis, grounded theory (see Flick, 2014 for an overview). Sometimes these features are directly observable (like in our classroom example), sometimes they are themselves the result of an interpretative process (e.g., Scheunpflug et al., 2016 ).

In contrast to quantitative methodologies, there have been little attempts to formalize qualitative research strategies (compare, however, Rihoux and Ragin, 2009 ). However, there are several statistical approaches to non-numerical data that deal with constructing abstract categories and establishing relations between these categories ( Agresti, 2013 ). Some of these methods are very similar to qualitative category formation on a conceptual level. For example, cluster analysis groups cases into homogenous categories (clusters) based on their similarity on a distance metric.

Although category formation can be formalized in a mathematically rigorous way ( Ganter and Wille, 1999 ), qualitative research hardly acknowledges these approaches. 3 However, in order to find a common ground with quantitative science, it is certainly helpful to provide a formal interpretation of category systems.

Let us reconsider the above example of students in a classroom. The quantitative strategy was to assign numbers to the students with regard to variables and to relate these variables via a mathematical function. We can analyze the same empirical structure by grouping the behaviors to form abstract categories. If the aim is to construct an empirically valid category system, this grouping is subject to constraints, analogous to those used to specify a measurement model. The first and most important constraint is that the behaviors must form equivalence classes, i.e., within categories, behaviors need to be equivalent, and across categories, they need to be distinct (formally, the relational structure must obey the axioms of an equivalence relation). When objects are grouped into equivalence classes, it is essential to specify the criterion for empirical equivalence. In qualitative methodology, this is sometimes referred to as the tertium comparationis ( Flick, 2014 ). One possible criterion is to group behaviors such that they constitute a set of specific common attributes of a group of people. In our example, we might group the behaviors “to listen,” “to take notes,” and “to doodle,” because these behaviors are common to the cases B, C, and D, and they are also specific for these cases, because no other person shows this particular combination of behaviors. The set of common behaviors then forms an abstract concept (e.g., “moderate distraction”), while the set of persons that show this configuration form a type (e.g., “the silent dreamer”). Formally, this means to identify the maximal rectangles in the underlying empirical relational structure (see Figure 3 ). This procedure is very similar to the way we constructed a Guttman scale, the only difference being that we now use different aspects of the empirical relational structure. 4 In fact, the set of maximal rectangles can be determined by an automated algorithm ( Ganter, 2010 ), just like the dimensionality of an empirical structure can be explored by psychometric scaling methods. Consequently, we can identify the empirical content of a category system or a typology as the set of empirical structures that conforms to it. 5 Whereas the quantitative strategy was to search for scalable sub-matrices and then relate the constructed variables by a mathematical function, the qualitative strategy is to construct an empirical typology by grouping cases based on their specific similarities. These types can then be related to one another by a conceptual model that describes their semantic and empirical overlap (see Figure 3 , right hand side).

Figure 3 . Constructing a conceptual model from an empirical relational structure; Individual behaviors are grouped to form abstract types based on them being shared among a specific subset of the cases. Each type constitutes a set of specific commonalities of a class of individuals (this is indicated by the rectangles on the left hand side). The resulting types (“active learner,” “silent dreamer,” “distracted listener,” and “troublemaker”) can then be related to one another to explicate their semantic and empirical overlap, as indicated by the Venn-diagram on the right hand side.

Variable-Based Models and Case-Based Models

In the previous section, we have argued that qualitative category formation and quantitative measurement can both be characterized as methods to construct abstract representations of empirical relational structures. Instead of focusing on different philosophical approaches to empirical science, we tried to stress the formal similarities between both approaches. However, it is worth also exploring the dissimilarities from a formal perspective.

Following the above analysis, the quantitative approach can be characterized by the use of variable-based models, whereas the qualitative approach is characterized by case-based models ( Ragin, 1987 ). Formally, we can identify the rows of an empirical person × behavior matrix with a person-space, and the columns with a corresponding behavior-space. A variable-based model abstracts from the single individuals in a person-space to describe the structure of behaviors on a population level. A case-based model, on the contrary, abstracts from the single behaviors in a behavior-space to describe individual case configurations on the level of abstract categories (see Table 1 ).

Table 1 . Variable-based models and case-based models.

From a representational perspective, there is no a priori reason to favor one type of model over the other. Both approaches provide different analytical tools to construct an abstract representation of an empirical relational structure. However, since the two modeling approaches make use of different information (person-space vs. behavior-space), this comes with some important implications for the researcher employing one of the two strategies. These are concerned with the role of deductive and inductive reasoning.

In variable-based models, empirical structures are represented by functional relations between variables. These are usually stated as scientific laws ( Carnap, 1928 ). Formally, these laws correspond to logical expressions of the form

In plain text, this means that y is a function of x for all objects i in the relational structure under consideration. For example, in the above example, one may formulate the following law: for all students in the classroom it holds that “distraction” is a monotone decreasing function of “engagement.” Such a law can be used to derive predictions for single individuals by means of logical deduction: if the above law applies to all students in the classroom, it is possible to calculate the expected distraction from a student's engagement. An empirical observation can now be evaluated against this prediction. If the prediction turns out to be false, the law can be refuted based on the principle of falsification ( Popper, 1935 ). If a scientific law repeatedly withstands such empirical tests, it may be considered to be valid with regard to the relational structure under consideration.

In case-based models, there are no laws about a population, because the model does not abstract from the cases but from the observed behaviors. A case-based model describes the underlying structure in terms of existential sentences. Formally, this corresponds to a logical expression of the form

In plain text, this means that there is at least one case i for which the condition XYZ holds. For example, the above category system implies that there is at least one active learner. This is a statement about a singular observation. It is impossible to deduce a statement about another person from an existential sentence like this. Therefore, the strategy of falsification cannot be applied to test the model's validity in a specific context. If one wishes to generalize to other cases, this is accomplished by inductive reasoning, instead. If we observed one person that fulfills the criteria of calling him or her an active learner, we can hypothesize that there may be other persons that are identical to the observed case in this respect. However, we do not arrive at this conclusion by logical deduction, but by induction.

Despite this important distinction, it would be wrong to conclude that variable-based models are intrinsically deductive and case-based models are intrinsically inductive. 6 Both types of reasoning apply to both types of models, but on different levels. Based on a person-space, in a variable-based model one can use deduction to derive statements about individual persons from abstract population laws. There is an analogous way of reasoning for case-based models: because they are based on a behavior space, it is possible to deduce statements about singular behaviors. For example, if we know that Peter is an active learner, we can deduce that he takes notes in the classroom. This kind of deductive reasoning can also be applied on a higher level of abstraction to deduce thematic categories from theoretical assumptions ( Braun and Clarke, 2006 ). Similarly, there is an analog for inductive generalization from the perspective of variable-based modeling: since the laws are only quantified over the person-space, generalizations to other behaviors rely on inductive reasoning. For example, it is plausible to assume that highly engaged students tend to do their homework properly–however, in our example this behavior has never been observed. Hence, in variable-based models we usually generalize to other behaviors by means of induction. This kind of inductive reasoning is very common when empirical results are generalized from the laboratory to other behavioral domains.

Although inductive and deductive reasoning are used in qualitative and quantitative research, it is important to stress the different roles of induction and deduction when models are applied to cases. A variable-based approach implies to draw conclusions about cases by means of logical deduction; a case-based approach implies to draw conclusions about cases by means of inductive reasoning. In the following, we build on this distinction to differentiate between qualitative (bottom-up) and quantitative (top-down) strategies of generalization.

Generalization and the Problem of Replication

We will now extend the formal analysis of quantitative and qualitative approaches to the question of generalization and replicability of empirical findings. For this sake, we have to introduce some concepts of formal logic. Formal logic is concerned with the validity of arguments. It provides conditions to evaluate whether certain sentences (conclusions) can be derived from other sentences (premises). In this context, a theory is nothing but a set of sentences (also called axioms). Formal logic provides tools to derive new sentences that must be true, given the axioms are true ( Smith, 2020 ). These derived sentences are called theorems or, in the context of empirical science, predictions or hypotheses . On the syntactic level, the rules of logic only state how to evaluate the truth of a sentence relative to its premises. Whether or not sentences are actually true, is formally specified by logical semantics.

On the semantic level, formal logic is intrinsically linked to set-theory. For example, a logical statement like “all dogs are mammals,” is true if and only if the set of dogs is a subset of the set of mammals. Similarly, the sentence “all chatting students doodle” is true if and only if the set of chatting students is a subset of the set of doodling students (compare Figure 3 ). Whereas, the first sentence is analytically true due to the way we define the words “dog” and “mammal,” the latter can be either true or false, depending on the relational structure we actually observe. We can thus interpret an empirical relational structure as the truth criterion of a scientific theory. From a logical point of view, this corresponds to the semantics of a theory. As shown above, variable-based and case-based models both give a formal representation of the same kinds of empirical structures. Accordingly, both types of models can be stated as formal theories. In the variable-based approach, this corresponds to a set of scientific laws that are quantified over the members of an abstract population (these are the axioms of the theory). In the case-based approach, this corresponds to a set of abstract existential statements about a specific class of individuals.

In contrast to mathematical axiom systems, empirical theories are usually not considered to be necessarily true. This means that even if we find no evidence against a theory, it is still possible that it is actually wrong. We may know that a theory is valid in some contexts, yet it may fail when applied to a new set of behaviors (e.g., if we use a different instrumentation to measure a variable) or a new population (e.g., if we draw a new sample).

From a logical perspective, the possibility that a theory may turn out to be false stems from the problem of contingency . A statement is contingent, if it is both, possibly true and possibly false. Formally, we introduce two modal operators: □ to designate logical necessity, and ◇ to designate logical possibility. Semantically, these operators are very similar to the existential quantifier, ∃, and the universal quantifier, ∀. Whereas ∃ and ∀ refer to the individual objects within one relational structure, the modal operators □ and ◇ range over so-called possible worlds : a statement is possibly true, if and only if it is true in at least one accessible possible world, and a statement is necessarily true if and only if it is true in every accessible possible world ( Hughes and Cresswell, 1996 ). Logically, possible worlds are mathematical abstractions, each consisting of a relational structure. Taken together, the relational structures of all accessible possible worlds constitute the formal semantics of necessity, possibility and contingency. 7

In the context of an empirical theory, each possible world may be identified with an empirical relational structure like the above classroom example. Given the set of intended applications of a theory (the scope of the theory, one may say), we can now construct possible world semantics for an empirical theory: each intended application of the theory corresponds to a possible world. For example, a quantified sentence like “all chatting students doodle” may be true in one classroom and false in another one. In terms of possible worlds, this would correspond to a statement of contingency: “it is possible that all chatting students doodle in one classroom, and it is possible that they don't in another classroom.” Note that in the above expression, “all students” refers to the students in only one possible world, whereas “it is possible” refers to the fact that there is at least one possible world for each of the specified cases.

To apply these possible world semantics to quantitative research, let us reconsider how generalization to other cases works in variable-based models. Due to the syntactic structure of quantitative laws, we can deduce predictions for singular observations from an expression of the form ∀ i : y i = f ( x i ). Formally, the logical quantifier ∀ ranges only over the objects of the corresponding empirical relational structure (in our example this would refer to the students in the observed classroom). But what if we want to generalize beyond the empirical structure we actually observed? The standard procedure is to assume an infinitely large, abstract population from which a random sample is drawn. Given the truth of the theory, we can deduce predictions about what we may observe in the sample. Since usually we deal with probabilistic models, we can evaluate our theory by means of the conditional probability of the observations, given the theory holds. This concept of conditional probability is the foundation of statistical significance tests ( Hogg et al., 2013 ), as well as Bayesian estimation ( Watanabe, 2018 ). In terms of possible world semantics, the random sampling model implies that all possible worlds (i.e., all intended applications) can be conceived as empirical sub-structures from a greater population structure. For example, the empirical relational structure constituted by the observed behaviors in a classroom would be conceived as a sub-matrix of the population person × behavior matrix. It follows that, if a scientific law is true in the population, it will be true in all possible worlds, i.e., it will be necessarily true. Formally, this corresponds to an expression of the form

The statistical generalization model thus constitutes a top-down strategy for dealing with individual contexts that is analogous to the way variable-based models are applied to individual cases (compare Table 1 ). Consequently, if we apply a variable-based model to a new context and find out that it does not fit the data (i.e., there is a statistically significant deviation from the model predictions), we have reason to doubt the validity of the theory. This is what makes the problem of low replicability so important: we observe that the predictions are wrong in a new study; and because we apply a top-down strategy of generalization to contexts beyond the ones we observed, we see our whole theory at stake.

Qualitative research, on the contrary, follows a different strategy of generalization. Since case-based models are formulated by a set of context-specific existential sentences, there is no need for universal truth or necessity. In contrast to statistical generalization to other cases by means of random sampling from an abstract population, the usual strategy in case-based modeling is to employ a bottom-up strategy of generalization that is analogous to the way case-based models are applied to individual cases. Formally, this may be expressed by stating that the observed qualia exist in at least one possible world, i.e., the theory is possibly true:

This statement is analogous to the way we apply case-based models to individual cases (compare Table 1 ). Consequently, the set of intended applications of the theory does not follow from a sampling model, but from theoretical assumptions about which cases may be similar to the observed cases with respect to certain relevant characteristics. For example, if we observe that certain behaviors occur together in one classroom, following a bottom-up strategy of generalization, we will hypothesize why this might be the case. If we do not replicate this finding in another context, this does not question the model itself, since it was a context-specific theory all along. Instead, we will revise our hypothetical assumptions about why the new context is apparently less similar to the first one than we originally thought. Therefore, if an empirical finding does not replicate, we are more concerned about our understanding of the cases than about the validity of our theory.

Whereas statistical generalization provides us with a formal (and thus somehow more objective) apparatus to evaluate the universal validity of our theories, the bottom-up strategy forces us to think about the class of intended applications on theoretical grounds. This means that we have to ask: what are the boundary conditions of our theory? In the above classroom example, following a bottom-up strategy, we would build on our preliminary understanding of the cases in one context (e.g., a public school) to search for similar and contrasting cases in other contexts (e.g., a private school). We would then re-evaluate our theoretical description of the data and explore what makes cases similar or dissimilar with regard to our theory. This enables us to expand the class of intended applications alongside with the theory.

Of course, none of these strategies is superior per se . Nevertheless, they rely on different assumptions and may thus be more or less adequate in different contexts. The statistical strategy relies on the assumption of a universal population and invariant measurements. This means, we assume that (a) all samples are drawn from the same population and (b) all variables refer to the same behavioral classes. If these assumptions are true, statistical generalization is valid and therefore provides a valuable tool for the testing of empirical theories. The bottom-up strategy of generalization relies on the idea that contexts may be classified as being more or less similar based on characteristics that are not part of the model being evaluated. If such a similarity relation across contexts is feasible, the bottom-up strategy is valid, as well. Depending on the strategy of generalization, replication of empirical research serves two very different purposes. Following the (top-down) principle of generalization by deduction from scientific laws, replications are empirical tests of the theory itself, and failed replications question the theory on a fundamental level. Following the (bottom-up) principle of generalization by induction to similar contexts, replications are a means to explore the boundary conditions of a theory. Consequently, failed replications question the scope of the theory and help to shape the set of intended applications.

We have argued that quantitative and qualitative research are best understood by means of the structure of the employed models. Quantitative science mainly relies on variable-based models and usually employs a top-down strategy of generalization from an abstract population to individual cases. Qualitative science prefers case-based models and usually employs a bottom-up strategy of generalization. We further showed that failed replications have very different implications depending on the underlying strategy of generalization. Whereas in the top-down strategy, replications are used to test the universal validity of a model, in the bottom-up strategy, replications are used to explore the scope of a model. We will now address the implications of this analysis for psychological research with regard to the problem of replicability.

Modern day psychology almost exclusively follows a top-down strategy of generalization. Given the quantitative background of most psychological theories, this is hardly surprising. Following the general structure of variable-based models, the individual case is not the focus of the analysis. Instead, scientific laws are stated on the level of an abstract population. Therefore, when applying the theory to a new context, a statistical sampling model seems to be the natural consequence. However, this is not the only possible strategy. From a logical point of view, there is no reason to assume that a quantitative law like ∀ i : y i = f ( x i ) implies that the law is necessarily true, i.e.,: □(∀ i : y i = f ( x i )). Instead, one might just as well define the scope of the theory following an inductive strategy. 8 Formally, this would correspond to the assumption that the observed law is possibly true, i.e.,: ◇(∀ i : y i = f ( x i )). For example, we may discover a functional relation between “engagement” and “distraction” without referring to an abstract universal population of students. Instead, we may hypothesize under which conditions this functional relation may be valid and use these assumptions to inductively generalize to other cases.

If we take this seriously, this would require us to specify the intended applications of the theory: in which contexts do we expect the theory to hold? Or, equivalently, what are the boundary conditions of the theory? These boundary conditions may be specified either intensionally, i.e., by giving external criteria for contexts being similar enough to the ones already studied to expect a successful application of the theory. Or they may be specified extensionally, by enumerating the contexts where the theory has already been shown to be valid. These boundary conditions need not be restricted to the population we refer to, but include all kinds of contextual factors. Therefore, adopting a bottom-up strategy, we are forced to think about these factors and make them an integral part of our theories.

In fact, there is good reason to believe that bottom-up generalization may be more adequate in many psychological studies. Apart from the pitfalls associated with statistical generalization that have been extensively discussed in recent years (e.g., p-hacking, underpowered studies, publication bias), it is worth reflecting on whether the underlying assumptions are met in a particular context. For example, many samples used in experimental psychology are not randomly drawn from a large population, but are convenience samples. If we use statistical models with non-random samples, we have to assume that the observations vary as if drawn from a random sample. This may indeed be the case for randomized experiments, because all variation between the experimental conditions apart from the independent variable will be random due to the randomization procedure. In this case, a classical significance test may be regarded as an approximation to a randomization test ( Edgington and Onghena, 2007 ). However, if we interpret a significance test as an approximate randomization test, we test not for generalization but for internal validity. Hence, even if we use statistical significance tests when assumptions about random sampling are violated, we still have to use a different strategy of generalization. This issue has been discussed in the context of small-N studies, where variable-based models are applied to very small samples, sometimes consisting of only one individual ( Dugard et al., 2012 ). The bottom-up strategy of generalization that is employed by qualitative researchers, provides such an alternative.

Another important issue in this context is the question of measurement invariance. If we construct a variable-based model in one context, the variables refer to those behaviors that constitute the underlying empirical relational structure. For example, we may construct an abstract measure of “distraction” using the observed behaviors in a certain context. We will then use the term “distraction” as a theoretical term referring to the variable we have just constructed to represent the underlying empirical relational structure. Let us now imagine we apply this theory to a new context. Even if the individuals in our new context are part of the same population, we may still get into trouble if the observed behaviors differ from those used in the original study. How do we know whether these behaviors constitute the same variable? We have to ensure that in any new context, our measures are valid for the variables in our theory. Without a proper measurement model, this will be hard to achieve ( Buntins et al., 2017 ). Again, we are faced with the necessity to think of the boundary conditions of our theories. In which contexts (i.e., for which sets of individuals and behaviors) do we expect our theory to work?

If we follow the rationale of inductive generalization, we can explore the boundary conditions of a theory with every new empirical study. We thus widen the scope of our theory by comparing successful applications in different contexts and unsuccessful applications in similar contexts. This may ultimately lead to a more general theory, maybe even one of universal scope. However, unless we have such a general theory, we might be better off, if we treat unsuccessful replications not as a sign of failure, but as a chance to learn.

Author Contributions

MB conceived the original idea and wrote the first draft of the paper. MS helped to further elaborate and scrutinize the arguments. All authors contributed to the final version of the manuscript.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to thank Annette Scheunpflug for helpful comments on an earlier version of the manuscript.

1. ^ A person × behavior matrix constitutes a very simple relational structure that is common in psychological research. This is why it is chosen here as a minimal example. However, more complex structures are possible, e.g., by relating individuals to behaviors over time, with individuals nested within groups etc. For a systematic overview, compare Coombs (1964) .

2. ^ This notion of empirical content applies only to deterministic models. The empirical content of a probabilistic model consists in the probability distribution over all possible empirical structures.

3. ^ For example, neither the SAGE Handbook of qualitative data analysis edited by Flick (2014) nor the Oxford Handbook of Qualitative Research edited by Leavy (2014) mention formal approaches to category formation.

4. ^ Note also that the described structure is empirically richer than a nominal scale. Therefore, a reduction of qualitative category formation to be a special (and somehow trivial) kind of measurement is not adequate.

5. ^ It is possible to extend this notion of empirical content to the probabilistic case (this would correspond to applying a latent class analysis). But, since qualitative research usually does not rely on formal algorithms (neither deterministic nor probabilistic), there is currently little practical use of such a concept.

6. ^ We do not elaborate on abductive reasoning here, since, given an empirical relational structure, the concept can be applied to both types of models in the same way ( Schurz, 2008 ). One could argue that the underlying relational structure is not given a priori but has to be constructed by the researcher and will itself be influenced by theoretical expectations. Therefore, abductive reasoning may be necessary to establish an empirical relational structure in the first place.

7. ^ We shall not elaborate on the metaphysical meaning of possible worlds here, since we are only concerned with empirical theories [but see Tooley (1999) , for an overview].

8. ^ Of course, this also means that it would be equally reasonable to employ a top-down strategy of generalization using a case-based model by postulating that □(∃ i : XYZ i ). The implications for case-based models are certainly worth exploring, but lie beyond the scope of this article.

Agresti, A. (2013). Categorical Data Analysis, 3rd Edn. Wiley Series In Probability And Statistics . Hoboken, NJ: Wiley.

Google Scholar

Borsboom, D. (2005). Measuring the Mind: Conceptual Issues in Contemporary Psychometrics . Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511490026

CrossRef Full Text | Google Scholar

Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qual. Res. Psychol . 3, 77–101. doi: 10.1191/1478088706qp063oa

Buntins, M., Buntins, K., and Eggert, F. (2017). Clarifying the concept of validity: from measurement to everyday language. Theory Psychol. 27, 703–710. doi: 10.1177/0959354317702256

Carnap, R. (1928). The Logical Structure of the World . Berkeley, CA: University of California Press.

Coombs, C. H. (1964). A Theory of Data . New York, NY: Wiley.

Creswell, J. W. (2015). A Concise Introduction to Mixed Methods Research . Los Angeles, CA: Sage.

Dugard, P., File, P., and Todman, J. B. (2012). Single-Case and Small-N Experimental Designs: A Practical Guide to Randomization Tests 2nd Edn . New York, NY: Routledge. doi: 10.4324/9780203180938

Edgington, E., and Onghena, P. (2007). Randomization Tests, 4th Edn. Statistics. Hoboken, NJ: CRC Press. doi: 10.1201/9781420011814

Everett, J. A. C., and Earp, B. D. (2015). A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Front. Psychol . 6:1152. doi: 10.3389/fpsyg.2015.01152

PubMed Abstract | CrossRef Full Text | Google Scholar

Flick, U. (Ed.). (2014). The Sage Handbook of Qualitative Data Analysis . London: Sage. doi: 10.4135/9781446282243

Freeman, M., Demarrais, K., Preissle, J., Roulston, K., and St. Pierre, E. A. (2007). Standards of evidence in qualitative research: an incitement to discourse. Educ. Res. 36, 25–32. doi: 10.3102/0013189X06298009

Ganter, B. (2010). “Two basic algorithms in concept analysis,” in Lecture Notes In Computer Science. Formal Concept Analysis, Vol. 5986 , eds D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, et al. (Berlin, Heidelberg: Springer Berlin Heidelberg), 312–340. doi: 10.1007/978-3-642-11928-6_22

Ganter, B., and Wille, R. (1999). Formal Concept Analysis . Berlin, Heidelberg: Springer Berlin Heidelberg. doi: 10.1007/978-3-642-59830-2

Guttman, L. (1944). A basis for scaling qualitative data. Am. Sociol. Rev . 9:139. doi: 10.2307/2086306

Hogg, R. V., Mckean, J. W., and Craig, A. T. (2013). Introduction to Mathematical Statistics, 7th Edn . Boston, MA: Pearson.

Hughes, G. E., and Cresswell, M. J. (1996). A New Introduction To Modal Logic . London; New York, NY: Routledge. doi: 10.4324/9780203290644

Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., et al. (2014). Investigating variation in replicability. Soc. Psychol. 45, 142–152. doi: 10.1027/1864-9335/a000178

Krantz, D. H., Luce, D., Suppes, P., and Tversky, A. (1971). Foundations of Measurement Volume I: Additive And Polynomial Representations . New York, NY; London: Academic Press. doi: 10.1016/B978-0-12-425401-5.50011-8

Leavy, P. (2014). The Oxford Handbook of Qualitative Research . New York, NY: Oxford University Press. doi: 10.1093/oxfordhb/9780199811755.001.0001

Maxwell, S. E., Lau, M. Y., and Howard, G. S. (2015). Is psychology suffering from a replication crisis? what does “failure to replicate” really mean? Am. Psychol. 70, 487–498. doi: 10.1037/a0039400

Miles, M. B., Huberman, A. M., and Saldaña, J. (2014). Qualitative Data Analysis: A Methods Sourcebook, 3rd Edn . Los Angeles, CA; London; New Delhi; Singapore; Washington, DC: Sage.

Open Science, Collaboration (2015). Estimating the reproducibility of psychological science. Science 349:Aac4716. doi: 10.1126/science.aac4716

Popper, K. (1935). Logik Der Forschung . Wien: Springer. doi: 10.1007/978-3-7091-4177-9

Ragin, C. (1987). The Comparative Method : Moving Beyond Qualitative and Quantitative Strategies . Berkeley, CA: University Of California Press.

Rihoux, B., and Ragin, C. (2009). Configurational Comparative Methods: Qualitative Comparative Analysis (Qca) And Related Techniques . Thousand Oaks, CA: Sage Publications, Inc. doi: 10.4135/9781452226569

Scheunpflug, A., Krogull, S., and Franz, J. (2016). Understanding learning in world society: qualitative reconstructive research in global learning and learning for sustainability. Int. Journal Dev. Educ. Glob. Learn. 7, 6–23. doi: 10.18546/IJDEGL.07.3.02

Schurz, G. (2008). Patterns of abduction. Synthese 164, 201–234. doi: 10.1007/s11229-007-9223-4

Shrout, P. E., and Rodgers, J. L. (2018). Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol . 69, 487–510. doi: 10.1146/annurev-psych-122216-011845

Smith, P. (2020). An Introduction To Formal Logic . Cambridge: Cambridge University Press. doi: 10.1017/9781108328999

Suppes, P., Krantz, D. H., Luce, D., and Tversky, A. (1971). Foundations of Measurement Volume II: Geometrical, Threshold, and Probabilistic Representations . New York, NY; London: Academic Press.

Tooley, M. (Ed.). (1999). Necessity and Possibility. The Metaphysics of Modality . New York, NY; London: Garland Publishing.

Trafimow, D. (2018). An a priori solution to the replication crisis. Philos. Psychol . 31, 1188–1214. doi: 10.1080/09515089.2018.1490707

Watanabe, S. (2018). Mathematical Foundations of Bayesian Statistics. CRC Monographs On Statistics And Applied Probability . Boca Raton, FL: Chapman And Hall.

Wiggins, B. J., and Chrisopherson, C. D. (2019). The replication crisis in psychology: an overview for theoretical and philosophical psychology. J. Theor. Philos. Psychol. 39, 202–217. doi: 10.1037/teo0000137

Keywords: qualitative research, representational measurement, research methodology, modal logic, generalizability, replication crisis

Citation: Borgstede M and Scholz M (2021) Quantitative and Qualitative Approaches to Generalization and Replication–A Representationalist View. Front. Psychol. 12:605191. doi: 10.3389/fpsyg.2021.605191

Received: 11 September 2020; Accepted: 11 January 2021; Published: 05 February 2021.

Reviewed by:

Copyright © 2021 Borgstede and Scholz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Matthias Borgstede, matthias.borgstede@uni-bamberg.de

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Research Methodology, Methods and Design

Cite this chapter.

• Garry Thomas  

537 Accesses

The main aim of this research is to explore and describe if community intelligence has an impact on local neighbourhood policing. The research questions the extent to which community intelligence impacts on local neighbourhood policing, within the context of the National Intelligence Model (MM) and considers a number of other associated objectives, including:

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Similar content being viewed by others

Doing Research on, for and with Police in Canada and Switzerland: Practical and Methodological Insights

Researching Policing from the Perspective of the Policed: Studying Human Rights from Below

Evidence-Based Policing Is Here to Stay: Innovative Research, Meaningful Practice, and Global Reach

Barriball, K.L. and While, A. (1994) ‘Collecting Data Using a Semi-Structured Interview: A Discussion Paper’, Journal of Advanced Nursing , 19(2): 328–335.

Article   Google Scholar  

Bloor, M. (1978) ‘On the Analysis of Observational Data: A Discussion on the Worth and Uses of Inductive Techniques and Respondent Validation’, Sociology , 12(3): 545–552.

Bogen, K. (1996) ‘The Effect of Questionnaire Length on Response Rates: A Review of the Literature’, Proceedings of the Section on Survey Research Methods (American Statistical Association): Volume 91, 1020–1025.

Google Scholar  

Bramel, D. and Friend, R. (1981) ‘Hawthorne, the Myth of the Docile Worker, and CIass Bias in Psychology’, A m eri ca n Psychologist , 36(8): 867–878.

British Sociological Association. (2004) Statement of Ethical Practice for the British Sociological Association , Durham: British Sociological Association (BSA).

Brown, J. (1996) ‘Police Research: Some Critical Issues’, in Leishman, F., Loveday, B. and Savage, S. (Eds) Core Issues in Policing , New York, NY: Longman, 249–263.

Bryman, A. (2006) ‘Integrating Quantitative and Qualitative Research: How Is It Done?’ Qualitative Research , 6(1): 97–113.

Bryman, A. (2012) Social Research Methods (4th ed.), Oxford: Oxford University Press.

Bryman, A., Becker, S. and Sempik, J. (2008) ‘Quality Criteria for Quantitative, Qualitative and Mixed Methods Research: A View from Social Policy’, International Journal of Social Research Methodology , 11(4): 261–276.

Bulmer, M. (1982) ‘When Is Disguise Justified? Alternatives to Covert Participant Observation’, Qualitative Sociology , 5(4): 251–264.

Campbell, D.T. and Fiske, D.W. (1959) ‘Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix’, Psychological Bulletin , 56(2): 81–105.

Church, R.M. (2001) ‘The Effective Use of Secondary Data’, Learning and Motivation , 33: 32–45.

Corti, L., Foster, J. and Thompson, P. (1995) ‘Archiving Qualitative Research Data’, Social Research Update (Autumn 1995), Issue 10. Available at: http://sru.soc.surrey.ac.uk /SRU10.html [Accessed: 12 June 2013].

Council of Europe. (2010) The European Convention on Human Rights (Convention for the Protection of Human Rights and Fundamental Freedoms), as Amended by Protocols Numbers 11 and 14 , Strasbourg: Council of Europe Publishing.

Creswell, J.W. and Miller, D.L. (2000) ‘Determining Validity in Qualitative Inquiry’, Theory into Practice , 39(3): 124–131.

Creswell, J.W. and Piano Clark, V.L. (2007) Designing and Conducting Mixed Methods Research , Thousand Oaks, CA: Sage.

Dale, A., Arber, S. and Proctor, M. (1988) Doing Secondary Analysis , London: Unwin Hyman.

Denzin, N.K. (1989) Interpretive Interactionism , Newbury Park, CA: Sage.

Denzin, N.K. (1978) The Research Act A Theoretical Introduction to Sociological Methods , New York, NY: Praeger.

Denzin, N.K. (2009) The Research Act: A Theoretical Introduction to Sociological Methods (Paperback), Piscataway, NJ: Transaction.

Diener, E. and Crandall, R. (1978) Ethics in Social and Behavioral Research , Chicago, IL: University of Chicago Press.

Diesing, P. (1971) Patterns of Discovery in the Social Sciences , Chicago, IL: AldineAtherton.

Economic and Social Research Council. (2010) Framework for Research Ethics , Swindon: Economic and Social Research Council (ESRC).

Finlay, L. (2002) ‘Negotiating the Swamp: The Opportunity and Challenge of Reflexivity in Research Practice’, Qualitative Research , 2(2): 209–230.

Geertz, C. (1973) The Interpretation of Cultures: Selected Essays , New York, NY: Basic Books.

Golafshani, N. (2003) ‘Understanding Reliability and Validity in Qualitative Research’, The Qualitative Report , 8(4): 597–607.

Gorard, S. (2010) ‘Research Design, as Independent of Methods’, in Tashakkori, A. and Teddlie, C. (Eds) SAGE Handbook of Mixed Methods in Social and Behavioral Research (2nd ed.), Thousand Oaks, CA: Sage.

Grix, J. (2002) ‘Introducing Students to the Generic Terminology of Social Research’, Politics , 22(3): 175–186.

Guba, E.G. and Lincoln, Y.S. (1989) Fourth Generation Evaluation , Newbury Park, CA: Sage.

Guillemin, M. and Gillam, L. (2004) ‘Ethics, Reflexivity, and “Ethically Important Moments” in Research’, Qualitative Inquiry , 10(2): 261–280.

Hakim, C. (1982) Secondary Analysis in Social Research: A Guide to Data Sources and Methods with Examples , London: Allen and Unwin.

Hammersley, M. (1995) The Politics of Social Research , London: Sage.

Hammersley, M. and Atkinson, P. (1995) Ethnography: Principles in Practice (2nd ed.), London: Routledge.

Hammersley, M. and Atkinson, P. (2007) Ethnography: Principles in Practice (3rd ed.), London: Routledge.

Hammersley, M. and Traianou, A. (2012) Ethics and Educational Research , London: British Educational Research Association.

Handwerker, W.P. (2005) ‘Sample Design’, Encyclopedia of Social Measurement , 3: 429–436.

Harrison, J., MacGibbon, L. and Morton, M. (2001) ‘Regimes of Trustworthiness in Qualitative Research: The Rigors of Reciprocity’, Qualitative Inquiry , 7(3): 323–345.

Heaton, J. (2000) Secondary Analysis of Qualitative Data: A Review of the Literature , York: Social Policy Research Unit (SPRU), University of York.

Heaton, J. (2008) ‘Secondary Analysis of Qualitative Data: An Overview’, Historical Social Research , 33(3): 33–45.

Home Office. (1998a) Data Protection Act 1998 , London: The Stationery Office.

Home Office. (1998b) Human Rights Act 1998 , London: The Stationery Office.

Jupp, V. (1989) Methods of Criminological Research , London: Routledge.

Book   Google Scholar  

Kimmel, A.J. (1988) Ethics and Values in Applied Social Research , Newbury Park, CA: Sage.

Kirk, J. and Miller, M.L. (1986) Reliability and Validity in Qualitative Research , Newbury Park, CA: Sage.

Kvale, S. (1996) Interviews: An Introduction to Qualitative Research Interviewing , London: Sage.

Lee, R.M. (1993) Doing Research on Sensitive Topics , London: Sage.

Leming, M.R. (1997) Research and Sampling Designs: Techniques for Evaluating Hypotheses . Available at: http://www.stolaf.edu /people/leming/soc371res/ research.html [Accessed: 19 July 2013].

Lincoln, Y.S. and Guba, E.G. (1985) Naturalistic Inquiry , Newbury Park, CA: Sage.

Lugosi, P. (2006) ‘Between Overt and Covert Research: Concealment and Disclosure in an Ethnographic Study of Commercial Hospitality’, Qualitative Inquiry , 12(3): 541–561.

Lynch, M. (2000) ‘Against Reflexivity as an Academic Virtue and Source of Privileged Knowledge’, Theory Culture and Society , 17(3): 26–54.

MacMillan, K. (1996) Trance-scripts: The Poetics of a Reflexive Guide to Hypnosis and Trance Talk (Unpublished PhD Thesis), Loughborough: Loughborough Llniversity.

Mason, J. (2002) Qualitative Researching (2nd ed.), London: Sage.

Mason, J. (2004) ‘Semistructured Interview’, in Lewis-Beck, M., Bryman, A. and Futing Liao, T. (Eds) The SAGE Encyclopaedia of Social Science Research Methods (Volume 1), London: Sage.

Mertens, D.M. (2005) Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative and Mixed Methods (2nd ed.), Thousand Oaks, CA: Sage.

Morse, J.M., Barrett, M., Mayan, M., Olson, K. and Spiers, J. (2002) ‘Verification Strategies for Establishing Reliability and Validity in Qualitative Research’, International Journal of Qualitative Methods , 1(2): 13–22.

Neuman, W.L. (2000) Social Research Methods: Qualitative and Quantitative Approaches (4th ed.), Boston, MA: Allyn and Bacon.

Newell, R. (1993) ‘Questionnaires’, in Gilbert, N. (Ed.) (1993) Researching Social Life , London: Sage.

Olson, R., Verley, J., Santos, L. and Salas, C. (2004) ‘What We Teach Students About the Hawthorne Studies: A Review of Content Within a Sample of Introductory I-O and OB Textbooks’, The Industrial-Organizational Psychologist , 41(3): 23–39.

Payne, G. and Payne, J. (2004) Key Concepts in Social Research , London: Sage.

Pels, D. (2000) ‘Reflexivity: One Step Up’, Theory Culture and Society , 17(3): 1–25.

Punch, M. (1994) ‘Politics and Ethics in Qualitative Research’, in Denzin, N.K. and Lincoln, Y.S. (Eds) Handbook of Qualitative Research , Thousand Oaks, CA: Sage.

Rogers, C. (2007) Community Intelligence Initiative in ‘A’ Division (BCU): An Independent Evaluation , Pontypridd: University of Glamorgan.

Rogers, C. and Gravelle, J. (2008) Use of the Community Intelligence Diary (CID) System: An Independent Evaluation , Pontypridd: University of Glamorgan.

Scott, J. and Marshall, G. (Eds) (2009) Oxford Dictionary of Sociology (3rd ed. Revised), Oxford: Oxford University Press.

Shenton, A.K. (2004) ‘Strategies for Ensuring Trustworthiness in Qualitative Research Projects’, Education for Information , 22: 63–75.

Sieber, J.E. (1992) Planning Ethically Responsible Research , London: Sage.

Social Research Association. (2003) Ethical Guidelines , London: Social Research Association.

Spicker, P. (2011) ‘Ethical Covert Research’, Sociology , 45(1): 118–133.

Stanley, B., Sieber, J.E. and Melton, G.B. (1987) ‘Empirical Studies of Ethical Issues in Research: A Research Agenda’, American Psychologist , 42(7): 735–741.

Tashakkori, A. and Teddlie, C. (2003) Handbook of Mixed Methods in Social and Behavioral Research , Thousand Oaks, CA: Sage.

Teddlie, C. and Yu, F. (2007) ‘Mixed Methods Sampling: A Typology with Examples’, Journal of Mixed Methods Research , 1(1): 77–100.

Watt, D. (2007) ‘On Becoming a Qualitative Researcher: The Value of Reflexivity’, The Qualitative Report , 12(1): 82–101.

Webb, E.J., Campbell, D.T., Schwartz, R.D. and Sechrest, L. (1966) Unobtrusive Measures: Nonreactive Research in the Social Sciences , Chicago, IL: Rand McNally.

Wiles, R., Heath, S., Crow, G. and Charles, V. (2004) Informed Consent in Social Research: A Literature Review , Southampton: Economic and Social Research Council (ESRQ, National Centre for Research Methods.

Download references

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

University of South Wales, UK

James Gravelle ( Head of Policing and Security ) & Colin Rogers ( Professor of Police Science ) ( Head of Policing and Security ) &  ( Professor of Police Science )

Copyright information

© 2014 Garry Thomas

About this chapter

Thomas, G. (2014). Research Methodology, Methods and Design. In: Gravelle, J., Rogers, C. (eds) Researching the Police in the 21st Century. Palgrave Macmillan, London. https://doi.org/10.1057/9781137357489_4

Download citation

DOI : https://doi.org/10.1057/9781137357489_4

Publisher Name : Palgrave Macmillan, London

Print ISBN : 978-1-137-35747-2

Online ISBN : 978-1-137-35748-9

eBook Packages : Palgrave Social Sciences Collection Social Sciences (R0)

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

De Luca D , Bonadies L , Alonso-Ojembarrena A, et al. Quantitative Lung Ultrasonography to Guide Surfactant Therapy in Neonates Born Late Preterm and Later. JAMA Netw Open. 2024;7(5):e2413446. doi:10.1001/jamanetworkopen.2024.13446

Manage citations:

© 2024

• Permissions

Quantitative Lung Ultrasonography to Guide Surfactant Therapy in Neonates Born Late Preterm and Later

• 1 Division of Pediatrics and Neonatal Critical Care, “A. Béclère” Hospital, AP-HP–Paris Saclay University, Paris, France
• 2 Physiopathology and Therapeutic Innovation Unit–INSERM U999, Paris Saclay University, Paris, France
• 3 Neonatal Intensive Care Unit, Department of Women’s and Children’s Health, University Hospital of Padova and Institute of Pediatric Research “Città della Speranza,” Padua, Italy
• 4 Neonatal Intensive Care Unit, Puerta del Mar University Hospital, Cádiz, Spain
• 5 Biomedical Research and Innovation Institute of Cádiz (INiBICA), Research Unit, Puerta del Mar University Hospital, Cádiz, Spain
• 6 Department of Pediatrics, Division of Neonatal and Developmental Medicine, Stanford University, School of Medicine, Palo Alto, California
• 7 Division of Neonatology, Department of Translational Medical Sciences, University of Naples Federico II, Naples, Italy

Question   Is the neonatal lung ultrasonography score (LUS) equally accurate to predict surfactant need in late preterm through full-term neonates as in neonates born more prematurely?

Findings   In this multicenter diagnostic study of 157 late preterm through full-term neonates with respiratory failure early after birth, the accuracy of LUS in these neonates was similar to that observed in early preterm neonates. An LUS higher than 8 and 4 or lower had the highest global accuracy (replacement test) and highest sensitivity (triage test), respectively.

Meaning   The findings suggest that the LUS can accurately guide surfactant administration in late preterm through full-term neonates with respiratory failure occurring shortly after birth.

Importance   Surfactant administration may be needed in late preterm through full-term neonates, but the pathophysiology of their respiratory failure can be different from that of early preterm neonates. The lung ultrasonography score (LUS) is accurate to guide surfactant replacement in early preterm neonates, but to our knowledge, it has not yet been studied in the late preterm through full-term neonatal population.

Objective   To assess whether LUS is equally accurate to predict surfactant need in late preterm through full-term neonates as in early preterm neonates.

Design, Setting, and Participants   This prospective, international, multicenter diagnostic study was performed between December 2022 and November 2023 in tertiary academic neonatal intensive care units in France, Italy, Spain, and the US. Late preterm through full-term neonates (≥34 weeks’ gestation) with respiratory failure early after birth were enrolled.

Exposure   Point-of-care lung ultrasonography to calculate the neonatal LUS (range, 0-18, with higher scores indicating worse aeration), which was registered in dedicated research databases and unavailable for clinical decision-making.

Main Outcomes and Measures   The main outcomes were the area under the curve (AUC) in receiver operating characteristic analysis and derived accuracy variables, considering LUS as a replacement for other tests (ie, highest global accuracy) and as a triage test (ie, highest sensitivity). Sample size was calculated to assess noninferiority of LUS to predict surfactant need in the study population compared with neonates born more prematurely. Correlations of LUS with the ratio of hemoglobin oxygen saturation as measured by pulse oximetry (SpO 2 ) to fraction of inspired oxygen (FiO 2 ) and with the oxygen saturation index (OSI) were assessed.

Results   A total of 157 neonates (96 [61.1%] male) were enrolled and underwent lung ultrasonography at a median of 3 hours (IQR, 2-7 hours) of life; 32 (20.4%) needed surfactant administration (pretest probability, 20%). The AUC was 0.87 (95% CI, 0.81-0.92). The highest global accuracy and sensitivity were reached for LUS values higher than 8 or 4 or lower, respectively. Subgroup analysis gave similar diagnostic accuracy in neonates born late preterm (AUC, 0.89; 95% CI, 0.81-0.97; n = 111) and early term and later (AUC, 0.84; 95% CI, 0.73-0.96; n = 46). After adjusting for gestational age, LUS was significantly correlated with SpO 2 :FiO 2 (adjusted β, −10.4; 95% CI, −14.0 to −6.7; P  < .001) and OSI (adjusted β, 0.2; 95% CI, 0.1-0.3; P  < .001).

Conclusions and Relevance   In this diagnostic study of late preterm through full-term neonates with respiratory failure early after birth, LUS accuracy to predict surfactant need was not inferior to that observed in earlier preterm neonates. An LUS higher than 8 was associated with highest global accuracy (replacement test), suggesting that it can be used to guide surfactant administration. An LUS value of 4 or lower was associated with the highest sensitivity (triage test), suggesting it is unlikely for this population to need surfactant.

Point-of-care lung ultrasonography is becoming widely used for its ease, noninvasiveness, and accuracy, as it allows a refined diagnosis of the main neonatal respiratory disorders. 1 Ultrasonography findings may be assessed quantitatively using dedicated scores that can guide respiratory interventions, 2 and a specific lung ultrasonography score (LUS) has been validated for neonatal use. 3

Surfactant is a cornerstone of neonatal critical care and is the licensed treatment for respiratory distress syndrome (RDS). 4 , 5 Since the early trials, surfactant administration has been based mainly on fraction of inspired oxygen (FiO 2 ) levels. 6 This policy spread thereafter as other tools to guide surfactant therapy appeared to be either cumbersome or inaccurate and to lack sufficient clinical development. 7 Using FiO 2 thresholds is, however, an oversimplified method since inspired oxygen is only one of the many factors influencing oxygenation. 7 Moreover, oxygenation impairment is the last consequence in the pathophysiology cascade of events 8 ; thus, FiO 2 can increase after the optimal time window for surfactant administration (ie, the first 2-3 hours of life). 9 Despite these drawbacks, FiO 2 thresholds are still widely used to decide whether to administer surfactant. 7

Ultrasonography-guided surfactant administration can allow personalized therapy and reduce delayed treatments. 10 , 11 The LUS has been demonstrated to predict surfactant need in preterm and extremely preterm neonates with RDS treated with continuous positive airway pressure (CPAP), 3 , 12 , 13 and meta-analyses reported the highest global accuracy with cutoff values between 6 and 8. 7 , 14 , 15 Diagnostic accuracy might be different in neonates born more prematurely since RDS prevalence is inversely proportional to gestational age and, in these patients, respiratory failure may have different pathophysiology and be caused by other disorders, including transient tachypnea of the neonate (TTN). Nonetheless, some patients with TTN may also develop concomitant RDS due to relative surfactant deficiency and may benefit from surfactant replacement. 16 In addition, late preterm through full-term neonates represent challenging cases as they are often delivered in level I or II perinatal centers and, in some areas, the occurrence of respiratory failure may demand their transfer to referral neonatal intensive care units (NICUs) for evaluation and surfactant treatment if needed. Though LUS may have the potential to guide surfactant replacement in late preterm through full-term neonates, to our knowledge, it has not yet been systematically studied in this population. We aimed to assess whether LUS is equally accurate to predict surfactant need in late preterm through full-term neonates as it is in neonates born more prematurely.

This was an international, multicenter diagnostic accuracy study conducted in 5 referral NICUs in France, Italy, Spain, and the US between December 2022 and November 2023. The study was pragmatic as it used only data routinely obtained during clinical care that were not changed for study purposes. Ethical approval was granted in each participating center, and if required by local regulations, written or oral parental informed consent was obtained at NICU admission. Data were prospectively collected in a dedicated, secured, and deidentified database for each participating hospital and subsequently merged at Paris Saclay University, which served as the coordinating center. Relevant privacy regulations were respected. Manuscript preparation followed the Standards for Reporting of Diagnostic Accuracy ( STARD ) guideline. 17

Neonates admitted to the NICU within the first 72 hours of life for respiratory failure were consecutively enrolled if their gestational age was 34 weeks or more. Gestational age was considered based on the best obstetric estimate. Respiratory support consisted of nasal mask– or binasal prongs–delivered CPAP set at 5 to 6 cm H 2 O as per local practice and was started when patients had dyspnea (ie, Silverman score ≥1) with need for supplemental oxygen to achieve hemoglobin oxygen saturation as measured by pulse oximetry (SpO 2 ) of 90% or greater. Conversely, when ongoing resuscitation was needed, invasive ventilation was used per local practice. Supplemental oxygen was added when the respiratory support in room air was insufficient to achieve SpO 2 of 90% or greater. The remaining perinatal management was essentially based on current international guidelines. 18 , 19 Surfactant (poractant alfa, 200 mg/kg) was administered when FiO 2 was persistently greater than 0.30, as currently advised. 20 Ultrasonography findings were only considered qualitatively for diagnostic or educational purposes, 1 and LUS was not used to decide surfactant administration or any clinical intervention. The type of respiratory failure was diagnosed according to prespecified integrated consensus criteria based on perinatal history, biology, and clinical evolution according to the Montreux consensus criteria (eTable 1 in Supplement 1 ) 21 and was classified as RDS, TTN, or neonatal acute respiratory distress syndrome (NARDS). Exclusion criteria were major congenital malformations or chromosomal anomalies, air leaks (ie, pneumothorax, pneumomediastinum) preventing comprehensive ultrasonography visualization of the lung parenchyma, surgery during the first week of life, hemodynamic instability (defined as need for any inotrope), congenital surfactant anomalies, pulmonary hypoplasia or congenital lung malformation, persistent pulmonary hypertension (defined as need for nitric oxide or other pulmonary vasodilators), and need for extracorporeal life support.

The LUS was the index test, calculated at NICU admission and always before surfactant administration, if administered. Ultrasonography was performed with microlinear, hockey stick–shaped, high-frequency (15-18 MHz) probes; the machine setting was as previously described. 22 Lung ultrasonography scores were calculated on 6 thoracic areas (3 per each hemithorax [upper and lower anterior and lateral]), assigning to each area a value of 0 to 3 based on classic ultrasonography semiology (0 for normal, 1 for interstitial-alveolar, 2 for severe interstitial-alveolar [ie, white lung] pattern, and 3 for consolidated areas), as originally published. 3 Thus, the score ranges from 0 (best aeration) to 18 (worst aeration). The score was calculated by investigators proficient in the technique (ie, with at least 1 year of lung ultrasonography experience) (L.B., A.A.-O., D.M., I.G.-R., B.L., and L.C.). The LUS was registered in dedicated research databases, which were unavailable for clinical decision-making and were not used to indicate surfactant administration. This strategy was previously applied 3 and considered the best way to mask LUS since perfect blinding was impossible as lung ultrasonography is routinely used in the participating centers. In detail, clinicians not performing ultrasonography were unaware of the LUS, but it was impossible to conceal patient conditions, such as vital monitoring and clinical appearance, to investigators performing ultrasonography. Nonetheless, previous studies have demonstrated equally optimal interobserver agreement for lung ultrasonography interpretation with or without operators’ blinding. 3 , 23 , 24 Lung ultrasonography values of 6 and 8 were the prespecified positive cutoffs, as these have been associated with the highest global accuracy in early preterm neonates 7 , 14 ; thus, we used them to investigate noninferiority in the study population. The reference standard was an FiO 2 level of 0.30, as this is the threshold suggesting surfactant administration in the European guidelines 20 and is widely used.

Preductal SpO 2 was measured with artifact-filtering monitors when the signal was regularly smooth and was registered together with FiO 2 in the patient’s electronic file per local NICU policies. Oxygenation was described using the SpO 2 :FiO 2 ratio and the oxygen saturation index (OSI; calculated as mean airway pressure × FiO 2 :SpO 2 ) assessed at the time of lung ultrasonography. For nonintubated neonates, the CPAP level was considered as the mean airway pressure and leaks were reduced with patient positioning and gentle mouth closure.

The statistical plan was decided before the end of the study and is available in the IRSCTN registry. 25 For an LUS in late preterm through full-term neonates to be as accurate as it is in early preterm neonates, receiver operating characteristic (ROC) analysis should give a similar area under the curve (AUC). We set a target AUC of 0.93 (95% CI, 0.86-0.99), as this was originally found to guide surfactant administration with LUS in preterm neonates. 3 We considered an AUC of 0.80 as the null hypothesis (ie, we considered global accuracy in the study population to be inferior to that in neonates born more prematurely if the AUC was <0.80) since this is the value reported to guide surfactant replacement with FiO 2 in preterm neonates. 7 The proportion of late preterm through full-term neonates with respiratory failure needing surfactant treatment was considered to be 20%, as previously reported. 3 Power was set at 80% and α at 0.05. With these parameters, the needed sample size was a cohort of 145 patients (29 positive cases and 116 negative cases). 26

Clinical characteristics were compared between study participants who did and did not receive surfactant using χ 2 , Fisher exact, t , or Mann-Whitney U tests, as appropriate. The ROC analysis was performed, and derived diagnostic accuracy parameters (sensitivity, specificity, positive and negative likelihood ratios, positive and negative predictive values, global accuracy, and positive and negative posttest probability) were calculated with their 95% CIs.

The AUC (and derived diagnostic parameters) was our main outcome, and we evaluated LUS as a replacement for other tests—that is, with the highest sensitivity and specificity (ie, highest global accuracy) possible. Additionally, we investigated the reliability of LUS as a triage test—that is, with the highest sensitivity irrespective of specificity. 27 The ROC analysis was performed for the whole population and for 2 prespecified subgroups represented by patients born late preterm and early term and later (ie, with gestational age between 34 and 36 6/7 weeks or 37 weeks or more, respectively) to investigate the effect of gestational age on diagnostic accuracy. 27 The AUC was compared between subgroups, with AUCs originally reported in early preterm neonates 3 , 12 and with the summary AUC obtained by a recent meta-analysis 15 using the Hanley method. 26

Finally, the correlation between LUS and oxygenation metrics was investigated with Spearman correlation coefficients and adjusted for gestational age using linear regression. 27 Multicollinearity was evaluated as previously published. 28 Analyses were performed with SPSS, version 29 (IBM Corp) and MedCalc, version 13.3 (MedCalc Software Ltd), and 2-sided P  < .05 was considered significant.

Figure 1 shows the study flowchart; the index test (LUS) and reference standard (FiO 2 ) had no missing or indeterminate data, and the same applied to surfactant data. All patients completed the study. Table 1 and eTable 2 in Supplement 1 give basic population details; 157 neonates were enrolled (mean [SD] gestational age, 35.7 [2.3] weeks; 61 [38.9%] female, 96 [61.1%] male). Patients who needed surfactant had worse oxygenation metrics and LUS compared with those who did not. Lung ultrasonography was conducted at a median of 3 hours (IQR, 2-7 hours) of life. Thirty-two neonates (20.4%) needed surfactant administration (pretest probability, 20%). At the time of ultrasonography, 145 neonates (92.4%) were supported by CPAP and 12 (7.6%) received invasive ventilation. Forty-eight (30.6%), 93 (59.2%), and 16 (10.2%) neonates were diagnosed with RDS, TTN, and NARDS, respectively; 24 (50.0%) with RDS, 2 (2.2%) with TTN, and 6 (37.5%) with NARDS received surfactant. Only 2 neonates (1.3%) developed signs of respiratory failure beyond the first day of life (one at 40 hours and another at 72 hours of life). NARDS was triggered by meconium aspiration and perinatal infection in 12 cases (75.0%) and 4 cases (25.0%), respectively. Surfactant administration occurred at a median postnatal age of 6 hours (IQR, 3-10 hours) of life. All but 2 neonates (1.3%) survived; median NICU stay was 6 days (IQR, 3-12 days).

Figure 2 shows the ROC curve as well as sensitivity and specificity values for the entire study cohort. The AUC was 0.87 (95% CI, 0.81-0.92; P  < .001), and the LUS cutoff associated with both the highest sensitivity and specificity—that is, the highest global accuracy (ie, LUS as a replacement test)—was 8 (Youden index, 0.65). The highest absolute sensitivity values (ie, LUS as a triage test) were reached for an LUS between 0 and 4 (sensitivity, 97%-100%). Table 2 and eTable 3 in Supplement 1 report the diagnostic accuracy parameters for the 2 prespecified cutoff values investigated for using LUS as a replacement test and as a triage test, respectively. A ROC analysis performed excluding patients with NARDS (AUC, 0.89; 95% CI, 0.82-0.96; P  < .001) or those receiving invasive ventilation (AUC, 0.90; 95% CI, 0.84-0.96; P  < .001) gave similar results.

The AUC was not different from that originally described for preterm 3 (AUC, 0.93; 95% CI, 0.86-0.99; comparison test P  = .20) and extremely preterm 12 (AUC, 0.94; 95% CI, 0.90-0.98; comparison test P  = .07) neonates or from the summary AUC reported by a recent meta-analysis 15 (AUC, 0.88; 95% CI, 0.82-0.91; comparison test P  = .74). Subgroup analysis showed a similar diagnostic accuracy in neonates born late preterm (AUC, 0.89; 95% CI, 0.81-0.97; P  < .001; n = 111) and early term and later (AUC, 0.84; 95% CI, 0.73-0.96; P  < .001; n = 46); AUCs were similar between these 2 subgroups ( P  = .45). The LUS was significantly correlated with SpO 2 :FiO 2 (ρ, −0.47; P  < .001) and OSI (ρ, 0.42; P  < .001) and remained so after adjustment for gestational age (SpO 2 :FiO 2 : adjusted β, −10.4; 95% CI, −14.0 to −6.7; P  < .001; OSI: adjusted β, 0.2; 95% CI, 0.1-0.3; P  < .001).

In this study, we found that the diagnostic accuracy of quantitative lung ultrasonography as a replacement test to predict surfactant need in late preterm through full-term neonates was comparable to that shown in patients born more prematurely. We also found that the technique was reliable as a triage test. These results were obtained using the same score and positive cutoff values 3 , 7 , 14 and by comparing results with those previously reported in early preterm neonates. 3 , 7 , 12 , 14 , 15 Our findings also found 8 to be the LUS cutoff associated with the highest global accuracy (replacement test) and indicated that values of 4 or lower had the highest sensitivity (triage test).

These and other characteristics make our findings coherent. The study was performed with a multicenter design including several centers with similarly established ultrasonography proficiency as well as comparable devices and practice. We also applied the best possible methods for a technique that was already embedded in routine clinical care within the point-of-care policy. 29 The subgroup analysis found that gestational age did not significantly influence LUS diagnostic accuracy since similar AUCs were found in late preterm through full-term neonates. Consistently, there was an association between LUS-assessed lung aeration and oxygenation irrespective of patient age, and this is consistent with what our group previously reported in neonates born more prematurely. 3 , 12 , 30 Additionally, ROC analysis was repeated after excluding patients with NARDS, for whom surfactant therapy is off label, and results were unchanged.

To our knowledge, this was the first study specifically dedicated to ultrasonography-guided surfactant administration in late preterm through full-term neonates, and the results are clinically relevant. Surfactant treatment in this population represents an open clinical problem since patients may be affected by different types of respiratory failure with variable surfactant deficiency or dysfunction, and these are difficult to assess at the bedside. 8 Clinical, biologic, or imaging tests available to date have been either cumbersome or inaccurate, leaving surfactant treatment unguided. 8 Quantitative lung ultrasonography is known to describe the lung volume available for gas exchange (ie, lung aeration) and has been validated against a number of techniques. 2 , 31 In particular, lung aeration is correlated with surfactant adsorption early after birth, 32 , 33 and this makes LUS pathobiologically sound to detect surfactant deficiency or dysfunction.

No clear literature guidance is available to date to guide surfactant administration in late preterm through full-term neonates with signs of respiratory failure. Our data indicated that if these patients had an LUS higher than 8, their probability to need surfactant was approximately 2 times higher ( Table 2 ). This finding should inform clinical practice that has been largely based on FiO 2 and clinical monitoring to date; an LUS higher than 8 in a neonate with respiratory failure in the first hours of life may be used with good accuracy to indicate surfactant administration (replacement test) and reduce delayed administration or at least personalize the clinical monitoring. Conversely, using quantitative lung ultrasonography with a lower LUS cutoff—that is, with higher sensitivity—may be useful to rule out surfactant need (triage test); a patient with an LUS of 4 or lower is unlikely to have worsening respiratory failure needing surfactant (eTable 2 in Supplement 1 ). Since LUS calculation is easy and not reliant on the operator’s expertise, its use as a triage test may be particularly important for neonates born in hospitals lacking advanced neonatal care and needing transfer to referral centers. 22 Moreover, LUS calculation is not affected by patient transportation and can also be realized in mobile NICUs. 34 Thus, LUS can help clinicians to reduce subjectivity when making decisions, such as neonatal transportation, that are associated with relevant consequences from the medical and public health perspective. This may be helpful to reserve NICU beds for patients who actually need them and is particularly important during disease outbreaks or resources shortage. 35

This study has limitations. A multicenter design was needed to recruit enough patients since severe respiratory failure is relatively less common in more-mature neonates than in preterm neonates. The shared ultrasonography expertise and technique was also an asset, and since lung ultrasonography is relatively easy to learn, 36 this may facilitate the applicability of our findings. The main limitation was that we used an FiO 2 threshold as the reference standard to identify surfactant need, and this cannot be considered a gold standard for the aforementioned reasons. However, there is no consensus on a gold standard to identify surfactant deficiency or dysfunction at the bedside, and the FiO 2 is the most widely used criterion; thus, our results are pragmatically useful. 8 The studied sample size may seem relatively small but was comparable to that of previous studies recruiting preterm neonates 3 , 12 and respected the targeted sample size calculation. We included neonates who already had signs of respiratory failure (ie, high suspicion index); thus, our data cannot support LUS to be used as a pure screening test in asymptomatic neonates. Previous studies on preterm populations have followed the same design, and surfactant replacement, when realized, occurred few hours after ultrasonography. 3 , 12 This highlights the need for and difficulty of blinding the procedure and having a test quick enough to make clinical decisions in rapidly evolving situations. We acknowledge that our blinding was not perfect, but it was the best we could have provided for the nature of the studied intervention (ie, point-of-care ultrasonography), which unavoidably required observation of the patient. We do not know yet, however, whether LUS accuracy may be improved by repeating the examination—that is, if changes in LUS might be more accurate than a single assessment. Similarly, machine learning and artificial intelligence–assisted interpretation might improve the diagnostic accuracy and make LUS a type of automatized monitoring. Finally, we cannot clarify the effect of more modern respiratory support techniques on LUS, as these are usually reserved for preterm neonates 37 , 38 and were not used in this study population. These issues warrant dedicated clinical studies to be elucidated.

In this study, the diagnostic accuracy of LUS to predict surfactant need in late preterm through full-term neonates with respiratory failure shortly after birth was similar to that observed in preterm neonates. An LUS higher than 8 was associated with the highest global accuracy (replacement test), suggesting it can be used to guide surfactant administration. Neonatal LUS values of 4 or lower were associated with the highest sensitivity (triage test), suggesting an unlikely need for surfactant in this population.

Accepted for Publication: March 25, 2024.

Published: May 28, 2024. doi:10.1001/jamanetworkopen.2024.13446

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 De Luca D et al. JAMA Network Open .

Corresponding Author: Daniele De Luca, MD, PhD, Division of Pediatrics and Neonatal Critical Care, “A. Béclère” Hospital, AP-HP Université Paris Saclay, 157 rue de la Porte de Trivaux, 92140 Clamart, France ( [email protected] ).

Author Contributions: Prof De Luca had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: De Luca, Davis.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Loi, Baraldi.

Critical review of the manuscript for important intellectual content: De Luca, Bonadies, Alonso-Ojembarrena, Martino, Gutierrez-Rosa, Dasani, Capasso, Davis, Raimondi.

Statistical analysis: De Luca.

Administrative, technical, or material support: De Luca, Alonso-Ojembarrena, Gutierrez-Rosa, Dasani, Davis.

Supervision: Baraldi, Davis, Raimondi.

Conflict of Interest Disclosures: Prof De Luca reported receiving personal fees and grants from Chiesi and grants from Airway Therapeutics outside the submitted work. Prof Baraldi reported receiving personal fees from AstraZeneca and Sanofi for meeting lectures and serving on advisory boards outside the submitted work. No other disclosures were reported.

Meeting Presentation: This paper was presented at the 2024 Pediatric Academic Societies Meeting; May 6, 2024; Toronto, Ontario, Canada.

Data Sharing Statement: See Supplement 2 .

Additional Contributions: Maria Rosaria Gualano, MD, iUniCamillus–Saint Camillus International University of Health and Medical Sciences, and Silvia Mongodi, MD, and Victor Sartorius, MD, Fondazione IRCCS Policklinico San Matteo, provided critical manuscript review; no compensation was provided. We thank all the nurses and nurse practitioners who helped in the data collection.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Athl Train
• v.45(1); Jan-Feb 2010

Study/Experimental/Research Design: Much More Than Statistics

Kenneth l. knight.

Brigham Young University, Provo, UT

The purpose of study, experimental, or research design in scientific manuscripts has changed significantly over the years. It has evolved from an explanation of the design of the experiment (ie, data gathering or acquisition) to an explanation of the statistical analysis. This practice makes “Methods” sections hard to read and understand.

To clarify the difference between study design and statistical analysis, to show the advantages of a properly written study design on article comprehension, and to encourage authors to correctly describe study designs.

Description:

The role of study design is explored from the introduction of the concept by Fisher through modern-day scientists and the AMA Manual of Style . At one time, when experiments were simpler, the study design and statistical design were identical or very similar. With the complex research that is common today, which often includes manipulating variables to create new variables and the multiple (and different) analyses of a single data set, data collection is very different than statistical design. Thus, both a study design and a statistical design are necessary.

Advantages:

Scientific manuscripts will be much easier to read and comprehend. A proper experimental design serves as a road map to the study methods, helping readers to understand more clearly how the data were obtained and, therefore, assisting them in properly analyzing the results.

Study, experimental, or research design is the backbone of good research. It directs the experiment by orchestrating data collection, defines the statistical analysis of the resultant data, and guides the interpretation of the results. When properly described in the written report of the experiment, it serves as a road map to readers, 1 helping them negotiate the “Methods” section, and, thus, it improves the clarity of communication between authors and readers.

A growing trend is to equate study design with only the statistical analysis of the data. The design statement typically is placed at the end of the “Methods” section as a subsection called “Experimental Design” or as part of a subsection called “Data Analysis.” This placement, however, equates experimental design and statistical analysis, minimizing the effect of experimental design on the planning and reporting of an experiment. This linkage is inappropriate, because some of the elements of the study design that should be described at the beginning of the “Methods” section are instead placed in the “Statistical Analysis” section or, worse, are absent from the manuscript entirely.

Have you ever interrupted your reading of the “Methods” to sketch out the variables in the margins of the paper as you attempt to understand how they all fit together? Or have you jumped back and forth from the early paragraphs of the “Methods” section to the “Statistics” section to try to understand which variables were collected and when? These efforts would be unnecessary if a road map at the beginning of the “Methods” section outlined how the independent variables were related, which dependent variables were measured, and when they were measured. When they were measured is especially important if the variables used in the statistical analysis were a subset of the measured variables or were computed from measured variables (such as change scores).

The purpose of this Communications article is to clarify the purpose and placement of study design elements in an experimental manuscript. Adopting these ideas may improve your science and surely will enhance the communication of that science. These ideas will make experimental manuscripts easier to read and understand and, therefore, will allow them to become part of readers' clinical decision making.

WHAT IS A STUDY (OR EXPERIMENTAL OR RESEARCH) DESIGN?

The terms study design, experimental design, and research design are often thought to be synonymous and are sometimes used interchangeably in a single paper. Avoid doing so. Use the term that is preferred by the style manual of the journal for which you are writing. Study design is the preferred term in the AMA Manual of Style , 2 so I will use it here.

A study design is the architecture of an experimental study 3 and a description of how the study was conducted, 4 including all elements of how the data were obtained. 5 The study design should be the first subsection of the “Methods” section in an experimental manuscript (see the Table ). “Statistical Design” or, preferably, “Statistical Analysis” or “Data Analysis” should be the last subsection of the “Methods” section.

Table. Elements of a “Methods” Section

The “Study Design” subsection describes how the variables and participants interacted. It begins with a general statement of how the study was conducted (eg, crossover trials, parallel, or observational study). 2 The second element, which usually begins with the second sentence, details the number of independent variables or factors, the levels of each variable, and their names. A shorthand way of doing so is with a statement such as “A 2 × 4 × 8 factorial guided data collection.” This tells us that there were 3 independent variables (factors), with 2 levels of the first factor, 4 levels of the second factor, and 8 levels of the third factor. Following is a sentence that names the levels of each factor: for example, “The independent variables were sex (male or female), training program (eg, walking, running, weight lifting, or plyometrics), and time (2, 4, 6, 8, 10, 15, 20, or 30 weeks).” Such an approach clearly outlines for readers how the various procedures fit into the overall structure and, therefore, enhances their understanding of how the data were collected. Thus, the design statement is a road map of the methods.

The dependent (or measurement or outcome) variables are then named. Details of how they were measured are not given at this point in the manuscript but are explained later in the “Instruments” and “Procedures” subsections.

Next is a paragraph detailing who the participants were and how they were selected, placed into groups, and assigned to a particular treatment order, if the experiment was a repeated-measures design. And although not a part of the design per se, a statement about obtaining written informed consent from participants and institutional review board approval is usually included in this subsection.

The nuts and bolts of the “Methods” section follow, including such things as equipment, materials, protocols, etc. These are beyond the scope of this commentary, however, and so will not be discussed.

The last part of the “Methods” section and last part of the “Study Design” section is the “Data Analysis” subsection. It begins with an explanation of any data manipulation, such as how data were combined or how new variables (eg, ratios or differences between collected variables) were calculated. Next, readers are told of the statistical measures used to analyze the data, such as a mixed 2 × 4 × 8 analysis of variance (ANOVA) with 2 between-groups factors (sex and training program) and 1 within-groups factor (time of measurement). Researchers should state and reference the statistical package and procedure(s) within the package used to compute the statistics. (Various statistical packages perform analyses slightly differently, so it is important to know the package and specific procedure used.) This detail allows readers to judge the appropriateness of the statistical measures and the conclusions drawn from the data.

STATISTICAL DESIGN VERSUS STATISTICAL ANALYSIS

Avoid using the term statistical design . Statistical methods are only part of the overall design. The term gives too much emphasis to the statistics, which are important, but only one of many tools used in interpreting data and only part of the study design:

The most important issues in biostatistics are not expressed with statistical procedures. The issues are inherently scientific, rather than purely statistical, and relate to the architectural design of the research, not the numbers with which the data are cited and interpreted. 6

Stated another way, “The justification for the analysis lies not in the data collected but in the manner in which the data were collected.” 3 “Without the solid foundation of a good design, the edifice of statistical analysis is unsafe.” 7 (pp4–5)

The intertwining of study design and statistical analysis may have been caused (unintentionally) by R.A. Fisher, “… a genius who almost single-handedly created the foundations for modern statistical science.” 8 Most research did not involve statistics until Fisher invented the concepts and procedures of ANOVA (in 1921) 9 , 10 and experimental design (in 1935). 11 His books became standard references for scientists in many disciplines. As a result, many ANOVA books were titled Experimental Design (see, for example, Edwards 12 ), and ANOVA courses taught in psychology and education departments included the words experimental design in their course titles.

Before the widespread use of computers to analyze data, designs were much simpler, and often there was little difference between study design and statistical analysis. So combining the 2 elements did not cause serious problems. This is no longer true, however, for 3 reasons: (1) Research studies are becoming more complex, with multiple independent and dependent variables. The procedures sections of these complex studies can be difficult to understand if your only reference point is the statistical analysis and design. (2) Dependent variables are frequently measured at different times. (3) How the data were collected is often not directly correlated with the statistical design.

For example, assume the goal is to determine the strength gain in novice and experienced athletes as a result of 3 strength training programs. Rate of change in strength is not a measurable variable; rather, it is calculated from strength measurements taken at various time intervals during the training. So the study design would be a 2 × 2 × 3 factorial with independent variables of time (pretest or posttest), experience (novice or advanced), and training (isokinetic, isotonic, or isometric) and a dependent variable of strength. The statistical design , however, would be a 2 × 3 factorial with independent variables of experience (novice or advanced) and training (isokinetic, isotonic, or isometric) and a dependent variable of strength gain. Note that data were collected according to a 3-factor design but were analyzed according to a 2-factor design and that the dependent variables were different. So a single design statement, usually a statistical design statement, would not communicate which data were collected or how. Readers would be left to figure out on their own how the data were collected.

MULTIVARIATE RESEARCH AND THE NEED FOR STUDY DESIGNS

With the advent of electronic data gathering and computerized data handling and analysis, research projects have increased in complexity. Many projects involve multiple dependent variables measured at different times, and, therefore, multiple design statements may be needed for both data collection and statistical analysis. Consider, for example, a study of the effects of heat and cold on neural inhibition. The variables of H max and M max are measured 3 times each: before, immediately after, and 30 minutes after a 20-minute treatment with heat or cold. Muscle temperature might be measured each minute before, during, and after the treatment. Although the minute-by-minute data are important for graphing temperature fluctuations during the procedure, only 3 temperatures (time 0, time 20, and time 50) are used for statistical analysis. A single dependent variable H max :M max ratio is computed to illustrate neural inhibition. Again, a single statistical design statement would tell little about how the data were obtained. And in this example, separate design statements would be needed for temperature measurement and H max :M max measurements.

As stated earlier, drawing conclusions from the data depends more on how the data were measured than on how they were analyzed. 3 , 6 , 7 , 13 So a single study design statement (or multiple such statements) at the beginning of the “Methods” section acts as a road map to the study and, thus, increases scientists' and readers' comprehension of how the experiment was conducted (ie, how the data were collected). Appropriate study design statements also increase the accuracy of conclusions drawn from the study.

CONCLUSIONS

The goal of scientific writing, or any writing, for that matter, is to communicate information. Including 2 design statements or subsections in scientific papers—one to explain how the data were collected and another to explain how they were statistically analyzed—will improve the clarity of communication and bring praise from readers. To summarize:

• Purge from your thoughts and vocabulary the idea that experimental design and statistical design are synonymous.
• Study or experimental design plays a much broader role than simply defining and directing the statistical analysis of an experiment.
• A properly written study design serves as a road map to the “Methods” section of an experiment and, therefore, improves communication with the reader.
• Study design should include a description of the type of design used, each factor (and each level) involved in the experiment, and the time at which each measurement was made.
• Clarify when the variables involved in data collection and data analysis are different, such as when data analysis involves only a subset of a collected variable or a resultant variable from the mathematical manipulation of 2 or more collected variables.

Acknowledgments

Thanks to Thomas A. Cappaert, PhD, ATC, CSCS, CSE, for suggesting the link between R.A. Fisher and the melding of the concepts of research design and statistics.

• Open access
• Published: 23 May 2024

Identifying potential drug targets for idiopathic pulmonary fibrosis: a mendelian randomization study based on the druggable genes

• Zetao Liu 1 , 2 ,
• Zhiyu Peng 1 , 2 ,
• Huahang Lin 1 , 2 ,
• Ke Zhou 1 , 2 ,
• Linchuan Liang 1 , 2 ,
• Jie Cao 1 , 2 ,
• Zhaokang Huang 1 , 2 &
• Jiandong Mei 1 , 2  

Respiratory Research volume  25 , Article number:  217 ( 2024 ) Cite this article

354 Accesses

Metrics details

Idiopathic pulmonary fibrosis (IPF) is a chronic fibrotic interstitial lung disease characterized by progressive dyspnea and decreased lung function, yet its exact etiology remains unclear. It is of great significance to discover new drug targets for IPF.

We obtained the cis-expression quantitative trait locus (cis-eQTL) of druggable genes from eQTLGen Consortium as exposure and the genome wide association study (GWAS) of IPF from the International IPF Genetics Consortium as outcomes to simulate the effects of drugs on IPF by employing mendelian randomization analysis. Then colocalization analysis was performed to calculate the probability of both cis-eQTL of druggable genes and IPF sharing a causal variant. For further validation, we conducted protein quantitative trait locus (pQTL) analysis to reaffirm our findings.

The expression of 45 druggable genes was significantly associated with IPF susceptibility at FDR < 0.05. The expression of 23 and 15 druggable genes was significantly associated with decreased forced vital capacity (FVC) and diffusing capacity of the lungs for carbon monoxide (DLco) in IPF patients, respectively. IPF susceptibility and two significant genes ( IL-7 and ABCB2 ) were likely to share a causal variant. The results of the pQTL analysis demonstrated that high levels of IL-7 in plasma are associated with a reduced risk of IPF (OR = 0.67, 95%CI: 0.47–0.97).

IL-7 stands out as the most promising potential drug target to mitigate the risk of IPF. Our study not only sheds light on potential drug targets but also provides a direction for future drug development in IPF.

Introduction

Idiopathic pulmonary fibrosis (IPF) is a chronic progressive fibrotic interstitial lung disease with poor prognosis characterized by progressive dyspnea and decline in lung function [ 1 ]. In the past two decades, the incidence of IPF has increased, especially among the elderly [ 2 ]. Unfortunately, the exact etiology and pathogenesis of IPF remain elusive, with potential risk factors including genetic variations, long-term exposure to air pollution, smoking, certain viral infections, and gastroesophageal reflux disease [ 3 ]. Although anti-fibrotic drugs like pirfenidone and nintedanib, recommended by current guidelines, have displayed modest ability in slowing disease progression, halting or reversing the process of IPF remains a challenge [ 4 ]. Thus, the identification of novel drug targets capable of preventing IPF or delaying its progression assumes paramount significance.

Mendelian randomization (MR) is an approach that employs genetic variants associated with specific exposures as instrumental variables to estimate causal relationships between the exposure of interest and the desired outcome (Fig.  1 ) [ 4 ]. Guided by the laws of gene segregation and independent assortment, alleles segregate and genes on non-homologous chromosomes recombine freely during gamete formation. Subsequently, the combination of parental gametes determines the presence or absence of certain genes, facilitating the random distribution of lifetime-long exposures. MR analysis has similar power to randomized controlled trial (RCT) with less bias and no reverse causality [ 5 ].

Overview of MR analysis. Choose cis-eQTL of druggable genes as instrumental variables (IVs) to investigate the causal relationship between the expression of druggable genes and IPF. The “X” between IVs and confounders indicates that the IVs are independent of any confounding factors

The “X” between IVs and outcome indicates that the IVs only affect the outcome through the exposure rather than other potential pathways. IV, instrumental variable; SNP, single-nucleotide polymorphisms; IPF, Idiopathic pulmonary fibrosis

In drug target MR analysis, single nucleotide polymorphisms (SNPs) associated with gene expression levels, known as expression quantitative trait loci (eQTL), are employed as instrumental variables to examine the effects of druggable genes. Specifically, cis-eQTLs in the genomic regions proximal to the target gene are often selected due to their close relationship with gene expression. This methodology has garnered widespread application across various diseases, including Parkinson’s disease, aortic aneurysms, and even the COVID-19 [ 6 , 7 , 8 ].

Building upon this foundation, the present study aims to leverage the power of MR analysis to unearth potential drug targets for IPF from a pool of 4,479 druggable genes encoding drug targets or proteins related to drug targets through MR method, whether to prevent disease or delay the progression.

Study design

The flowchart visually describing the overall of the study is shown in Fig.  2 . In short, we performed a two-sample MR analysis utilizing cis-eQTL of druggable genes in the blood as exposure and the genome wide association study (GWAS) of IPF as outcomes to investigate the causal relationship between the expression of druggable genes and susceptibility and progression of IPF. According to strict inclusion and exclusion criteria, appropriate SNPs were selected as instrumental variables (IVs). A series of sensitivity analyses was conducted to control the quality of MR analysis. For the druggable genes that exhibited significant MR results, we performed colocalization analysis to assess whether the same causal variant was shared by both the cis-eQTL and IPF. Additionally, we conducted protein quantitative trait locus (pQTL) analysis, which provided further validation of these druggable genes by examining the effect of the protein levels on the outcome.

The flowchart of our study design. MR, mendelian randomization; eQTL, expression quantitative trait locus; pQTL, protein quantitative trait locus; FVC, forced vital capacity; DLco, diffusing capacity of the lungs for carbon monoxide

Exposure data

The druggable genes are defined as a set of genes encoding proteins with potential to be modulated by a drug-like small molecule based on sequence and structural similarity to the targets of existing drugs [ 9 ]. A total of 4,479 druggable genes were identified by Finan et al. including 1,427 genes encoding approved or clinical-phase drug targets, 682 genes encoding proteins that bind to known drug molecules or are similar to approved drug targets and 2,370 genes that were members of key druggable gene families or encoding proteins with distant similarity to approved drug targets [ 9 ]. This diverse collection of druggable genes offered a wide range of potential targets for investigation (Supplementary material: Table S2 ).

The cis-eQTL data in the blood for only 2,525 genes out of 4,479 druggable genes was obtained by searching in eQTLGen Consortium [ 10 ]. This consortium incorporates 37 datasets with a total of 31,684 individuals, predominantly of European ancestry. The eQTL data facilitates the identification of genetic variants associated with gene expression levels in blood samples, situated within a 1 Mb distance from the central location of each gene. The minor allele frequency (MAF) of every variant is greater than 0.01.

The pQTL data was available from the INTERVAL study encompassing 3,301 healthy participants of European descent [ 11 ]. In this study, a total of 1,927 pQTLs about 1,478 plasma proteins were identified. We selected the pQTL for druggable genes significantly colocalized with IPF outcomes to further investigate the relationship between levels of protein encoded by druggable genes and outcomes.

Instrumental variables (IVs) selection

To ensure the reliability and accuracy of our results, it is crucial to satisfy three important assumptions in MR analysis: (1) The IVs are strongly associated with exposure; (2) The IVs are independent of any confounding factors; (3) There is no presence of horizontal pleiotropy, meaning that the IVs only affect the outcome through the exposure and not through any other potential pathways.

In line with these assumptions, a rigorous selection process was implemented for each druggable gene in our study. Firstly, we employed a stringent threshold and selected SNPs from the cis-eQTL data, ensuring that only those with P -values lower than the genome-wide significance threshold (5.0 × 10 − 8 ) were considered. Next, in order to achieve a set of mutually independent SNPs, the SNPs for every druggable genes were clumped based on the 1,000 Genomes Project European population and the linkage disequilibrium (LD) threshold was set to r 2  < 0.1 with a clumping window of 10,000kb [ 12 ]. Thirdly, incompatible SNPs between the exposures and outcomes (e.g., A/G vs. A/C) were excluded and positive strand alleles were inferred using allele frequencies for palindromes or the palindromic SNPs were excluded directly if there were no allele frequencies. Finally, the following formula was used to calculate the F -statistic [ 13 ].

The F -statistic serve as an essential metric in MR analysis, determining the strength of the IVs’ association with the exposure variable and aiding in the assessment of possible bias or weak instrument issues. In this formula, R 2 is the proportion of variance explained by the IVs, N is the sample size, and k is the number of IVs. The SNPs with F -statistic less than 20 were excluded to avoid weak instrument bias [ 13 ].

Outcome data

The GWAS statistics for IPF susceptibility and progression were obtained from the International IPF Genetics Consortium. For the GWAS of IPF susceptibility, a meta-analysis was conducted across five studies, comprising a total of 4,125 cases and 20,464 controls [ 14 ]. For the GWAS of IPF progression, two key measurements, namely forced vital capacity (FVC) and diffusing capacity of the lungs for carbon monoxide (DLco), were employed to identify variants that may contribute to a more rapid decline in lung capacity or gas transfer among IPF patients. There were 1,048 cases a total of 4,560 FVC measures and 729 cases with a total of 2,795 DLco measures [ 15 ].

Mendelian randomization and colocalization

MR analysis was conducted using the R package “TwoSampleMR” (version 0.5.6). For the MR analysis, Wald ratio method was used when there was only one SNP as the IV. And inverse variance weighted (IVW), MR-Egger, weighted median, simple mode and weighted mode five methods were utilized if the IV contained two or more SNPs. Previous research has indicated that the IVW method is more conservative but robust compared to the other four methods [ 16 ]. Therefore, whether or not there is heterogeneity, the results were mainly based on the IVW method, supplemented by the others. To account for multiple testing, FDR (false discovery rate) corrections were applied to identify significant MR results.

Then the sensitivity analysis was performed by several methods. The potential heterogeneity of IVs was examined by Cochrane’s Q test [ 17 ]. If the P -value of Cochrane’s Q test was less than 0.05, it was indicative of heterogeneity. And MR-Egger regression was used to detect potential pleiotropy in the association between the exposures and outcomes [ 17 ]. If the P -value of MR-Egger regression intercept was less than 0.05, it suggested the presence of pleiotropy and rendered the MR analysis results unreliable.

For the druggable genes exhibiting significant MR results, colocalization analysis was conducted using R package “coloc” (version 5.1.0.1) [ 18 ]. The default prior probability was P 1 = 1.0 × 10 − 4 , P 2 = 1.0 × 10 − 4 , P 12 = 1.0 × 10 − 5 , representing respectively a SNP is associated with the expression of the druggable genes, the outcome, or both. The posterior probabilities for the following 5 hypotheses were generated from colocation analysis: PPH0, no association with either expression of the druggable genes or outcome; PPH1, association with expression of the druggable genes, but not outcome; PPH2, association with outcome, but not expression of the druggable genes; PPH3, association with expression of the druggable genes and outcome, with different causal variants; PPH4, association with expression of the druggable genes and outcome, with a shared causal variant. PPH4 > 0.80 was considered strong evidence for colocalization and the genes colocalized with IPF were regarded as potential targets. The variant most closely associated with exposure (with the lowest P -value) was selected as the reference variant and variants ± 500 kb of the reference variant were included in colocalization analysis.

According to the selection criteria of IVs, a total of 4,0356 SNPs were used as IVs for 2,525 druggable genes. The F -statistic of IVs all exceeded 20, indicating no evidence of weak instrument bias. Details about the IVs are shown in Supplementary material: Table S3 .

• Mendelian randomization

Based on the IVW method, we found the expression of 45 druggable genes was significantly associated with IPF susceptibility at FDR < 0.05. The expression of 23 and 15 druggable genes was significantly associated with decreased FVC and DLco levels in IPF patients, respectively (Figs.  3 and 4 ).

Significant MR results between the expression of druggable genes and IPF susceptibility after FDR correction

Significant MR results between the expression of druggable genes and DLco decline after FDR correction

The results of Cochran’s Q test showed no heterogeneity in IVs for significant genes (Supplementary material: Table S4-6 ). Furthermore, for some significant genes, pleiotropy was detected by MR-Egger regression methods and the corresponding results for these genes were considered unreliable (Supplementary material: Table S4-6 ).

Colocalization

For the druggable genes with significant MR results, we conducted colocalization analysis to calculate probability of cis-eQTL and IPF outcomes sharing a causal variant. The results of colocalization analysis indicated IPF susceptibility and two significant genes ( IL-7 and ABCA2 ) were likely to share a causal variant, with a posterior probability of PP.H4 > 0.80% ( IL-7 : 84.00%, ABCA2 : 81.50%). But there was no evidence of colocalization between IPF progression and the significant genes (Supplementary material: Table S7 ). Therefore, IL-7 and ABCA2 were identified as potential drug targets for reducing IPF risk based on MR and colocalization analyses.

pQTL analysis for  IL-7

To verify the effect of druggable gene expression on IPF susceptibility, we further investigated plasma protein levels using pQTL data. The pQTL data for IL-7 was obtained from the INTERVAL study. Unfortunately, we could not find any pQTL data for ABCA2 .

We filtered out the SNPs with P -values less than the genome-wide significance threshold and clumped with r 2  < 0.001 and clumping window of 10,000 kb. Only one SNP (rs72673751) was screened as IV representing IL-7 protein level for pQTL analysis (Supplementary material: Table S8 ). To ensure the validity of result, we searched on PhenoScanner website to exclude the existence of pleiotropy which could affect outcome through potentially other pathways.

The results of the pQTL analysis demonstrated that high levels of IL-7 in plasma are associated with a reduced risk of IPF (OR = 0.67, 95%CI: 0.47–0.97, P  = 0.035), which is consistent with the findings of eQTL analysis (Supplementary material: Table S9 ).

In order to identify potential drug targets for IPF, we conducted a large-scale MR analysis to evaluate the role of 2,429 druggable gene expression in IPF susceptibility and progression. After a series of sensitivity analyzes and further analyses, including Cochrane’s Q test, MR-Egger regression, colocalization analysis, pQTL analysis, we have discovered that IL-7 holds the most promising potential as a therapeutic target for IPF susceptibility. However, it is important to note that the therapeutic effect of IL-7 was not replicated in the IPF progression cohort.

Although the pathogenesis of IPF has not been fully elucidated, there is sufficient evidence that transforming growth factor–β (TGF-β) plays a key role. Overexpressed TGF-β induces epithelial-mesenchymal transition (EMT) and promotes abnormal deposition of extracellular matrix (ECM), leading to pulmonary fibrosis [ 19 ]. There have been some studies exploring how IL-7 affects TGF-β to reduce the risk of IPF. Huang et al. [ 20 ] demonstrated that IL-7 can not only down-regulate the synthesis of TGF-β in lung fibroblasts but also block TGF-β signaling through the intact JAK1/STAT1 pathway to reduce collagen synthesis. In addition, further studies found that IL-7 also inhibited PKC-δ activity to reduce TGF-β-induced expression of collagen genes COL1A1 and COL3A1 [ 21 ]. They also found that IL-7 was able to alleviate bleomycin-induced pulmonary fibrosis in vivo [ 20 ]. In an observational study using direct hemoperfusion with a polymyxin B-immobilized fiber column (PMX-DHP) for acute exacerbations of IPF, plasma IL-7 level was significantly higher in survivors compared with non-survivors on day 30 after treatment, which may indicate IL-7 has potential anti-fibrotic effects [ 22 ]. These previous studies suggest that IL-7 has therapeutic potential for IPF. Different from the perspective of the above studies, our study proved this genetically through MR analysis.

Some MR analyzes about IPF have been published, including lung cancer, gastroesophageal reflux disease, allergic rhinitis, but our study is the first to apply drug target MR analysis using eQTL to IPF. One of the strengths of our study lies in the size and diversity of the GWAS data used. To the best of our knowledge, these GWAS data are currently the largest available for IPF research. Furthermore, we ensured that there was no overlap between the population samples used in different GWAS, which adds to the reliability and validity of our findings. We implemented strict screening criteria during the IVs selection process. By following these stringent procedures and ensuring the fulfillment of key assumptions, we aimed to minimize the risk of bias and obtain reliable results in our MR analysis. These rigorous steps were essential in upholding the validity and integrity of our findings, thereby bolstering the overall robustness of our study. Colocalization analysis showed that IL-7 and IPF are likely to share the same causal variant, which strengthens the causal relationship. Of course, this result may be caused by pleiotropy [ 23 , 24 ]. But our study using cis-eQTL variants is supported by a clear and unidirectional biological principle (the central dogma) with less likelihood of other pathways, reducing potential horizontal pleiotropy [ 15 ]. In addition to IL-7, our study also identified other targets. Although they were not supported by colocalization analysis, their potential value cannot be completely denied, still providing broad possibilities for the development of IPF drugs.

There are several limitations in our study. Drug target MR only simulates the lifetime low-dose exposure of drugs under ideal conditions, and the actual situation will be more complicated due to the interference of other factors, so it cannot completely replace clinical trials and the actual efficacy of drugs is uncertain. Therefore, clinical trials remain necessary, and our study provides valuable insight and direction for the development of new drugs for IPF. Secondly, MR can only evaluate the impact of single druggable gene expression on outcome separately. However, many drugs exert their effects through the superposition of multiple targets. Thirdly, this study only included eQTL in blood, because we did not obtain appropriate eQTL data in the lung tissue. In case of unavailability of eQTL data in the lung tissue, biomarkers from the lungs will be released into the blood in the context of disease and blood serves as a valuable proxy tissue that offers a systemic perspective on disease processes. Blood carries molecular signals and cellular components from various organs and tissues, to a certain extent reflecting the dynamic interplay of systemic processes. The choice of blood has its limitations, including the dilution effect of systemic circulation and the potential masking of tissue-specific signals. Some molecular signals of the disease may not be fully revealed in blood eQTLs. Fortunately, some experiments [ 20 , 21 ] have demonstrated the anti-fibrotic effect of IL-7 in lung tissue, which makes up for this deficiency in our study. Furthermore, it is important to note that some studies have pointed out that the inhibition of TGF-β will show a variety of side effects, due to its wide range of effects [ 19 ]. And high levels of IL-7 are associated with autoimmune diseases such as rheumatoid arthritis [ 25 ], whether boosting IL-7 would have similar side effects as inhibiting TGF-β or more is not known, which may limit the application of IL-7 boosting strategy to IPF patients. Finally, the participants in the GWAS used were almost exclusively of European ancestry. This restriction may limit the generalizability of our results to other populations.

Conclusions

Drug target MR opens a new avenue for identifying potential drug targets utilizing druggable genetic data and disease GWAS data. In conclusion, through the drug target MR analysis based on the druggable genes, we have found that IL-7 holds promise as a potential target to reduce the risk of IPF in high-risk population. However, it is imperative to conduct further research to validate the effect of IL-7 in preventing IPF.

Data availability

All data used in this study are publicly available and listed in Table S1 . The cis-eQTL data were obtained from the eQTLGen Consortium ( https://www.eqtlgen.org/cis-eqtls.html ). The pQTL data was available from the INTERVAL study ( https://gwas.mrcieu.ac.uk/datasets/prot-a-1543/ ). The GWAS statistics for IPF susceptibility and progression were obtained from the International IPF Genetics Consortium ( https://github.com/genomicsITER/PFgenetics ).

Abbreviations

• Idiopathic pulmonary fibrosis

Expression quantitative trait locus

Protein quantitative trait locus

Genome wide association study

Forced vital capacity

Diffusing capacity of the lungs for carbon monoxide

Single nucleotide polymorphism

Instrumental variable

Inverse variance weighted

False discovery rate

Transforming growth factor–β

Raghu G, et al. Idiopathic pulmonary fibrosis (an update) and progressive pulmonary fibrosis in adults: an Official ATS/ERS/JRS/ALAT Clinical Practice Guideline. Am J Respir Crit Care Med. 2022;205(9):e18–47.

Article   PubMed   PubMed Central   Google Scholar  

Podolanczuk AJ et al. Idiopathic pulmonary fibrosis: state of the art for 2023. Eur Respir J, 2023. 61(4).

Cui F et al. Air pollutants, genetic susceptibility and risk of incident idiopathic pulmonary fibrosis. Eur Respir J, 2023. 61(2).

Greenland S. An introduction to instrumental variables for epidemiologists. Int J Epidemiol. 2000;29(4):722–9.

Article   CAS   PubMed   Google Scholar  

Emdin CA, Khera AV, Kathiresan S. Mendelian Randomization Jama. 2017;318(19):1925–6.

Article   PubMed   Google Scholar  

Storm CS, et al. Finding genetically-supported drug targets for Parkinson’s disease using mendelian randomization of the druggable genome. Nat Commun. 2021;12(1):7342.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Chen Y, et al. Genetic insights into therapeutic targets for aortic aneurysms: a mendelian randomization study. EBioMedicine. 2022;83:104199.

Gaziano L, et al. Actionable druggable genome-wide mendelian randomization identifies repurposing opportunities for COVID-19. Nat Med. 2021;27(4):668–76.

Finan C et al. The druggable genome and support for target identification and validation in drug development. Sci Transl Med, 2017. 9(383).

Võsa U, et al. Large-scale cis- and trans-eQTL analyses identify thousands of genetic loci and polygenic scores that regulate blood gene expression. Nat Genet. 2021;53(9):1300–10.

Sun BB, et al. Genomic atlas of the human plasma proteome. Nature. 2018;558(7708):73–9.

Gkatzionis A, Burgess S, Newcombe PJ. Statistical methods for cis-mendelian randomization with two-sample summary-level data. Genet Epidemiol. 2023;47(1):3–25.

Pierce BL, Ahsan H, Vanderweele TJ. Power and instrument strength requirements for mendelian randomization studies using multiple genetic variants. Int J Epidemiol. 2011;40(3):740–52.

Allen RJ, et al. Genome-wide association study across five cohorts identifies five novel loci associated with idiopathic pulmonary fibrosis. Thorax. 2022;77(8):829–33.

Allen RJ, et al. Longitudinal lung function and gas transfer in individuals with idiopathic pulmonary fibrosis: a genome-wide association study. Lancet Respir Med. 2023;11(1):65–73.

Burgess S, Dudbridge F, Thompson SG. Combining information on multiple instrumental variables in mendelian randomization: comparison of allele score and summarized data methods. Stat Med. 2016;35(11):1880–906.

Hemani G, Bowden J, Davey Smith G. Evaluating the potential role of pleiotropy in mendelian randomization studies. Hum Mol Genet. 2018;27(R2):R195–208.

Giambartolomei C, et al. Bayesian test for colocalisation between pairs of genetic association studies using summary statistics. PLoS Genet. 2014;10(5):e1004383.

Peng D, et al. Targeting TGF-β signal transduction for fibrosis and cancer therapy. Mol Cancer. 2022;21(1):104.

Huang M, et al. IL-7 inhibits fibroblast TGF-beta production and signaling in pulmonary fibrosis. J Clin Invest. 2002;109(7):931–7.

Zhang L, et al. Interleukin-7 and transforming growth factor-beta play counter-regulatory roles in protein kinase C-delta-dependent control of fibroblast collagen synthesis in pulmonary fibrosis. J Biol Chem. 2004;279(27):28315–9.

Tachibana K, et al. Polymyxin-B hemoperfusion for acute exacerbation of idiopathic pulmonary fibrosis: serum IL-7 as a prognostic marker. Sarcoidosis Vasc Diffuse Lung Dis. 2011;28(2):113–22.

CAS   PubMed   Google Scholar  

Cano-Gamez E, Trynka G. From GWAS to function: using Functional Genomics to identify the mechanisms underlying Complex diseases. Front Genet. 2020;11:424.

Zuber V, et al. Combining evidence from mendelian randomization and colocalization: review and comparison of approaches. Am J Hum Genet. 2022;109(5):767–82.

Meyer A, Parmar PJ, Shahrara S. Significance of IL-7 and IL-7R in RA and autoimmunity. Autoimmun Rev. 2022;21(7):103120.

Download references

Acknowledgements

We would like to thank all members of the eQTLGen Consortium, the International IPF Genetics Consortium and the author of INTERVAL study for making the data publicly available.

No funding.

Author information

Authors and affiliations.

Department of Thoracic Surgery, West China Hospital, Sichuan University, Chengdu, 610041, China

Zetao Liu, Zhiyu Peng, Huahang Lin, Ke Zhou, Linchuan Liang, Jie Cao, Zhaokang Huang & Jiandong Mei

Western China Collaborative Innovation Center for Early Diagnosis and Multidisciplinary Therapy of Lung Cancer, Sichuan University, Chengdu, China

You can also search for this author in PubMed   Google Scholar

Contributions

Z.L. and J.M. contributed to the design of this study. Z.P., H.L. and K.Z. contributed to data acquisition. L.L., J.C. and Z.H. contributed to analyze of data and draft the manuscript. All authors participated in revisions and reviewed the manuscript before submission.

Corresponding author

Correspondence to Jiandong Mei .

Ethics declarations

Ethics statements.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Liu, Z., Peng, Z., Lin, H. et al. Identifying potential drug targets for idiopathic pulmonary fibrosis: a mendelian randomization study based on the druggable genes. Respir Res 25 , 217 (2024). https://doi.org/10.1186/s12931-024-02848-5

Download citation

Received : 26 September 2023

Accepted : 13 May 2024

Published : 23 May 2024

DOI : https://doi.org/10.1186/s12931-024-02848-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Druggable genes

Respiratory Research

ISSN: 1465-993X

• General enquiries: [email protected]

• Systematic Review
• Open access
• Published: 24 May 2024

Turnover intention and its associated factors among nurses in Ethiopia: a systematic review and meta-analysis

• Eshetu Elfios 1 ,
• Israel Asale 1 ,
• Merid Merkine 1 ,
• Temesgen Geta 1 ,
• Kidist Ashager 1 ,
• Getachew Nigussie 1 ,
• Ayele Agena 1 ,
• Bizuayehu Atinafu 1 ,
• Eskindir Israel 2 &
• Teketel Tesfaye 3  

BMC Health Services Research volume  24 , Article number:  662 ( 2024 ) Cite this article

303 Accesses

Metrics details

Nurses turnover intention, representing the extent to which nurses express a desire to leave their current positions, is a critical global public health challenge. This issue significantly affects the healthcare workforce, contributing to disruptions in healthcare delivery and organizational stability. In Ethiopia, a country facing its own unique set of healthcare challenges, understanding and mitigating nursing turnover are of paramount importance. Hence, the objectives of this systematic review and meta-analysis were to determine the pooled proportion ofturnover intention among nurses and to identify factors associated to it in Ethiopia.

A comprehensive search carried out for studies with full document and written in English language through an electronic web-based search strategy from databases including PubMed, CINAHL, Cochrane Library, Embase, Google Scholar and Ethiopian University Repository online. Checklist from the Joanna Briggs Institute (JBI) was used to assess the studies’ quality. STATA version 17 software was used for statistical analyses. Meta-analysis was done using a random-effects method. Heterogeneity between the primary studies was assessed by Cochran Q and I-square tests. Subgroup and sensitivity analyses were carried out to clarify the source of heterogeneity.

This systematic review and meta-analysis incorporated 8 articles, involving 3033 nurses in the analysis. The pooled proportion of turnover intention among nurses in Ethiopia was 53.35% (95% CI (41.64, 65.05%)), with significant heterogeneity between studies (I 2  = 97.9, P  = 0.001). Significant association of turnover intention among nurses was found with autonomous decision-making (OR: 0.28, CI: 0.14, 0.70) and promotion/development (OR: 0.67, C.I: 0.46, 0.89).

Conclusion and recommendation

Our meta-analysis on turnover intention among Ethiopian nurses highlights a significant challenge, with a pooled proportion of 53.35%. Regional variations, such as the highest turnover in Addis Ababa and the lowest in Sidama, underscore the need for tailored interventions. The findings reveal a strong link between turnover intention and factors like autonomous decision-making and promotion/development. Recommendations for stakeholders and concerned bodies involve formulating targeted retention strategies, addressing regional variations, collaborating for nurse welfare advocacy, prioritizing career advancement, reviewing policies for nurse retention improvement.

Peer Review reports

Turnover intention pertaining to employment, often referred to as the intention to leave, is characterized by an employee’s contemplation of voluntarily transitioning to a different job or company [ 1 ]. Nurse turnover intention, representing the extent to which nurses express a desire to leave their current positions, is a critical global public health challenge. This issue significantly affects the healthcare workforce, contributing to disruptions in healthcare delivery and organizational stability [ 2 ].

The global shortage of healthcare professionals, including nurses, is an ongoing challenge that significantly impacts the capacity of healthcare systems to provide quality services [ 3 ]. Nurses, as frontline healthcare providers, play a central role in patient care, making their retention crucial for maintaining the functionality and effectiveness of healthcare delivery. However, the phenomenon of turnover intention, reflecting a nurse’s contemplation of leaving their profession, poses a serious threat to workforce stability [ 4 ].

Studies conducted globally shows that high turnover rates among nurses in several regions, with notable figures reported in Alexandria (68%), China (63.88%), and Jordan (60.9%) [ 5 , 6 , 7 ]. In contrast, Israel has a remarkably low turnover rate of9% [ 8 ], while Brazil reports 21.1% [ 9 ], and Saudi hospitals26% [ 10 ]. These diverse turnover rates highlight the global nature of the nurse turnover phenomenon, indicating varying degrees of workforce mobility in different regions.

The magnitude and severity of turnover intention among nurses worldwide underscore the urgency of addressing this issue. High turnover rates not only disrupt healthcare services but also result in a loss of valuable skills and expertise within the nursing workforce. This, in turn, compromises the continuity and quality of patient care, with potential implications for patient outcomes and overall health service delivery [ 11 ]. Extensive research conducted worldwide has identified a range of factors contributing to turnover intention among nurses [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ]. These factors encompass both individual and organizational aspects, such as high workload, inadequate support, limited career advancement opportunities, job satisfaction, conflict, payment or reward, burnout sense of belongingness to their work environment. The complex interplay of these factors makes addressing turnover intention a multifaceted challenge that requires targeted interventions.

In Ethiopia, a country facing its own unique set of healthcare challenges, understanding and mitigating nursing turnover are of paramount importance. The healthcare system in Ethiopia grapples with issues like resource constraints, infrastructural limitations, and disparities in healthcare access [ 18 ]. Consequently, the factors influencing nursing turnover in Ethiopia may differ from those in other regions. Previous studies conducted in the Ethiopian context have started to unravel some of these factors, emphasizing the need for a more comprehensive examination [ 18 , 19 ].

Although many cross-sectional studies have been conducted on turnover intention among nurses in Ethiopia, the results exhibit variations. The reported turnover intention rates range from a minimum of 30.6% to a maximum of 80.6%. In light of these disparities, this systematic review and meta-analysis was undertaken to ascertain the aggregated prevalence of turnover intention among nurses in Ethiopia. By systematically analyzing findings from various studies, we aimed to provide a nuanced understanding of the factors influencing turnover intention specific to the Ethiopian healthcare context. Therefore, this systematic review and meta-analysis aimed to answer the following research questions.

What is the pooled prevalence of turnover intention among nurses in Ethiopia?

What are the factors associated with turnover intention among nurses in Ethiopia?

The primary objective of this review was to assess the pooled proportion of turnover intention among nurses in Ethiopia. The secondary objective was identifying the factors associated to turnover intention among nurses in Ethiopia.

Study design and search strategy

A comprehensive systematic review and meta-analysis was conducted, examining observational studies on turnover intention among nurses in Ethiopia. The procedure for this systematic review and meta-analysis was developed in accordance with the Preferred Reporting Items for Systematic review and Meta-analysis Protocols (PRISMA-P) statement [ 20 ]. PRISMA-2015 statement was used to report the findings [ 21 , 22 ]. This systematic review and meta-analysis were registered on PROSPERO with the registration number of CRD42024499119.

We conducted systematic and an extensive search across multiple databases, including PubMed, CINAHL, Cochrane Library, Embase, Google Scholar and Ethiopian University Repository online to identify studies reporting turnover intention among nurses in Ethiopia. We reviewed the database available at http://www.library.ucsf.edu and the Cochrane Library to ensure that the intended task had not been previously undertaken, preventing any duplication. Furthermore, we screened the reference lists to retrieve relevant articles. The process involved utilizing EndNote (version X8) software for downloading, organizing, reviewing, and citing articles. Additionally, a manual search for cross-references was performed to discover any relevant studies not captured through the initial database search. The search employed a comprehensive set of the following search terms:“prevalence”, “turnover intention”, “intention to leave”, “attrition”, “employee attrition”, “nursing staff turnover”, “Ethiopian nurses”, “nurses”, and “Ethiopia”. These terms were combined using Boolean operators (AND, OR) to conduct a thorough and systematic search across the specified databases.

Eligibility criteria

Inclusion criteria.

The established inclusion criteria for this meta-analysis and systematic review are as follows to guide the selection of articles for inclusion in this review.

Population: Nurses working in Ethiopia.

Study period: studies conducted or published until 23November 2023.

Study design: All observational study designs, such as cross-sectional, longitudinal, and cohort studies, were considered.

Setting: Only studies conducted in Ethiopia were included.

Outcome; turnover intention.

Study: All studies, whether published or unpublished, in the form of journal articles, master’s theses, and dissertations, were included up to the final date of data analysis.

Language: This study exclusively considered studies in the English language.

Exclusion criteria

Excluded were studies lacking full text or Studies with a Newcastle–Ottawa Quality Assessment Scale (NOS) score of 6 or less. Studies failing to provide information on turnover intention among nurses or studies for which necessary details could not be obtained were excluded. Three authors (E.E., T.G., K.A) independently assessed the eligibility of retrieved studies, other two authors (E.I & M.M) input sought for consensus on potential in- or exclusion.

Quality assessment and data extraction

Two authors (E.E, A.A, G.N) independently conducted a critical appraisal of the included studies. Joanna Briggs Institute (JBI) checklists of prevalence study was used to assess the quality of the studies. Studies with a Newcastle–Ottawa Quality Assessment Scale (NOS) score of seven or more were considered acceptable [ 23 ]. The tool has nine parameters, which have yes, no, unclear, and not applicable options [ 24 ]. Two reviewers (I.A, B.A) were involved when necessary, during the critical appraisal process. Accordingly, all studies were included in our review. ( Table  1 ) Questions to evaluate the methodological quality of studies on turnover intention among nurses and its associated factors in Ethiopia are the followings:

Q1 = was the sample frame appropriate to address the target population?

Q2. Were study participants sampled appropriately.

Q3. Was the sample size adequate?

Q4. Were the study subjects and the setting described in detail?

Q5. Was the data analysis conducted with sufficient coverage of the identified sample?

Q6. Were the valid methods used for the identification of the condition?

Q7. Was the condition measured in a standard, reliable way for all participants?

Q8. Was there appropriate statistical analysis?

Q9. Was the response rate adequate, and if not, was the low response rate.

managed appropriately?

Data was extracted and recorded in a Microsoft Excel as guided by the Joanna Briggs Institute (JBI) data extraction form for observational studies. Three authors (E.E, M.G, T.T) independently conducted data extraction. Recorded data included the first author’s last name, publication year, study setting or country, region, study design, study period, sample size, response rate, population, type of management, proportion of turnover intention, and associated factors. Discrepancies in data extraction were resolved through discussion between extractors.

Data processing and analysis

Data analysis procedures involved importing the extracted data into STATA 14 statistical software for conducting a pooled proportion of turnover intention among nurses. To evaluate potential publication bias and small study effects, both funnel plots and Egger’s test were employed [ 25 , 26 ]. We used statistical tests such as the I statistic to quantify heterogeneity and explore potential sources of variability. Additionally, subgroup analyses were conducted to investigate the impact of specific study characteristics on the overall results. I 2 values of 0%, 25%, 50%, and 75% were interpreted as indicating no, low, medium, and high heterogeneity, respectively [ 27 ].

To assess publication bias, we employed several methods, including funnel plots and Egger’s test. These techniques allowed us to visually inspect asymmetry in the distribution of study results and statistically evaluate the presence of publication bias. Furthermore, we conducted sensitivity analyses to assess the robustness of our findings to potential publication bias and other sources of bias.

Utilizing a random-effects method, a meta-analysis was performed to assess turnover intention among nurses, employing this method to account for observed variability [ 28 ]. Subgroup analyses were conducted to compare the pooled magnitude of turnover intention among nurses and associated factors across different regions. The results of the pooled prevalence were visually presented in a forest plot format with a 95% confidence interval.

Study selection

After conducting the initial comprehensive search concerning turnover intention among nurses through Medline, Cochran Library, Web of Science, Embase, Ajol, Google Scholar, and other sources, a total of 1343 articles were retrieved. Of which 575 were removed due to duplication. Five hundred ninety-three articles were removed from the remaining 768 articles by title and abstract. Following theses, 44 articles which cannot be retrieved were removed. Finally, from the remaining 131 articles, 8 articles with a total 3033 nurses were included in the systematic review and meta-analysis (Fig.  1 ).

PRISMA flow diagram of the selection process of studies on turnover intention among nurses in Ethiopia, 2024

Study characteristics

All included 8 studies had a cross-sectional design and of which, 2 were from Tigray region, 2 were from Addis Ababa(Capital), 1 from south region, 1 from Amhara region, 1 from Sidama region, and 1 was multiregional and Nationwide. The prevalence of turnover intention among nurses ‘ranges from 30.6 to 80.6%. Table  2 .

Pooled prevalence of turnover intention among nurses in Ethiopia

Our comprehensive meta-analysis revealed a notable turnover intention rate of 53.35% (95% CI: 41.64, 65.05%) among Ethiopian nurses, accompanied by substantial heterogeneity between studies (I 2  = 97.9, P  = 0.000) as depicted in Fig.  2 . Given the observed variability, we employed a random-effects model to analyze the data, ensuring a robust adjustment for the significant heterogeneity across the included studies.

Forest plot showing the pooled proportion of turnover intention among nurses in Ethiopia, 2024

Subgroup analysis of turnover intention among nurses in Ethiopia

To address the observed heterogeneity, we conducted a subgroup analysis based on regions. The results of the subgroup analysis highlighted considerable variations, with the highest level of turnover intention identified in Addis Ababa at 69.10% (95% CI: 46.47, 91.74%) and substantial heterogeneity (I 2  = 98.1%). Conversely, the Sidama region exhibited the lowest level of turnover intention among nurses at 30.6% (95% CI: 25.18, 36.02%), accompanied by considerable heterogeneity (I 2  = 100.0%) ( Fig.  3 ).

Subgroup analysis of systematic review and meta-analysis by region of turnover intention among nurses in Ethiopia, 2024

Publication bias of turnover intention among nurses in Ethiopia

The Egger’s test result ( p  = 0.64) is not statistically significant, indicating no evidence of publication bias in the meta-analysis (Table  3 ). Additionally, the symmetrical distribution of included studies in the funnel plot (Fig.  4 ) confirms the absence of publication bias across studies.

Funnel plot of systematic review and meta-analysis on turnover intention among nurses in Ethiopia, 2024

Sensitivity analysis

The leave-out-one sensitivity analysis served as a meticulous evaluation of the influence of individual studies on the comprehensive pooled prevalence of turnover intention within the context of Ethiopian nurses. In this systematic process, each study was methodically excluded from the analysis one at a time. The outcomes of this meticulous examination indicated that the exclusion of any particular study did not lead to a noteworthy or statistically significant alteration in the overall pooled estimate of turnover intention among nurses in Ethiopia. The findings are visually represented in Fig.  5 , illustrating the stability and robustness of the overall pooled estimate even with the removal of specific studies from the analysis.

Sensitivity analysis of pooled prevalence for each study being removed at a time for systematic review and meta-analysis of turnover intention among nurses in Ethiopia

Factors associated with turnover intention among nurses in Ethiopia

In our meta-analysis, we comprehensively reviewed and conducted a meta-analysis on the determinants of turnover intention among nurses in Ethiopia by examining eight relevant studies [ 6 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ]. We identified a significant association between turnover intention with autonomous decision-making (OR: 0.28, CI: 0.14, 0.70) (Fig.  6 ) and promotion/development (OR: 0.67, CI: 0.46, 0.89) (Fig.  7 ). In both instances, the odds ratios suggest a negative association, signifying that increased levels of autonomous decision-making and promotion/development were linked to reduced odds of turnover intention.

Forest plot of the association between autonomous decision making with turnover intention among nurses in Ethiopia2024

Forest plot of the association between promotion/developpment with turnover intention among nurses in Ethiopia, 2024

In our comprehensive meta-analysis exploring turnover intention among nurses in Ethiopia, our findings revealed a pooled proportion of turnover intention at 53.35%. This significant proportion warrants a comparative analysis with turnover rates reported in other global regions. Distinct variations emerge when compared with turnover rates in Alexandria (68%), China (63.88%), and Jordan (60.9%) [ 5 , 6 , 7 ]. This comparison highlights that the multifaceted nature of turnover intention, influenced by diverse contextual, cultural, and organizational factors. Conversely, Ethiopia’s turnover rate among nurses contrasts with substantially lower figures reported in Israel (9%) [ 8 ], Brazil (21.1%) [ 9 ], and Saudi hospitals (26%) [ 10 ]. Challenges such as work overload, economic constraints, limited promotional opportunities, lack of recognition, and low job rewards are more prevalent among nurses in Ethiopia, contributing to higher turnover intention compared to their counterparts [ 7 , 29 , 36 ].

The highest turnover intention was observed in Addis Ababa, while Sidama region displayed the lowest turnover intention among nurses, These differences highlight the complexity of turnover intention among Ethiopian nurses, showing the importance of specific interventions in each region to address unique factors and improve nurses’ retention.

Our systematic review and meta-analysis in the Ethiopian nursing context revealed a significant inverse association between turnover intention and autonomous decision-making. The odd of turnover intention is approximately reduced by 72% in employees with autonomous decision-making compared to those without autonomous decision-making. This finding was supported by other similar studies conducted in South Africa, Tanzania, Kenya, and Turkey [ 37 , 38 , 39 , 40 ].

The significant association of turnover intention with promotion/development in our study underscores the crucial role of career advancement opportunities in alleviating turnover intention among nurses. Specifically, our analysis revealed that individuals with promotion/development had approximately 33% lower odds of turnover intention compared to those without such opportunities. These results emphasize the pivotal influence of organizational support in shaping the professional environment for nurses, providing substantive insights for the formulation of evidence-based strategies targeted at enhancing workforce retention. This finding is in line with former researches conducted in Taiwan, Philippines and Italy [ 41 , 42 , 43 ].

Our meta-analysis on turnover intention among Ethiopian nurses reveals a considerable challenge, with a pooled proportion of 53.35%. Regional variations highlight the necessity for region-specific strategies, with Addis Ababa displaying the highest turnover intention and Sidama region the lowest. A significant inverse association was found between turnover intention with autonomous decision-making and promotion/development. These insights support the formulation of evidence-based strategies and policies to enhance nurse retention, contributing to the overall stability of the Ethiopian healthcare system.

Recommendations

Federal ministry of health (fmoh).

The FMoH should consider the regional variations in turnover intention and formulate targeted retention strategies. Investment in professional development opportunities and initiatives to enhance autonomy can be integral components of these strategies.

Ethiopian nurses association (ENA)

ENA plays a pivotal role in advocating for the welfare of nurses. The association is encouraged to collaborate with healthcare institutions to promote autonomy, create mentorship programs, and advocate for improved working conditions to mitigate turnover intention.

Healthcare institutions

Hospitals and healthcare facilities should prioritize the provision of career advancement opportunities and recognize the value of professional autonomy in retaining nursing staff. Tailored interventions based on regional variations should be considered.

Policy makers

Policymakers should review existing healthcare policies to identify areas for improvement in nurse retention. Policy changes that address challenges such as work overload, limited promotional opportunities, and economic constraints can positively impact turnover rates.

Future research initiatives

Further research exploring the specific factors contributing to turnover intention in different regions of Ethiopia is recommended. Understanding the nuanced challenges faced by nurses in various settings will inform the development of more targeted interventions.

Strength and limitations

Our systematic review and meta-analysis on nurse turnover intention in Ethiopia present several strengths. The comprehensive inclusion of diverse studies provides a holistic view of the issue, enhancing the generalizability of our findings. The use of a random-effects model accounts for potential heterogeneity, ensuring a more robust and reliable synthesis of data.

However, limitations should be acknowledged. The heterogeneity observed across studies, despite the use of a random-effects model, may impact the precision of the pooled estimate. These considerations should be taken into account when interpreting and applying the results of our analysis.

Data availability

Data set used on this analysis will available from corresponding author upon reasonable request.

Abbreviations

Ethiopian Nurses Association

Federal Ministry of Health

Joanna Briggs Institute

Preferred Reporting Items for Systematic review and Meta-analysis Protocols

Kanchana L, Jayathilaka R. Factors impacting employee turnover intentions among professionals in Sri Lankan startups. PLoS ONE. 2023;18(2):e0281729.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Boateng AB, et al. Factors influencing turnover intention among nurses and midwives in Ghana. Nurs Res Pract. 2022;2022:4299702.

PubMed   PubMed Central   Google Scholar  

Organization WH. WHO Guideline on Health Workforce Development Attraction, Recruitment and Retention in Rural and Remote Areas, 2021, pp. 1-104.

Hayes LJ, et al. Nurse turnover: a literature review. Int J Nurs Stud. 2006;43(2):237–63.

Article   PubMed   Google Scholar  

Yang H, et al. Validation of work pressure and associated factors influencing hospital nurse turnover: a cross-sectional investigation in Shaanxi Province, China. BMC Health Serv Res. 2017;17:1–11.

Article   Google Scholar  

Ayalew E et al. Nurses’ intention to leave their job in sub-Saharan Africa: A systematic review and meta-analysis. Heliyon, 2021. 7(6).

Al Momani M. Factors influencing public hospital nurses’ intentions to leave their current employment in Jordan. Int J Community Med Public Health. 2017;4(6):1847–53.

DeKeyser Ganz F, Toren O. Israeli nurse practice environment characteristics, retention, and job satisfaction. Isr J Health Policy Res. 2014;3(1):1–8.

de Oliveira DR, et al. Intention to leave profession, psychosocial environment and self-rated health among registered nurses from large hospitals in Brazil: a cross-sectional study. BMC Health Serv Res. 2017;17(1):21.

Article   PubMed   PubMed Central   Google Scholar  

Dall’Ora C, et al. Association of 12 h shifts and nurses’ job satisfaction, burnout and intention to leave: findings from a cross-sectional study of 12 European countries. BMJ Open. 2015;5(9):e008331.

Lu H, Zhao Y, While A. Job satisfaction among hospital nurses: a literature review. Int J Nurs Stud. 2019;94:21–31.

Ramoo V, Abdullah KL, Piaw CY. The relationship between job satisfaction and intention to leave current employment among registered nurses in a teaching hospital. J Clin Nurs. 2013;22(21–22):3141–52.

Al Sabei SD, et al. Nursing work environment, turnover intention, Job Burnout, and Quality of Care: the moderating role of job satisfaction. J Nurs Scholarsh. 2020;52(1):95–104.

Wang H, Chen H, Chen J. Correlation study on payment satisfaction, psychological reward satisfaction and turnover intention of nurses. Chin Hosp Manag. 2018;38(03):64–6.

Google Scholar  

Loes CN, Tobin MB. Interpersonal conflict and organizational commitment among licensed practical nurses. Health Care Manag (Frederick). 2018;37(2):175–82.

Wei H, et al. The state of the science of nurse work environments in the United States: a systematic review. Int J Nurs Sci. 2018;5(3):287–300.

Nantsupawat A, et al. Effects of nurse work environment on job dissatisfaction, burnout, intention to leave. Int Nurs Rev. 2017;64(1):91–8.

Article   CAS   PubMed   Google Scholar  

Ayalew F, et al. Factors affecting turnover intention among nurses in Ethiopia. World Health Popul. 2015;16(2):62–74.

Debie A, Khatri RB, Assefa Y. Contributions and challenges of healthcare financing towards universal health coverage in Ethiopia: a narrative evidence synthesis. BMC Health Serv Res. 2022;22(1):866.

Moher D, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Reviews. 2015;4(1):1–9.

Moher D, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9.

Moher D et al. Group, P.-P.(2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.

Institute JB. Checklist for Prevalence Studies. Checkl prevalance Stud [Internet]. 2016;7.

Sakonidou S, et al. Interventions to improve quantitative measures of parent satisfaction in neonatal care: a systematic review. BMJ Paediatr Open. 2020;4(1):e000613.

Egger M, Smith GD. Meta-analysis: potentials and promise. BMJ. 1997;315(7119):1371.

Tura G, Fantahun M, Worku A. The effect of health facility delivery on neonatal mortality: systematic review and meta-analysis. BMC Pregnancy Childbirth. 2013;13:18.

Lin L. Comparison of four heterogeneity measures for meta-analysis. J Eval Clin Pract. 2020;26(1):376–84.

McFarland LV. Meta-analysis of probiotics for the prevention of antibiotic associated diarrhea and the treatment of Clostridium difficile disease. Am J Gastroenterol. 2006;101(4):812–22.

Asegid A, Belachew T, Yimam E. Factors influencing job satisfaction and anticipated turnover among nurses in Sidama zone public health facilities, South Ethiopia Nursing research and practice, 2014. 2014.

Wubetie A, Taye B, Girma B. Magnitude of turnover intention and associated factors among nurses working in emergency departments of governmental hospitals in Addis Ababa, Ethiopia: a cross-sectional institutional based study. BMC Nurs. 2020;19:97.

Getie GA, Betre ET, Hareri HA. Assessment of factors affecting turnover intention among nurses working at governmental health care institutions in east Gojjam, Amhara region, Ethiopia, 2013. Am J Nurs Sci. 2015;4(3):107–12.

Gebregziabher D, et al. The relationship between job satisfaction and turnover intention among nurses in Axum comprehensive and specialized hospital Tigray, Ethiopia. BMC Nurs. 2020;19(1):79.

Negarandeh R et al. Magnitude of nurses’ intention to leave their jobs and its associated factors of nurses working in tigray regional state, north ethiopia: cross sectional study 2020.

Nigussie Bolado G, et al. The magnitude of turnover intention and Associated factors among nurses working at Governmental Hospitals in Southern Ethiopia: a mixed-method study. Nursing: Research and Reviews; 2023. pp. 13–29.

Woldekiros AN, Getye E, Abdo ZA. Magnitude of job satisfaction and intention to leave their present job among nurses in selected federal hospitals in Addis Ababa, Ethiopia. PLoS ONE. 2022;17(6):e0269540.

Rhoades L, Eisenberger R. Perceived organizational support: a review of the literature. J Appl Psychol. 2002;87(4):698.

Lewis M. Causal factors that influence turnover intent in a manufacturing organisation. University of Pretoria (South Africa); 2008.

Kuria S, Alice O, Wanderi PM. Assessment of causes of labour turnover in three and five star-rated hotels in Kenya International journal of business and social science, 2012. 3(15).

Blaauw D, et al. Comparing the job satisfaction and intention to leave of different categories of health workers in Tanzania, Malawi, and South Africa. Global Health Action. 2013;6(1):19287.

Masum AKM, et al. Job satisfaction and intention to quit: an empirical analysis of nurses in Turkey. PeerJ. 2016;4:e1896.

Song L. A study of factors influencing turnover intention of King Power Group at Downtown Area in Bangkok, Thailand. Volume 2. International Review of Research in Emerging Markets & the Global Economy; 2016. 3.

Karanikola MN, et al. Moral distress, autonomy and nurse-physician collaboration among intensive care unit nurses in Italy. J Nurs Manag. 2014;22(4):472–84.

Labrague LJ, McEnroe-Petitte DM, Tsaras K. Predictors and outcomes of nurse professional autonomy: a cross-sectional study. Int J Nurs Pract. 2019;25(1):e12711.

Download references

No funding was received.

Author information

Authors and affiliations.

School of Nursing, College of Health Science and Medicine, Wolaita Sodo University, Wolaita Sodo, Ethiopia

Eshetu Elfios, Israel Asale, Merid Merkine, Temesgen Geta, Kidist Ashager, Getachew Nigussie, Ayele Agena & Bizuayehu Atinafu

Department of Midwifery, College of Health Science and Medicine, Wolaita Sodo University, Wolaita Sodo, Ethiopia

Eskindir Israel

Department of Midwifery, College of Health Science and Medicine, Wachamo University, Hossana, Ethiopia

Teketel Tesfaye

You can also search for this author in PubMed   Google Scholar

Contributions

E.E. conceptualized the study, designed the research, performed statistical analysis, and led the manuscript writing. I.A, T.G, M.M contributed to the study design and provided critical revisions. K.A., G.N, B.A., E.I., and T.T. participated in data extraction and quality assessment. M.M. and T.G. K.A. and G.N. contributed to the literature review. I.A, A.A. and B.A. assisted in data interpretation. E.I. and T.T. provided critical revisions to the manuscript. All authors read and approved the final version.

Corresponding author

Correspondence to Eshetu Elfios .

Ethics declarations

Ethical approval.

Ethical approval and informed consent are not required, as this study is a systematic review and meta-analysis that only involved the use of previously published data.

Ethical guidelines

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Elfios, E., Asale, I., Merkine, M. et al. Turnover intention and its associated factors among nurses in Ethiopia: a systematic review and meta-analysis. BMC Health Serv Res 24 , 662 (2024). https://doi.org/10.1186/s12913-024-11122-9

Download citation

Received : 20 January 2024

Accepted : 20 May 2024

Published : 24 May 2024

DOI : https://doi.org/10.1186/s12913-024-11122-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Turnover intention
• Systematic review
• Meta-analysis

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]