Reliability and Validity
Reliability means that individual scores from an instrument should be the same or nearly the same from one administration of the instrument to another. The instrument can be assumed to be free of bias and measurement error (68). Alpha coefficients are often used to report an estimate of internal consistency. Scores of .70 or higher indicate that the instrument has high reliability when the stakes are moderate. Scores of .80 and higher are appropriate when the stakes are high.
Validity means that individual scores from a particular instrument are meaningful, make sense, and allow researchers to draw conclusions from the sample to the population that is being studied (69) Researchers often refer to "content" or "face" validity. Content validity or face validity is the extent to which questions on an instrument are representative of the possible questions that a researcher could ask about that particular content or skills.
Watson-Glaser Critical Thinking Appraisal-FS (WGCTA-FS)
The WGCTA-FS is a 40-item inventory created to replace Forms A and B of the original test, which participants reported was too long.70 This inventory assesses test takers' skills in:
(a) Inference: the extent to which the individual recognizes whether assumptions are clearly stated (b) Recognition of assumptions: whether an individual recognizes whether assumptions are clearly stated (c) Deduction: whether an individual decides if certain conclusions follow the information provided (d) Interpretation: whether an individual considers evidence provided and determines whether generalizations from data are warranted (e) Evaluation of arguments: whether an individual distinguishes strong and relevant arguments from weak and irrelevant arguments
Researchers investigated the reliability and validity of the WGCTA-FS for subjects in academic fields. Participants included 586 university students. Internal consistencies for the total WGCTA-FS among students majoring in psychology, educational psychology, and special education, including undergraduates and graduates, ranged from .74 to .92. The correlations between course grades and total WGCTA-FS scores for all groups ranged from .24 to .62 and were significant at the p < .05 of p < .01. In addition, internal consistency and test-retest reliability for the WGCTA-FS have been measured as .81. The WGCTA-FS was found to be a reliable and valid instrument for measuring critical thinking (71).
Cornell Critical Thinking Test (CCTT)
There are two forms of the CCTT, X and Z. Form X is for students in grades 4-14. Form Z is for advanced and gifted high school students, undergraduate and graduate students, and adults. Reliability estimates for Form Z range from .49 to .87 across the 42 groups who have been tested. Measures of validity were computed in standard conditions, roughly defined as conditions that do not adversely affect test performance. Correlations between Level Z and other measures of critical thinking are about .50.72 The CCTT is reportedly as predictive of graduate school grades as the Graduate Record Exam (GRE), a measure of aptitude, and the Miller Analogies Test, and tends to correlate between .2 and .4.73
California Critical Thinking Disposition Inventory (CCTDI)
Facione and Facione have reported significant relationships between the CCTDI and the CCTST. When faculty focus on critical thinking in planning curriculum development, modest cross-sectional and longitudinal gains have been demonstrated in students' CTS.74 The CCTDI consists of seven subscales and an overall score. The recommended cut-off score for each scale is 40, the suggested target score is 50, and the maximum score is 60. Scores below 40 on a specific scale are weak in that CT disposition, and scores above 50 on a scale are strong in that dispositional aspect. An overall score of 280 shows serious deficiency in disposition toward CT, while an overall score of 350 (while rare) shows across the board strength. The seven subscales are analyticity, self-confidence, inquisitiveness, maturity, open-mindedness, systematicity, and truth seeking (75).
In a study of instructional strategies and their influence on the development of critical thinking among undergraduate nursing students, Tiwari, Lai, and Yuen found that, compared with lecture students, PBL students showed significantly greater improvement in overall CCTDI (p = .0048), Truth seeking (p = .0008), Analyticity (p =.0368) and Critical Thinking Self-confidence (p =.0342) subscales from the first to the second time points; in overall CCTDI (p = .0083), Truth seeking (p= .0090), and Analyticity (p =.0354) subscales from the second to the third time points; and in Truth seeking (p = .0173) and Systematicity (p = .0440) subscales scores from the first to the fourth time points (76). California Critical Thinking Skills Test (CCTST)
Studies have shown the California Critical Thinking Skills Test captured gain scores in students' critical thinking over one quarter or one semester. Multiple health science programs have demonstrated significant gains in students' critical thinking using site-specific curriculum. Studies conducted to control for re-test bias showed no testing effect from pre- to post-test means using two independent groups of CT students. Since behavioral science measures can be impacted by social-desirability bias-the participant's desire to answer in ways that would please the researcher-researchers are urged to have participants take the Marlowe Crowne Social Desirability Scale simultaneously when measuring pre- and post-test changes in critical thinking skills. The CCTST is a 34-item instrument. This test has been correlated with the CCTDI with a sample of 1,557 nursing education students. Results show that, r = .201, and the relationship between the CCTST and the CCTDI is significant at p< .001. Significant relationships between CCTST and other measures including the GRE total, GRE-analytic, GRE-Verbal, GRE-Quantitative, the WGCTA, and the SAT Math and Verbal have also been reported. The two forms of the CCTST, A and B, are considered statistically significant. Depending on the testing, context KR-20 alphas range from .70 to .75. The newest version is CCTST Form 2000, and depending on the testing context, KR-20 alphas range from .78-.84.77
The Health Science Reasoning Test (HSRT)
Items within this inventory cover the domain of CT cognitive skills identified by a Delphi group of experts whose work resulted in the development of the CCTDI and CCTST. This test measures health science undergraduate and graduate students' CTS. Although test items are set in health sciences and clinical practice contexts, test takers are not required to have discipline-specific health sciences knowledge. For this reason, the test may have limited utility in dental education (78).
Preliminary estimates of internal consistency show that overall KR-20 coefficients range from .77 to .83.79 The instrument has moderate reliability on analysis and inference subscales, although the factor loadings appear adequate. The low K-20 coefficients may be result of small sample size, variance in item response, or both (see following table).
Table 8. Estimates of Internal Consistency and Factor Loading by Subscale for HSRT
Inductive | .76 | .332-.769 |
Deductive | .71 | .366-.579 |
Analysis | .54 | .369-.599 |
Inference | .52 | .300-.664 |
Evaluation | .77 | .359-.758 |
Professional Judgment Rating Form (PJRF)
The scale consists of two sets of descriptors. The first set relates primarily to the attitudinal (habits of mind) dimension of CT. The second set relates primarily to CTS.
A single rater should know the student well enough to respond to at least 17 or the 20 descriptors with confidence. If not, the validity of the ratings may be questionable. If a single rater is used and ratings over time show some consistency, comparisons between ratings may be used to assess changes. If more than one rater is used, then inter-rater reliability must be established among the raters to yield meaningful results. While the PJRF can be used to assess the effectiveness of training programs for individuals or groups, the evaluation of participants' actual skills are best measured by an objective tool such as the California Critical Thinking Skills Test.
Teaching for Thinking Student Course Evaluation Form
Course evaluations typically ask for responses of "agree" or "disagree" to items focusing on teacher behavior. Typically the questions do not solicit information about student learning. Because contemporary thinking about curriculum is interested in student learning, this form was developed to address differences in pedagogy and subject matter, learning outcomes, student demographics, and course level characteristic of education today. This form also grew out of a "one size fits all" approach to teaching evaluations and a recognition of the limitations of this practice. It offers information about how a particular course enhances student knowledge, sensitivities, and dispositions. The form gives students an opportunity to provide feedback that can be used to improve instruction.
Holistic Critical Thinking Scoring Rubric
This assessment tool uses a four-point classification schema that lists particular opposing reasoning skills for select criteria. One advantage of a rubric is that it offers clearly delineated components and scales for evaluating outcomes. This rubric explains how students' CTS will be evaluated, and it provides a consistent framework for the professor as evaluator. Users can add or delete any of the statements to reflect their institution's effort to measure CT. Like most rubrics, this form is likely to have high face validity since the items tend to be relevant or descriptive of the target concept. This rubric can be used to rate student work or to assess learning outcomes. Experienced evaluators should engage in a process leading to consensus regarding what kinds of things should be classified and in what ways.80 If used improperly or by inexperienced evaluators, unreliable results may occur.
Peer Evaluation of Group Presentation Form
This form offers a common set of criteria to be used by peers and the instructor to evaluate student-led group presentations regarding concepts, analysis of arguments or positions, and conclusions.81 Users have an opportunity to rate the degree to which each component was demonstrated. Open-ended questions give users an opportunity to cite examples of how concepts, the analysis of arguments or positions, and conclusions were demonstrated.
Table 8. Proposed Universal Criteria for Evaluating Students' Critical Thinking Skills
Accuracy |
Adequacy |
Clarity |
Completeness |
Consistency |
Depth |
Fairness |
Logic |
Precision |
Realism |
Relevance |
Significance |
Specificity |
Aside from the use of the above-mentioned assessment tools, Dexter et al. recommended that all schools develop universal criteria for evaluating students' development of critical thinking skills (82).
Their rationale for the proposed criteria is that if faculty give feedback using these criteria, graduates will internalize these skills and use them to monitor their own thinking and practice (see Table 4).
IMAGES
COMMENTS
Using the Holistic Critical Thinking Scoring Rubric. 1. Understand What this Rubric is Intended to Address. Critical thinking is the process of making purposeful, reflective and fair‐minded judgments about what to believe or what to do. Individuals and groups use critical thinking in problem solving and decision making.
Critical thinking is a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion. Framing Language This rubric is designed to be transdisciplinary, reflecting the recognition that success in all disciplines requires habits of inquiry and analysis ...
1. Understand what the Rubric is intended to Address. Critical thinking is the process of making purposeful, reflective and fair-minded judgments about what to believe or what to do. It is used in problem solving and decision making. This four level rubric treats this process as a set of cognitive skills supported by certain personal dispositions.
1. Understand the Construct. This four level rubric treats critical thinking as a set of cognitive skills supported by certain personal dispositions. To reach a judicious, purposive judgment a good critical thinker engages in analysis, interpretation, evaluation, inference, explanation, and meta-cognitive self-regulation.
Microsoft PowerPoint - Designing Rubrics to Assess Critical Thinking.pptx. 3:00. Traditional assessment measures such as multiple choice questions are a form of selected response measures designed for knowledge recall and sometimes for decision‐making from a selection of options. In such measures, students are asked to think critically in the ...
With a Critical Thinking Master Rubric by Richard Paul and Linda Elder Foundation for Critical Thinking. ... concepts, and then provide rubrics for scoring. In the appendix we provide a brief overview of the theory underlying the competencies. It is important to note that, only when teachers understand the foundations ...
Critical Thinking Scoring Rubric. Does not attempt to or fails to identify and summarize accurately. Summarizes issue, though some aspects are incorrect or confused. Nuances and key details are missing or glossed over. Clearly identifies the challenge and subsidiary, embedded, or implicit aspects of the issue.
California State University, Fresno Critical Thinking Scoring Guide. Relevant/penetrating questions clarify facts, concepts, and relationships. Questions are insightful and go beyond the obvious. Detects sources of bias even subtle or well-disguised. Uses principles of logic to explain fallacies in "if/then" statements.
The Holistic Critical Thinking Scoring Rubric (HCTSR) is a rating measure used to assess the quality of critical thinking displayed in a verbal presentation or written text. One would use the HCTSR to rate a written document or presentation where the presenter is required to be explicit about their thinking process. It can be used in any ...
Noreen Facione and I developed the Holistic Critical Thinking Scoring Rubric (HCTSR) in 1994 in response to requests for a tool which (a) could be used to evaluate a variety of educational work products including essays, presentations, and demonstrations, and (b) works as both a pedagogical device to guide people to know ...
Critical Thinking Scoring Rubric. Does not attempt to or fails to identify and summarize accurately. Summarizes issue, though some aspects are incorrect or confused. Nuances and key details are missing or glossed over. Clearly identifies the challenge and subsidiary, embedded, or implicit aspects of the issue.
19 How To Use The Holistic Critical Thinking Scoring Rubric 1. Understand what the Rubric is intended to Address. Critical thinking is the process of making purposeful, reflective and fair-minded judgments about what to believe or what to do. Individuals and groups use critical thinking in problem solving and decision making.
Adapted from the AAC&U Critical Thinking, Inquiry & Analysis, and Creative Thinking VALUE Rubrics. This work is licensed under CC BY-NC-SA 4.0. Texas A&M University Core Curriculum Critical Thinking Rubric Advanced 8 7 Competent 6 5 Developing 4 3 Beginner 2 1 Not Present 0 Explanation of Issue/Problem Issue/problem to be considered
CRITICAL THINKING SCORING RUBRIC Based on the scale from 1 (NOVICE) to 4 (ADVANCED), use the following definitions to rate each student's critical thinking achievement for each criteria. RATING CRITERIA (0-1) NOVICE Speaker/writer has limited strengths regarding this criterion. Intervention is needed for revision or rethinking 2
The critical thinking rubric developed by the Association of American Colleges and Universities (AAC&U) as part its Valid Assessment of ... & Leydens, J. A. (2000). Scoring rubric development: validity and reliability. Practical Assessment, Research and Evaluation, 7, 1-11. Google Scholar Nakhleh, M. B. (1992). Why some students don't learn ...
The Holistic Critical Thinking Scoring Rubric. Purpose To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs). Method Between May and August 2016, clinical cases were developed, shared, and ...
The effects of using a critical thinking scoring rubric to assess undergraduate students' reading skills. Journal of College Reading and Learning, 43(1), 31-58. Crossref. Google Scholar. National Education Goals Panel. (1991). The national education goals report: Building a nation of learners. U.S. Government Printing Office.
Finding shows that the rubric is helpful for teachers as the statistical analysis proved that the reliability score was α = 0.87 and the validity score was α = 0.892. The teacher of school B said that the rubric raised her view on critical reading, that it is not as simple as encouraging students to deliver opinion.
VALUE Rubrics - Critical Thinking. The VALUE rubrics were developed by teams of faculty experts representing colleges and universities across the United States through a process that examined many existing campus rubrics and related documents for each learning outcome and incorporated additional feedback from faculty.
Holistic Critical Thinking Scoring Rubric. This assessment tool uses a four-point classification schema that lists particular opposing reasoning skills for select criteria. One advantage of a rubric is that it offers clearly delineated components and scales for evaluating outcomes. This rubric explains how students' CTS will be evaluated, and ...
4.3 Critical Thinking Skills Rubric Selection and Application. ... Explain, Influence and Conclusion. ∆CRTH2 represents the change of the composite critical thinking score from the first assignment to the last assignment and can be either CCTSa or CCTSf, where CCTSa is calculated by averaging the four individual critical thinking dimensions ...
1. Understand the Construct. This four level rubric treats critical thinking as a set of cognitive skills supported by certain personal dispositions. To reach a judicious, purposive judgment a good critical thinker engages in analysis, interpretation, evaluation, inference, explanation, and meta-cognitive self-regulation.
The Critical Thinking Analytic Rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments ... Furthermore, the use of holistic rubrics to score critical thinking implies the expectation that students will likely perform similarly across the cognitive skills that ...