holistic critical thinking scoring rubric

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

The Holistic Critical Thinking Scoring Rubric

1994, Assessment Update

Related Papers

Nancy Heine

Purpose To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs). Method Between May and August 2016, clinical cases were developed, shared, and administered across seven medical schools (990 students participated). Raters were calibrated using training protocols, and guidelines were developed collaboratively across sites to standardize scoring. Data included scores from standardized patient encounters for history taking, physical examination, and PNs. Descriptive statistics were used to examine scores from the different assessment components. Generalizability studies (G-studies) using variance components were conducted to estimate reliability for composite scores. Results Validity evidence was collected for response process (rater perception), internal structure (variance components, reliability), relations to other variables (interassessment correlations), and consequences (composite score). Student performance varied by case and task. In the PNs, justification of differential diagnosis was the most discriminating task. G-studies showed that schools accounted for less than 1% of total variance; however, for the PNs, there were differences in scores for varying cases and tasks across schools, indicating a school effect. Composite score reliability was maximized when the PN was weighted between 30% and 40%. Raters preferred using case-specific scoring guidelines with clear pointscoring systems. Conclusions This multisite study presents validity evidence for PN scores based on scoring rubric and case-specific scoring guidelines that offer rigor and feedback for learners. Variability in PN scores across participating sites may signal different approaches to teaching clinical reasoning among medical schools.

Journal of Educational Measurement

George Engelhard

Assessing Writing

Valerie Meier

… Assessment, Research & Evaluation

Barbara Moskal

Miguel Fernández Álvarez

Scoring productive skills is usually difficult if raters are not well prepared and if they do not use analytic scales. Sometimes, even when scales are used, there may be differences in the scores given by each rater. That is why double marking is so important. When there are big differences in the scores given, it may be thought that some raters may not have developed a solid understanding of what each scale category represents and thus tend to use the different categories in an indiscriminate fashion. In some cases, the rater may not have sufficient background or expertise in order to make the fine discriminations that are required to employ the scale categories consistently. Some raters may use the rating scales reliably when evaluating the responses of some subgroups of examinees, but they do not use those scales reliably when evaluating the responses of other examinee subgroups (or perhaps when rating examinees on some of the tasks, but not on other tasks). Some raters are sensitive to fatigue effects (or inattention). As a rating session proceeds, these raters may tire (or their attention may wane), which may result in their becoming increasingly inconsistent in their application of the rating scales over time. This paper presents some of the problems raters have to face when scoring written compositions, and explains how FACETS can help identify those raters who are not being consistent in their scoring.

Educational Sciences: Theory & Practice

müge uluman

Beverly Baker

Researchers of high-stakes, subjectively scored writing assessments have done much work to better understand the process that raters go through in applying a rating scale to a language performance to arrive at a score. However, there is still unexplained, systematic variability in rater scoring that resists rater training (see Hoyt & Kerns, 1999; McNamara, 1996; Weigle, 2002; Weir, 2005). The consideration of individual differences in rater cognition may explain some of this rater variability. This mixed-method exploratory case study (Yin, 2009) examined rater decision making in a high-stakes writing assessment for preservice teachers in Quebec, Canada, focussing on individual differences in decision-making style, or “stylistic differences in cognitive style that could affect decision-making” (Thunholm, 2004, p. 932). The General Decision Making Style Inventory questionnaire (Scott & Bruce, 1995) was administered to six raters of a high-stakes writing exam in Quebec, and information on the following rater behaviours was also collected for their potential for providing additional information on individual decision-making style (DMS): (a) the frequency at which a rater decides to defer his or her score, (b) the underuse of failing score levels, and (c) the comments provided by raters during the exam rating about their decisions (collected through “write-aloud” protocols; Gigerenzer & Hoffrage, 1995). The relative merits of each of these sources of data are discussed in terms of their potential for tapping into the construct of rater DMS. Although score effects of DMS have yet to be established, it is concluded that despite the exploratory nature of this study, there is potential for the consideration of individual sociocognitive differences in accounting for some rater variability in scoring.

Journal of Behavioral Decision Making

Aaron Bonham , Hal Arkes

Three studies explored both the advantages of and subjects' preferences for a disaggregated judgment procedure and a holistic one. The task in our first two studies consisted of evaluating colleges; the third study asked participants to evaluate job applicants. Holistic ratings consisted of providing an overall evaluation while considering all of the characteristics of the evaluation objects; disaggregated ratings consisted of evaluating each cue independently. Participants also made paired comparisons of the evaluation objects. We constructed preference orders for the disaggregated method by aggregating these ratings (unweighted or weighted characteristics). To compare the holistic, disaggregated, and weighted-disaggregated method we regressed the four cues on the participant's holistic rating, on the linearly aggregated disaggregated ratings, and on the average weighted disaggregated rating, using the participant's “importance points” for each cue as weights. Both types of combined disaggregated ratings related more closely to the cues in terms of proportion of variance accounted for in Experiments 1 and 2. In addition, the disaggregated ratings were more closely related to the paired-comparison orderings, but Experiment 2 showed that this was true for a small set (10) but not a large set (60) of evaluation objects. Experiment 3 tested the “gamesmanship” hypothesis: People prefer holistic ratings because it is easier to incorporate illegitimate but appealing criteria into one's judgment. The results suggested that the disaggregated procedure generally produced sharper distinctions between the most relevant and least relevant cues. Participants in all three of these studies preferred the holistic ratings despite their statistical inferiority. Copyright © 2009 John Wiley & Sons, Ltd.

Currents in Pharmacy Teaching and Learning

Michael Peeters , Caren Steinmiller

Assad Rezigalla

Background: Several methods have been proposed for setting an examination pass mark (PM), and the Angoff’s method or its modified version is the preferred one. Selection of raters is important and affects the PM. Aims and Objectives: This study aims to investigate the selection of raters in the Angoff’s method and the impact of academic degrees and experience on the PM decided on. Materials and Methods: Type A MCQs examination was used in this study as a model. Raters with different academic degrees and experience participated in the study. Raters estimations were statiscally analyzed. Results: The selection of raters was crucial. Agreement among raters could be achieved by those with relevant qualifications and expertise. There was an association between high estimation, academic degree, expertise and high PM. Conclusion: Selection of raters for the Angoff’s method should include those with different academic degrees, backgrounds and experience so that a satisfactory PM may be reached by means of a reasonable agreement. Key words: Academic degree, Angoff’s method, experience, raters’ selection, setting pass mark

RELATED PAPERS

Jasmin Shahbazi

Life sciences

Poornima Sharma

Estevao Teixeira

Ferdinand Hodler et le Léman

Institut Ferdinand Hodler

Mohammad Khairul Azhar Abdul Razab

Papers on Anthropology

Sergey Vasilyev

Revista San Gregorio

Vladimir Vega

International Journal of English Literature and Social Sciences

Dr Sunil Kumar Mishra

Brain Research

Kazue Semba

Advances in Classification Research Online

Miguel Ángel Romero Ruíz

Analytica Chimica Acta

Milton Hearn

Advances in Atmospheric Sciences

R. Dwi Susanto

Current Pharmacogenomics and Personalized Medicine

tatiana zhiganova

MQU毕业证成绩单 麦考瑞大学学历学位认证

Hereditary cancer in clinical practice

Ignacio Agustin Blanco

Marine Ecology Progress Series

Hartvig Christie

Applied Catalysis B: Environmental

Stéphanie Lambert

Revista de Chimie

Veronica Chiriac

Urban Ecosystems

Martín Escobar

Grosir CD Golden Nick

Pusat Grosir Legging Panjang

Jurnal Neutrino

Aprilia Nur Azizah

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• Clerc Center | PK-12 & Outreach
• KDES | PK-8th Grade School (D.C. Metro Area)
• MSSD | 9th-12th Grade School (Nationwide)
• Gallaudet University Regional Centers
• Parent Advocacy App
• K-12 ASL Content Standards
• National Resources
• Youth Programs
• Academic Bowl
• Battle Of The Books
• National Literary Competition
• Youth Debate Bowl
• Youth Esports Series
• Bison Sports Camp
• Discover College and Careers (DC²)
• Financial Wizards
• Immerse Into ASL
• Alumni Relations
• Alumni Association
• Homecoming Weekend
• Class Giving
• Get Tickets / BisonPass
• Sport Calendars
• Cross Country
• Swimming & Diving
• Track & Field
• Indoor Track & Field
• Cheerleading
• Winter Cheerleading
• Human Resources
• Plan a Visit
• Request Info

• Areas of Study
• Accessible Human-Centered Computing
• American Sign Language
• Art and Media Design
• Communication Studies
• Data Science
• Deaf Studies
• Early Intervention Studies Graduate Programs
• Educational Neuroscience
• Hearing, Speech, and Language Sciences
• Information Technology
• International Development
• Interpretation and Translation
• Linguistics
• Mathematics
• Philosophy and Religion
• Physical Education & Recreation
• Public Affairs
• Public Health
• Sexuality and Gender Studies
• Social Work
• Theatre and Dance
• World Languages and Cultures
• B.A. in American Sign Language
• B.A. in Art and Media Design
• B.A. in Biology
• B.A. in Communication Studies
• B.A. in Communication Studies for Online Degree Completion Program
• B.A. in Deaf Studies
• B.A. in Deaf Studies for Online Degree Completion Program
• B.A. in Education with a Specialization in Early Childhood Education
• B.A. in Education with a Specialization in Elementary Education
• B.A. in English
• B.A. in Government
• B.A. in Government with a Specialization in Law
• B.A. in History
• B.A. in Interdisciplinary Spanish
• B.A. in International Studies
• B.A. in Interpretation
• B.A. in Mathematics
• B.A. in Philosophy
• B.A. in Psychology
• B.A. in Psychology for Online Degree Completion Program
• B.A. in Social Work (BSW)
• B.A. in Sociology
• B.A. in Sociology with a concentration in Criminology
• B.A. in Theatre Arts: Production/Performance
• B.A. or B.S. in Education with a Specialization in Secondary Education: Science, English, Mathematics or Social Studies
• B.S in Risk Management and Insurance
• B.S. in Accounting
• B.S. in Accounting for Online Degree Completion Program
• B.S. in Biology
• B.S. in Business Administration
• B.S. in Business Administration for Online Degree Completion Program
• B.S. in Information Technology
• B.S. in Mathematics
• B.S. in Physical Education and Recreation
• B.S. In Public Health
• General Education
• Honors Program
• Peace Corps Prep program
• Self-Directed Major
• M.A. in Counseling: Clinical Mental Health Counseling
• M.A. in Counseling: School Counseling
• M.A. in Deaf Education
• M.A. in Deaf Education Studies
• M.A. in Deaf Studies: Cultural Studies
• M.A. in Deaf Studies: Language and Human Rights
• M.A. in Early Childhood Education and Deaf Education
• M.A. in Early Intervention Studies
• M.A. in Elementary Education and Deaf Education
• M.A. in International Development
• M.A. in Interpretation: Combined Interpreting Practice and Research
• M.A. in Interpretation: Interpreting Research
• M.A. in Linguistics
• M.A. in Secondary Education and Deaf Education
• M.A. in Sign Language Education
• M.S. in Accessible Human-Centered Computing
• M.S. in Speech-Language Pathology
• Master of Social Work (MSW)
• Au.D. in Audiology
• Ed.D. in Transformational Leadership and Administration in Deaf Education
• Ph.D. in Clinical Psychology
• Ph.D. in Critical Studies in the Education of Deaf Learners
• Ph.D. in Hearing, Speech, and Language Sciences
• Ph.D. in Linguistics
• Ph.D. in Translation and Interpreting Studies
• Ph.D. Program in Educational Neuroscience (PEN)
• Individual Courses and Training
• Summer On-Campus Courses
• Summer Online Courses
• Certificates
• Certificate in Sexuality and Gender Studies
• Educating Deaf Students with Disabilities (online, post-bachelor’s)
• American Sign Language and English Bilingual Early Childhood Deaf Education: Birth to 5 (online, post-bachelor’s)
• Peer Mentor Training (low-residency/hybrid, post-bachelor’s)
• Early Intervention Studies Graduate Certificate
• Online Degree Programs
• ODCP Minor in Communication Studies
• ODCP Minor in Deaf Studies
• ODCP Minor in Psychology
• ODCP Minor in Writing
• Online Degree Program General Education Curriculum
• University Capstone Honors for Online Degree Completion Program

Quick Links

• PK-12 & Outreach
• NSO Schedule

Developing a Scoring Criteria (Rubrics)

College Hall 410A

(202) 559-5370

202.651.5085

DISCLAIMER: This data in this section is fictitious and does not, in any way, represent any of the programs at Gallaudet University. This information is intended only as examples.  

Types of Scoring Criteria (Rubrics)

A rubric is a scoring guide used to assess performance against a set of criteria. At a minimum, it is a list of the components you are looking for when you evaluate an assignment. At its most advanced, it is a tool that divides an assignment into its parts and provides explicit expectations of acceptable and unacceptable levels of performance for each component. 

Types of Rubrics

1 – Checklists, the least complex form of scoring system, are simple lists indicating the presence, NOT the quality, of the elements. Therefore, checklists are NOT frequently used in higher education for program-level assessment. But faculty may find them useful for scoring and giving feedback on minor student assignments or practice/drafts of assignments. 

Example 1: Critical Thinking Checklist 

The student…

__ Accurately interprets evidence, statements, graphics, questions, etc.  

__ Identifies the salient arguments (reasons and claims)  

__ Offers analyzes and evaluates major alternative points of view  

__ Draws warranted, judicious, non-fallacious conclusions  

__ Justifies key results and procedures, explains assumptions and reasons  

__ Fair-mindedly follows where evidence and reasons lead 

Example 2: Presentation Checklist 

The student… 

__ engaged audience  

__ used an academic or consultative American Sign Language (ASL) register  

__ used adequate ASL syntactic and semantic features  

__ cited references adequately in ASL  

__ stayed within allotted time  

__ managed PowerPoint presentation technology smoothly 

2 – Basic Rating Scales are checklists of criteria that evaluate the quality of elements and include a scoring system. The main drawback with rating scales is that the meaning of the numeric ratings can be vague. Without descriptors for the ratings, the raters must make a judgment based on their perception of the meanings of the terms. For the same presentation, one rater might think a student rated “good,” and another rater might feel the same student was “marginal.” 

Example: Basic Rating Scale for Critical Thinking

3 – Holistic Rating Scales use a short narrative of characteristics to award a single score based on an overall impression of a student’s performance on a task. A drawback to using holistic rating scales is that they do not provide specific areas of strengths and weaknesses and therefore are less useful to help you focus your improvement efforts. Use a holistic rating scale when the projects to be assessed will vary greatly (e.g., independent study projects submitted in a capstone course). Or when the number of assignments to be evaluated is significant (e.g., reviewing all the essays from applicants to determine who will need developmental courses). 

Example: Holistic Rating Scale for Critical Thinking Scoring

Rating scale.

The Holistic Critical Thinking Scoring Rubric: A Tool for Developing and Evaluating Critical Thinking. Retrieved April 12, 2010 from Insight Assessment . 4 – Analytic Rating Scales are rubrics that include explicit performance expectations for each possible rating, for each criterion. Analytic rating scales are especially appropriate for complex learning tasks with multiple criteria. Evaluate carefully whether this the most appropriate tool for your assessment needs. They can provide more detailed feedback on student performance; more consistent scoring among raters, but the disadvantage is that they can be time-consuming to develop and apply. Results can be aggregated to provide detailed information on the strengths and weaknesses of a program. Example: Critical Thinking Portion of the Gallaudet University Rubric for Assessing Written English 

Ideas and Critical Thinking

Steps for creating an analytic rating scale (rubric) from scratch.

There are different ways to approach building an analytic rating scale: logical or organic. For both the logical and the organic model, steps 1-3 are the same. 

Steps 1 – 3: Logical AND Organic Method

Determine the best tool.

• if there are multiple aspects of the product or process to be considered
• if a basic rating scale or holistic rating scale cannot provide the breadth of assessment you need.

Building the Shell

• Specify the skills, knowledge, and/or behaviors that you will be looking for.
• Limit the characteristics to those that are most important to the assessment.

The Columns

• Develop a rating scale with the levels of mastery that is meaningful.

Tip: Adding numbers to the ratings can make scoring easier. However, if you plan to use the rating scale for course-level assessment grading as well, a meaning must be attached to that score. For example, what is the minimum score that would be considered acceptable for a “C.” 

Components of Analytic Rating Scales  

• Criteria that link to the relevant learning objectives
• Rating scale that distinguishes between levels of mastery
• Descriptions that clarify the meaning of each criterion, at each level of mastery

Other possible descriptors include:

• Exemplary, Proficient, Marginal, Unacceptable
• Advanced, High, Intermediate, Novice
• Beginning, Developing, Accomplished, Exemplary
• Outstanding, Good, Satisfactory, Unsatisfactory

Writing the Performance Descriptors in the Cells

examples of inconsistent performance characteristics and suggested corrections.

• Use either the logical or the organic method to write the descriptions for each criterion at each level of mastery.

Tips: Keep list of characteristics manageable by only including critical evaluative components. Extremely long, overly-detailed lists make a rating scale hard to use. 

In addition to having descriptions brief, the language should be consistent. Below are several ideas to keep descriptors consistent: 

Keep the aspects of a performance stay the same across the levels but adding adjectives or adverbial phrases to show the qualitative difference  

A word of warning: numeric references on their own can be misleading. They are best teamed with a qualitative reference (eg three appropriate and relevant examples) to avoid ignoring quality at the expense of quantity. 

Steps 5-6: Logical AND Organic Methods

• Part 6. Scoring Rubric Group Orientation and Calibration” for directions for this process.
• Review and revise.

Steps for Adapting an Existing Analytic Rating Scale (Rubric)

• Does the rating scale relate to all or most the outcome(s) I need to assess?
• Does it address anything extraneous?
• Add missing criteria
• Delete extraneous criteria
• Adapt the rating scale
• Edit the performance descriptors
• Test the rating scale.
• Review and revise again, if necessary.

Uses of Rating Scales (Rubrics)

Use rating scales for program-level assessment to see trends in strengths and weaknesses of groups of students. 

• To evaluate a holistic project (e.g., theses, exhibitions, research project) in capstone course that pulls together all that students have learned in the program.
• Supervisors might use a rating scale developed by the program to evaluate students’ field experience and provide feedback to both the student and the program.
• Aggregate the scores of rating scale used to evaluate a course-level assignment. For example, the Biology department decides to develop a rating scale to evaluate students’ reports from 300- and 400-level sections. The professors use the scores to determine the students’ grades and provide students with feedback for improvement. The scores are also given to the department’s Assessment Coordinator to summarize to determine how well they are meeting their student learning outcome, “Make appropriate inferences and deductions from biological information.”

For more information on using course-level assessment to provide feedback to students and to determine grades, see University of Hawaii’s “ Part 7. Suggestions for Using Rubrics in Courses ” and the section on Converting Rubric Scores to Grades in Craig A. Mertler’s “Designing Scoring Rubrics for Your Classroom”.

Sample Rating Scales (Rubrics)

• Rubric Bank  (University of Hawai’i at Manoa)
• Sample Rubrics by type  (Winona State University)
• Rubrics  (UC, Berkeley)

Adapted from sources below:  

Allen, Mary. (January, 2006). Assessment Workshop Material . California State University, Bakersfield. Retrieved DATE from  http://www.csub.edu/TLC/options/resources/handouts/AllenWorkshopHandoutJan06.pdf  

http://www.uhm.hawaii.edu/assessment/howto/rubrics.htm  

http://www.teachervision.fen.com/teaching-methods-and-management/rubrics/4523.html?detoured=1  

Mueller, Jon. (2001). Rubrics. Authentic Assessment Toolbox. Retrieved April 12, 2010 from http://jonathan.mueller.faculty.noctrl.edu/toolbox/rubrics.htm   

http://en.wikipedia.org/wiki/Rubric_(academic)    

Tierney, Robin & Marielle Simon. (2004). What’s Still Wrong With Rubrics: Focusing on the Consistency of Performance Criteria Across Scale Levels . Practical Assessment, Research & Evaluation, 9(2).  

At a Glance

• Quick Facts
• University Leadership
• History & Traditions
• Accreditation
• Consumer Information
• Our 10-Year Vision: The Gallaudet Promise
• Annual Report of Achievements (ARA)
• The Signing Ecosystem
• Not Your Average University

Our Community

• Library & Archives
• Technology Support
• Interpreting Requests
• Ombuds Support
• Health and Wellness Programs
• Profile & Web Edits

Visit Gallaudet

• Explore Our Campus
• Virtual Tour
• Maps & Directions
• Shuttle Bus Schedule
• Kellogg Conference Hotel
• Welcome Center
• National Deaf Life Museum
• Apple Guide Maps

Engage Today

• Work at Gallaudet / Clerc Center
• Social Media Channels
• University Wide Events
• Sponsorship Requests
• Data Requests
• Media Inquiries
• Gallaudet Today Magazine
• Giving at Gallaudet
• Financial Aid
• Registrar’s Office
• Residence Life & Housing
• Safety & Security
• Undergraduate Admissions
• Graduate Admissions
• University Communications
• Clerc Center

Gallaudet University, chartered in 1864, is a private university for deaf and hard of hearing students.

Copyright © 2024 Gallaudet University. All rights reserved.

• Accessibility
• Cookie Consent Notice
• Privacy Policy
• File a Report

800 Florida Avenue NE, Washington, D.C. 20002

Holistic Critical Thinking Rubric by Peter A. Facione and Noreen C. Facione Use the following rubric to think about HOW you've made your historical argument. Critical thinking by historians or anyone else requires constructing arguments based on solid evidence. In contrast, opinion, close-mindedness, or irrationality reflect a lack of critical thinking. In such cases, one merely expresses preconceptions and biases not based on valid, supoprting evidence. On the scale below, 4 represents the highest level of critical thinking (A or A), followed by 3 (C), 2 (D), and 1 (F). Aspire to apply your best criitical thinkings skills in all your assignments. 4: Consistently does all or almost all of the following: Accurately interprets evidence, statements, graphics, questions, etc. Identifies the salient arguments (reasons and claims) pro and con. Thoughtfully analyzes and evaluates major alternative points of view. Draws warranted, judicious, non-fallacious conclusions. Justifies key results and procedures, explains assumptions and reasons. Fair-mindedly follows where evidence and reasons lead. 3: Does most or many of the following: Accurately interprets evidence, statements, graphics, questions, etc. Identifies relevant arguments (reasons and claims) pro and con. Offers analyses and evaluations of obvious alternative points of view. Draws warranted, non-fallacious conclusions. Justifies some results or procedures, explains reasons. Fair-mindedly follows where evidence and reasons lead. 2: Does most or many of the following Misinterprets evidence, statements, graphics, questions, etc. Fails to identify strong, relevant counter-arguments. Ignores or superficially evaluates obvious alternative points of view. Draws unwarranted or fallacious conclusions. Justifies few results or procedures, seldom explains reasons. Regardless of the evidence or reasons, maintains or defends views based on self-interest or preconceptions. 1: Consistently does all or almost all of the following: Offers biased interpretations of evidence, statements, graphics, questions, information, or the points of view of others. Fails to identify or hastily dismisses strong, relevant counter-arguments. Ignores or superficially evaluates obvious alternative points of view. Argues using fallacious or irrelevant reasons, and unwarranted claims. Does not justify results or procedures, nor explain reasons. Regardless of the evidence or reasons, maintains or defends views based on self-interest or preconceptions. Exhibits close-mindedness or hostility to reason. [Source: 1994, Peter A. Facione, Noreen C. Facione, and The California Academic Press. Copyright 2002, Insight Assessment and the California Academic Press. Critical Thinking as Reasoned Judgment , The Album , Page 24.]

• ADEA Connect

• Communities
• Career Opportunities
• New Thinking
• ADEA Governance
• House of Delegates
• Board of Directors
• Advisory Committees
• Sections and Special Interest Groups
• Governance Documents and Publications
• Dental Faculty Code of Conduct
• ADEAGies Foundation
• About ADEAGies Foundation
• ADEAGies Newsroom
• Gies Awards
• Press Center
• Strategic Directions
• 2023 Annual Report
• ADEA Membership
• Institutions
• Faculty and Staff
• Individuals
• Corporations
• ADEA Members
• Predoctoral Dental
• Allied Dental
• Nonfederal Advanced Dental
• U.S. Federal Dental
• Students, Residents and Fellows
• Corporate Members
• Member Directory
• Directory of Institutional Members (DIM)
• 5 Questions With
• ADEA Member to Member Recruitment
• Students, Residents, and Fellows
• Information For
• Deans & Program Directors
• Current Students & Residents
• Prospective Students
• Educational Meetings
• Upcoming Events
• 2025 Annual Session & Exhibition
• eLearn Webinars
• Past Events
• Professional Development
• eLearn Micro-credentials
• Leadership Institute
• Leadership Institute Alumni Association (LIAA)
• Faculty Development Programs
• ADEA Scholarships, Awards and Fellowships
• Academic Fellowship
• For Students
• For Dental Educators
• For Leadership Institute Fellows
• Teaching Resources
• ADEA weTeach®
• MedEdPORTAL

Critical Thinking Skills Toolbox

• Resources for Teaching
• Policy Topics
• Task Force Report
• Opioid Epidemic
• Financing Dental Education
• Holistic Review
• Sex-based Health Differences
• Access, Diversity and Inclusion
• ADEA Commission on Change and Innovation in Dental Education
• Tool Resources
• Campus Liaisons
• Policy Resources
• Policy Publications
• Holistic Review Workshops
• Leading Conversations Webinar Series
• Collaborations
• Summer Health Professions Education Program
• Minority Dental Faculty Development Program
• Federal Advocacy
• Dental School Legislators
• Policy Letters and Memos
• Legislative Process
• Federal Advocacy Toolkit
• State Information
• Opioid Abuse
• Tracking Map
• Loan Forgiveness Programs
• State Advocacy Toolkit
• Canadian Information
• Dental Schools
• Provincial Information
• ADEA Advocate
• Books and Guides
• About ADEA Publications
• 2023-24 Official Guide
• Dental School Explorer
• Dental Education Trends
• Ordering Publications
• ADEA Bookstore
• Newsletters
• About ADEA Newsletters
• Bulletin of Dental Education
• Charting Progress
• Subscribe to Newsletter
• Journal of Dental Education
• Subscriptions
• Submissions FAQs
• Data, Analysis and Research
• Educational Institutions
• Applicants, Enrollees and Graduates
• Dental School Seniors
• ADEA AADSAS® (Dental School)
• AADSAS Applicants
• Health Professions Advisors
• Admissions Officers
• ADEA CAAPID® (International Dentists)
• CAAPID Applicants
• Program Finder
• ADEA DHCAS® (Dental Hygiene Programs)
• DHCAS Applicants
• Program Directors
• ADEA PASS® (Advanced Dental Education Programs)
• PASS Applicants
• PASS Evaluators
• DentEd Jobs
• Information For:

• Introduction
• Overview of Critical Thinking Skills
• Teaching Observations
• Avenues for Research

CTS Tools for Faculty and Student Assessment

• Critical Thinking and Assessment
• Conclusions
• Bibliography
• Helpful Links
• Appendix A. Author's Impressions of Vignettes

A number of critical thinking skills inventories and measures have been developed:

     Watson-Glaser Critical Thinking Appraisal (WGCTA)      Cornell Critical Thinking Test      California Critical Thinking Disposition Inventory (CCTDI)      California Critical Thinking Skills Test (CCTST)      Health Science Reasoning Test (HSRT)      Professional Judgment Rating Form (PJRF)      Teaching for Thinking Student Course Evaluation Form      Holistic Critical Thinking Scoring Rubric      Peer Evaluation of Group Presentation Form

Excluding the Watson-Glaser Critical Thinking Appraisal and the Cornell Critical Thinking Test, Facione and Facione developed the critical thinking skills instruments listed above. However, it is important to point out that all of these measures are of questionable utility for dental educators because their content is general rather than dental education specific. (See Critical Thinking and Assessment .)

Table 7. Purposes of Critical Thinking Skills Instruments

  Reliability and Validity

Reliability means that individual scores from an instrument should be the same or nearly the same from one administration of the instrument to another. The instrument can be assumed to be free of bias and measurement error (68). Alpha coefficients are often used to report an estimate of internal consistency. Scores of .70 or higher indicate that the instrument has high reliability when the stakes are moderate. Scores of .80 and higher are appropriate when the stakes are high.

Validity means that individual scores from a particular instrument are meaningful, make sense, and allow researchers to draw conclusions from the sample to the population that is being studied (69) Researchers often refer to "content" or "face" validity. Content validity or face validity is the extent to which questions on an instrument are representative of the possible questions that a researcher could ask about that particular content or skills.

Watson-Glaser Critical Thinking Appraisal-FS (WGCTA-FS)

The WGCTA-FS is a 40-item inventory created to replace Forms A and B of the original test, which participants reported was too long.70 This inventory assesses test takers' skills in:

     (a) Inference: the extent to which the individual recognizes whether assumptions are clearly stated      (b) Recognition of assumptions: whether an individual recognizes whether assumptions are clearly stated      (c) Deduction: whether an individual decides if certain conclusions follow the information provided      (d) Interpretation: whether an individual considers evidence provided and determines whether generalizations from data are warranted      (e) Evaluation of arguments: whether an individual distinguishes strong and relevant arguments from weak and irrelevant arguments

Researchers investigated the reliability and validity of the WGCTA-FS for subjects in academic fields. Participants included 586 university students. Internal consistencies for the total WGCTA-FS among students majoring in psychology, educational psychology, and special education, including undergraduates and graduates, ranged from .74 to .92. The correlations between course grades and total WGCTA-FS scores for all groups ranged from .24 to .62 and were significant at the p < .05 of p < .01. In addition, internal consistency and test-retest reliability for the WGCTA-FS have been measured as .81. The WGCTA-FS was found to be a reliable and valid instrument for measuring critical thinking (71).

Cornell Critical Thinking Test (CCTT)

There are two forms of the CCTT, X and Z. Form X is for students in grades 4-14. Form Z is for advanced and gifted high school students, undergraduate and graduate students, and adults. Reliability estimates for Form Z range from .49 to .87 across the 42 groups who have been tested. Measures of validity were computed in standard conditions, roughly defined as conditions that do not adversely affect test performance. Correlations between Level Z and other measures of critical thinking are about .50.72 The CCTT is reportedly as predictive of graduate school grades as the Graduate Record Exam (GRE), a measure of aptitude, and the Miller Analogies Test, and tends to correlate between .2 and .4.73

California Critical Thinking Disposition Inventory (CCTDI)

Facione and Facione have reported significant relationships between the CCTDI and the CCTST. When faculty focus on critical thinking in planning curriculum development, modest cross-sectional and longitudinal gains have been demonstrated in students' CTS.74 The CCTDI consists of seven subscales and an overall score. The recommended cut-off score for each scale is 40, the suggested target score is 50, and the maximum score is 60. Scores below 40 on a specific scale are weak in that CT disposition, and scores above 50 on a scale are strong in that dispositional aspect. An overall score of 280 shows serious deficiency in disposition toward CT, while an overall score of 350 (while rare) shows across the board strength. The seven subscales are analyticity, self-confidence, inquisitiveness, maturity, open-mindedness, systematicity, and truth seeking (75).

In a study of instructional strategies and their influence on the development of critical thinking among undergraduate nursing students, Tiwari, Lai, and Yuen found that, compared with lecture students, PBL students showed significantly greater improvement in overall CCTDI (p = .0048), Truth seeking (p = .0008), Analyticity (p =.0368) and Critical Thinking Self-confidence (p =.0342) subscales from the first to the second time points; in overall CCTDI (p = .0083), Truth seeking (p= .0090), and Analyticity (p =.0354) subscales from the second to the third time points; and in Truth seeking (p = .0173) and Systematicity (p = .0440) subscales scores from the first to the fourth time points (76). California Critical Thinking Skills Test (CCTST)

Studies have shown the California Critical Thinking Skills Test captured gain scores in students' critical thinking over one quarter or one semester. Multiple health science programs have demonstrated significant gains in students' critical thinking using site-specific curriculum. Studies conducted to control for re-test bias showed no testing effect from pre- to post-test means using two independent groups of CT students. Since behavioral science measures can be impacted by social-desirability bias-the participant's desire to answer in ways that would please the researcher-researchers are urged to have participants take the Marlowe Crowne Social Desirability Scale simultaneously when measuring pre- and post-test changes in critical thinking skills. The CCTST is a 34-item instrument. This test has been correlated with the CCTDI with a sample of 1,557 nursing education students. Results show that, r = .201, and the relationship between the CCTST and the CCTDI is significant at p< .001. Significant relationships between CCTST and other measures including the GRE total, GRE-analytic, GRE-Verbal, GRE-Quantitative, the WGCTA, and the SAT Math and Verbal have also been reported. The two forms of the CCTST, A and B, are considered statistically significant. Depending on the testing, context KR-20 alphas range from .70 to .75. The newest version is CCTST Form 2000, and depending on the testing context, KR-20 alphas range from .78-.84.77

The Health Science Reasoning Test (HSRT)

Items within this inventory cover the domain of CT cognitive skills identified by a Delphi group of experts whose work resulted in the development of the CCTDI and CCTST. This test measures health science undergraduate and graduate students' CTS. Although test items are set in health sciences and clinical practice contexts, test takers are not required to have discipline-specific health sciences knowledge. For this reason, the test may have limited utility in dental education (78).

Preliminary estimates of internal consistency show that overall KR-20 coefficients range from .77 to .83.79 The instrument has moderate reliability on analysis and inference subscales, although the factor loadings appear adequate. The low K-20 coefficients may be result of small sample size, variance in item response, or both (see following table).

Table 8. Estimates of Internal Consistency and Factor Loading by Subscale for HSRT

Professional Judgment Rating Form (PJRF)

The scale consists of two sets of descriptors. The first set relates primarily to the attitudinal (habits of mind) dimension of CT. The second set relates primarily to CTS.

A single rater should know the student well enough to respond to at least 17 or the 20 descriptors with confidence. If not, the validity of the ratings may be questionable. If a single rater is used and ratings over time show some consistency, comparisons between ratings may be used to assess changes. If more than one rater is used, then inter-rater reliability must be established among the raters to yield meaningful results. While the PJRF can be used to assess the effectiveness of training programs for individuals or groups, the evaluation of participants' actual skills are best measured by an objective tool such as the California Critical Thinking Skills Test.

Teaching for Thinking Student Course Evaluation Form

Course evaluations typically ask for responses of "agree" or "disagree" to items focusing on teacher behavior. Typically the questions do not solicit information about student learning. Because contemporary thinking about curriculum is interested in student learning, this form was developed to address differences in pedagogy and subject matter, learning outcomes, student demographics, and course level characteristic of education today. This form also grew out of a "one size fits all" approach to teaching evaluations and a recognition of the limitations of this practice. It offers information about how a particular course enhances student knowledge, sensitivities, and dispositions. The form gives students an opportunity to provide feedback that can be used to improve instruction.

Holistic Critical Thinking Scoring Rubric

This assessment tool uses a four-point classification schema that lists particular opposing reasoning skills for select criteria. One advantage of a rubric is that it offers clearly delineated components and scales for evaluating outcomes. This rubric explains how students' CTS will be evaluated, and it provides a consistent framework for the professor as evaluator. Users can add or delete any of the statements to reflect their institution's effort to measure CT. Like most rubrics, this form is likely to have high face validity since the items tend to be relevant or descriptive of the target concept. This rubric can be used to rate student work or to assess learning outcomes. Experienced evaluators should engage in a process leading to consensus regarding what kinds of things should be classified and in what ways.80 If used improperly or by inexperienced evaluators, unreliable results may occur.

Peer Evaluation of Group Presentation Form

This form offers a common set of criteria to be used by peers and the instructor to evaluate student-led group presentations regarding concepts, analysis of arguments or positions, and conclusions.81 Users have an opportunity to rate the degree to which each component was demonstrated. Open-ended questions give users an opportunity to cite examples of how concepts, the analysis of arguments or positions, and conclusions were demonstrated.

Table 8. Proposed Universal Criteria for Evaluating Students' Critical Thinking Skills 

Aside from the use of the above-mentioned assessment tools, Dexter et al. recommended that all schools develop universal criteria for evaluating students' development of critical thinking skills (82).

Their rationale for the proposed criteria is that if faculty give feedback using these criteria, graduates will internalize these skills and use them to monitor their own thinking and practice (see Table 4).

• Application Information
• ADEA GoDental
• ADEA AADSAS
• ADEA CAAPID
• Events & Professional Development
• Scholarships, Awards & Fellowships
• Publications & Data
• Official Guide to Dental Schools
• Data, Analysis & Research
• Follow Us On:

• ADEA Privacy Policy
• Terms of Use
• Website Feedback
• Website Help

Instructions for Using the Holistic Critical Thinking Scoring Rubric

1. understand the construct..

This four level rubric treats critical thinking as a set of cognitive skills supported by certain personal dispositions. To reach a judicious, purposive judgment a good critical thinker engages in analysis, interpretation, evaluation, inference, explanation, and meta-cognitive self-regulation. The disposition to pursue fair-mindedly and open-mindedly the reasons and evidence wherever they lead is crucial to reaching sound, objective decisions and resolutions to complex, ill-structured problems. So are the other critical thinking dispositions, such as systematicity, reasoning self-confidence, cognitive maturity, analyticity, and inquisitiveness. [For details on the articulation of this concept refer to Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. ERIC Document Number: ED 315 423.]

2. Differentiate and Focus

Holistic scoring requires focus. In any essay, presentation, or clinical practice setting many elements must come together for overall success: critical thinking, content knowledge, and technical skill (craftsmanship). Deficits or strengths in any of these can draw the attention of the rater. However, in scoring for any one of the three, one must attempt to focus the evaluation on that element to the exclusion of the other two.

• Ideally, in a training session with other raters one will examine sample essays (videotaped presentations, etc.) which are paradigmatic of each of the four levels. Without prior knowledge of their level, raters will be asked to evaluate and assign ratings to these samples. After comparing these preliminary ratings, collaborative analysis with the other raters and the trainer is used to achieve consistency of expectations among those who will be involved in rating the actual cases. Training, practice, and inter-rater reliability are the keys to a high quality assessment.
• Usually, two raters will evaluate each essay/assignment/project/performance. If they disagree there are three possible ways that resolution can be achieved: (a) by mutual conversation between the two raters, (b) by using an independent third rater, or (c) by taking the average of the two initial ratings. The averaging strategy is strongly discouraged. Discrepancies between raters of more than one level suggest that detailed conversations about the CT construct and about project expectations are in order. This rubric is a four level scale, half point scoring is inconsistent with its intent and conceptual structure. Further, at this point in its history, the art and science of holistic critical thinking evaluation cannot justify asserting half-level differentiations.
• If working alone, or without paradigm samples, one can achieve a greater level of internal consistency by not assigning final ratings until a number of essays/projects/performances/assignments have been viewed and given preliminary ratings. Frequently natural clusters or groupings of similar quality soon come to be discernible. At that point one can be more confident in assigning a firmer critical thinking score using this four level rubric. After assigning preliminary ratings, a review of the entire set assures greater internal consistency and fairness in the final ratings.

� Copyright 1996, CAP, Inc. All rights reserved.