Methods of blueprint of a pediatric course in innovative curriculum

Received: 03-Aug-2021 Accepted: 27-Aug-2021 Published: 31-Aug-2021

Introduction

In medical education, a valid assessment is denoted as "the assessment which measures what it is supposed to measure." A proper academic assessment should ensure validity [1]. A valid assessment will guarantee that the examinee achieved an acceptable level of performance and competencies [2]. Evidence of assessment validity are many; among them is content validity. Content validity can be ensured through assessment blueprinting (or test specifications). Blueprint can help the assessors to avoid under or overrepresentation of assessment contents, which can challenge the assessment validity [3].

Blueprint is "a template that determines the representativeness of test content; it lists the number and type of questions across the course content, with learning objectives and relative weighting given to each topic" [4,5].

Weightages of blueprint contents can be determined by different methods such as the importance of topic/theme, contact hours, and numbers of specific learning outcomes [6]. Other methods of aligning blueprints contents were reported in the literature such as competency, organs or topics, disciplines, disease, presentations, and tasks [7] or the learning targets [8].

The important method is a widely used method of determining weightages in the blueprint. The importance of each them/topic is determined by multiplying the impact and frequency of the theme/topic [5]. The subject expert determines the impact of the topic. Commonly is calculated as; if not urgent, score 1; if serious but not life-threatening, score 2; and if life-threatening, score 3. The frequency of the topic is considered from medical records or the judgment of experts. Commonly the frequency is calculated as: if rarely seen, score 1; if relatively common, score 2; and if common, score 3 [1]. Both methods of Contact Hours (CH) and numbers of Specific Learning Outcomes (SLO) depend on the time spent on teaching and the number of SLO of the topic respectively [6]. The weightage is then calculated as a percentage out of the total course teaching time or SLO. Both methods need on extended calculation and no room for subject intervention apart from CH determination and the creation or selection of SLO.

This study aims to compare three methods of blueprint weightage, namely importance, contact hours and specific learning outcomes.

Material and Methods

Study design and setting

The study design is cross-sectional. The study was conducted at the College of Medicine, University of Bisha, Saudi Arabia (August 2019 to March 2020).

Study tools

The study uses the blueprint of the pediatrics course (MEU 211). The MEU 211 is conducted in semester two, level 12. The course is seven credit hours (three theoretical + four practical). The course is formed of ten themes with different numbers of SLO and contact hours (Table 1). The course was constructed by the department of pediatrics, curriculum committee and consultation of subject experts. The final course examination is formed of theoretical and practical parts. The theoretical examination is formed of type A MCQs (60). The practical examination formed of two portions, OSCE, consists of 11 stations, and OSPE, formed of 20 stations.

Tab. 1. Shows the themes, number of specific learning outcomes and contact hours

Methods of blueprint weightage

Three blueprints were designed for the final theoretical examination. The blueprint of importance method was prepared according to Patil, et al. [8]. Where they used rows for the subject or clinical problems to be tested and columns for the expected task to be tested [9-11].

The weightages of SLO, CH of each theme were determined as a percentage out of the total number, of course, SLO and CH, respectively [7].

Ethical consideration

The study was approved by the research and ethics committees at the College of Medicine, University of Bisha.

Statistical analysis

The obtained data were entered and analysed by SPSS v20. A p-value of <0.05 was considered statistically significant.

Results

Examination blueprints

Three examination blueprints were designed. The number, percentages of course theme/topic blueprints are presented in table 2.

Tab. 2. Shows the percentages of themes according to the specific learning outcomes and contact hours and the importance and the number of items in the theoretical examination

According to the weightage of SLO per theme/topic, the CNS, and GIT themes have the highest percentage, followed by CVS, infection, Neonatology, renal, Hemato-oncology, Endocrinology, Respiratory and Genetic. According to the weightage of CH per theme/topic, CNS, GIT, and Respiratory themes have the highest percentages; meanwhile, CVS, Neonatology, Hemato-oncology and Endocrinology have the same percentage followed by renal, infection and Genetic. The important method considered CNS is the most important theme, followed by Endocrinology, Renal, Respiratory, neonatology (equally), GIT, CVS, Hematooncology, infection and Genetic, respectively (Figure 1 and Table 2).

Figure 1: Shows the themes and the weightages of themes according to the number of specific learning outcomes and contact hours and the importance. SLO, specific learning outcome; CH, contact hour

CNS theme has the highest percentage according to CH and importance methods. GIT theme has the highest percentage according to the weightage of SLO and the second-highest according to the CH method. The genetic theme has the lowest percentage in the three methods of blueprinting, (Figure 1 and Table 2).

Discussion

In this paper, we compare the importance, SLO, and CH methods of determining weightages in the blueprint. Particular attention is paid to the important method of weightages. The three methods ended with different weightages for each theme/ topic and consequent different number of items representing in the examination.

The important method is superior to the SLO and CH methods and allows a fair representation of the theoretical and practical contents of the course. This is agrees with the University of Calgary (Canada), Exam Soft publications and others [12,13]. The superiority of the important method is based on the experience and judgment of subject experts and data from objective records.

In comparison to the important method, the CNS theme was the most represented in all methods. Neonatology was in the third rank and genetics is least presented among the three methods. Despite the agreement in theme ranking, chi-square and ANOVA Single-factor tests describe the association between the three methods as non-significant. Moreover, in general, every two methods have at least three themes in the same rank. However, statistically, the T-test between the three methods is non-significant.

Importance methods of weightage have solid support in literature in regards to implementation and efficiency [1,9]. The importance is considered as an objective method as it depends on the clinical presentation and the impact of the situation. The clinical presentation or frequency of the cases can be obtained from data records of hospitals or health authorities. Subjectivity is limited to the judgment of experts about the impact of the clinical case, which is one arm of the important method equation of weightage [12]. This subjectivity provides a positive impact on the method and enables it to be more flexible. Through expert judgment, the emerging diseases, outbreaks and the changing community's needs can be addressed without a change in the curricula.

In contrast to importance methods, both SLO and CH depend on direct calculation. Both methods need extensive mathematical work, ignore the impact of different diseases and treat them similarly. Moreover, in both methods, the subject expert interference is limited to SLO construction and CH determination during the early stages of course development. Any change or emerging new health requires dramatic changes in the curricula with developing new SLO and CH.

The importance method can be easily applied in clinical phases of medical curricula and allow a fair distribution of the theory or practical contents, which ultimately assure accountable graduates. In the pre-clinical phases of traditional curricula, the determination of clinical presentation and the impact of basic knowledge are a little bit difficult, so using the SLO or CH method is justifiable. While in the pre-clinical integrated course, the cooperation with clinicians will help the use of important methods [14,15].

Conclusion

Three methods for deterring the weightages of blueprint contents were discussed. Although no statistical difference exists, the importance of the theme or topic is seemed to be superior to other methods and provided a flexible blueprint. The disadvantage of using the specific learning outcomes and contact hours is their dependence upon calculation rather than the clinical practice. Specific learning outcomes and contact hours’ methods can be applied in pre-clinical phases and traditional curricula.

Study Strength

The study addresses an area that will enhance teaching and curriculum designs. The study describes three methods that can be used to develop an examination blueprint.

Study Weakness

The study was conducted in one discipline.

Recommendations

Conduction of a study on different disciplines, including basic and clinical courses to determine which method is the reasonable way for weightage determination for their blueprint, also compares the methods of blueprinting to student's results and satisfaction.

References

1. Coderre S, Woloschuk W, Mclaughlin K. Twelve tips for blueprinting. Med Teach. 2009;31:322-324.
2. Bordage G, Brailovsky C, Carretier H, Page G. Content validation of key features on a national examination of clinical decision-making skills. Acad Med. 1995;70:276-281.
3. Adkoli BV, Deepak KK. 2012. Blueprinting in assessment. Principles Asses Med Educ. 2012.
4. Patel T, Saurabh MK, Patel P. Perceptions of the Use of Blueprinting in a Formative Theory Assessment in Pharmacology Education. Sultan Qaboos Univ Med J. 2016;16:e475-e481.
5. Gujarathi AP, Dhakne Palwe S, Patil RN, Borade-Gedam P, Mahale MS, et al. Preparation of Blue Prints for formative Theory Assessment of Undergraduate Medical Students in Community Medicine. MVP J Med Sci. 2020;2:100-103.
6. Yudkowsky R, Park YS, Downing SM. Assessment in health professions education. Routledge. 2019.
7. Abdellatif H, AL-Shahrani AM. Effect of blueprinting methods on test difficulty, discrimination, and reliability indices: cross-sectional study in an integrated learning program. Adv Med Educ Pract. 2019;10:23-30.
8. Siddiqui I, Ware J. Test blueprinting for multiple choice questions exams. J Health Specialties. 2014;2:123-125.
9. Patil SY, Hashilkar NK, Hungund BR. Blueprinting in assessment: How much is imprinted in our practice? J Educ Res Med Teach. 2014;2:4-6.
10. Ware J, Mardi AE, Abdulghani H, Siddiqui I. Objective Structured Clinical Examination, Saudi Commission for Health Specialties. J Health Specialties. 2014;2:1-44.
11. Tombleson P, Fox RA, Dacre JA. 2000. Defining the content for the objective structured clinical examination component of the professional and linguistic assessments board examination: development of a blueprint. Med Edu. 2000:34:566-572.
12. Patil SY, Gosavi M, Bannur HB, Ratnakar A. Blueprinting in assessment: A tool to increase the validity of undergraduate written examinations in pathology. Int J Appl Basic Med Res. 2015;5:S76-S79.
13. Goyal SK, Kumar N, Badyal D, Kainth A, Singh T. Preparation of Blueprint for Clinical Assessment of Undergraduate Medical Students in Psychiatry. J Res Med Edu Ethics. 2019;7:205-208.
14. Hamdy H. Blueprinting for the assessment of health care professionals. Clin Teach. 2006;3:175-179.
15. Chandratilake M, Davis M, Ponnamperuma G. Evaluating and designing assessments for medical education: the utility formula. Internet J Med Edu. 2020;1:1-7.