Application of different standard settings in an innovative curriculum of College of Medicine, University of Bisha, Saudi Arabia

Received: 01-Jun-2021 Accepted: 30-Jul-2021 Published: 06-Aug-2021

Introduction

In view of the importance of assessment, which is considered as the corner stone of learning, it must be determined according to sounded, scientific, credible reliable and valid method. The decision of pass and failure is an important issue in education in general and medical education at large [1]. Many authors share many definitions for standard settings such as a clear cut distinction between failure/pass [2-4], a demarcation between acceptable and non-acceptable knowledge and skills [5], what is the optimum performance needed [2] a minimum acceptable level to make student to pass the exam [6] and finally determination of cut of score [7]. Many ways are exciting for determination of standard setting, however the most famous ones are the criterion (i.e. absolute) or the norm reverenced ( i.e. relative), where the process functioned regardless of the exam result in the absolute in contrast to the relative where exam result is needed [2-5,8-10]. If the objective of the test to make students ranking, then Norm-referenced methods (relative) will do the job, where proper judgment on assessment of the performance will necessary needs criterion-referenced methods [4,8]. The main feature of criterion-referenced methods such as Angoff can allows most of the students to pass or fail the test, because it is interested on desirable level of competencies needed, while the Norm-referenced methods used when fixed numbers of students needed to pass the test [2,11]. In fact, no consensus upon which is the best method of standard setting, all of them have their own cons and prons, but should all avoid arbitrary judgment [5,9]. The most commonly used methods of standard setting in medical education are the Angoff methods in its original or modified form. Angoff is the process through which each subject expert can judge the minimally competent students who can answer each questions, then to make a sum of all the questions, then the average for all the subject expert is taken. The Angoff is a good practical example of the criterionreferenced methods, needs subject experts to estimate the minimally competent examinee to answer a question properly (original Angoff), by taking probability from 0-1, or to use the modified Angoff by taking one of 8 options (0.2, 0.4, 0.5, 0.6, 0.75, 0.90, 0.95, ‘do not know’) [3-4,10-13] The main aim of this study to explore the opportunity of application of original Angoff type of standard setting in integrated curriculum and to compare student performance in Angoff standard setting with Norm-reference and arbitrary methods. The research questions; is Angoff standard setting fair enough to judge on students' performance in integrated curriculum, than other methods?

Methods

Cross sectional prospective study conducted for six courses in different levels in College of Medicine University of Bisha, (CM/ UB). (CM/UB) founded four years ago, adopting innovative integrated curriculum depending on collaborative learning such as Problem based, Team based learning and seminar. Assessment through Multiple choice Questions, OSCE, OSPE, Structured short answer question. The assessment of the students in the college depends on arbitrary method requiring at least 60% for pass rate.

The standard setting had been also used beside the arbitrary method, to explore the opportunity of application of original Angoff in integrated curriculum, to achieve this raters gathered before the conduction of the exams for six courses. The raters had been reminded about the process of Angoff by experts, then they look at the exam MCQs and pass through it to judge on to which extent a minimally competent student can answer each question with possibility from zero to one, then taking the average. A second round of discussion among the raters was held to see the discrepancies between raters and again chances given to individual raters to modify their evaluation. Process for Angoff standard setting went smooth without any difficulties among the raters and no much discrepancy among their judgments. Meanwhile the fixed method was calculated by mean of the results of students in each course minus one standard deviation [5] and the arbitrary pass mark is that decided by many intuitions as percentage like 50%, 60% and etc. (CM/UB) considered 60% or more as pass mark. The p value was calculated.

Calculation of pass score

The data were analysed using SPSS version 23.0 by a statistician. The pass score was set at mean minus 1 SD.

Ethical considerations

Permission to conduct this study was granted by the Research and Ethics Committee, college of Medicine, University of Bisha (Ref No: UBCOM/H-06-BH-087 (04/18), Thursday, April 16^th, 2020.

Results

For course 1(62.7%), (88.4%) and (62.8%) of the students pass the exam according to Arbitrary equal or>60, Fixed (Normreference mean- 1 SD) and Angoff methods respectively with significant p value (p=0.010). For course 2(20%), (84.4%) and (28.9% of the students pass the exam according to Arbitrary equal or >60, Fixed (Norm-reference mean- 1 SD) and Angoff methods respectively with significant p value (p=0.00). For course 3(65.1%), (79.1%) and (79.1%) of the students pass the exam according to Arbitrary equal or >60, Fixed (Norm-reference mean- 1 SD) and Angoff methods respectively within significant p value (p=0.231).For course 4(55.1%), (81.6 %and (55.1%) of the students pass the exam according to Arbitrary equal or >60, Fixed (Norm-reference mean- 1 SD) and Angoff methods respectively with more or less significant p value (p=0.007). For course 5(70% (70%) and (62.8%) of the students pass the exam according to Arbitrary equal or >60, Fixed (Norm-reference mean- 1 SD) and Angoff methods respectively with insignificant p value (p=0.356).For course 6(56.2%), (85.4%) and (89.6%) of the students pass the exam according to Arbitrary equal or >60, Fixed (Norm-reference mean- 1 Sd) and Angoff methods respectively with significant p value (p=0.001). (Table 1, Figure 1). Only 54.85% of the students can pass the exams according to Arbitrary equal or>60, 76.63% of the students can pass the exam when Fixed (Norm-reference mean- 1 SD) is used, while 63.05 of the students can pass the exam when Angoff methods is applied, p value 0.02 value (Table 2).

Table 1: Percentages of students who pass the exam according to different standard settings

Figure 1: Percentage of students who can pass across different standard settings

Table 2: Comparison between performances on different standard settings

Discussion

Apparently three out of six courses showed significant differences between Angoff standard setting in comparison to arbitrary and fixed methods with p value 0.010, 0.00 and 0.001 for course 1,2 and course 6 respectively, while course 4 showed more or less significant difference between Angoff standard setting in comparison to arbitrary and fixed methods with p value 0.07. Course 3,5 has no significant association with Angoff standard setting in comparison to arbitrary and fixed methods with p value 0.231, 0.356 respectively.

In fact, accountability and transparency in decision regarding fail or pass should be defendable, logical, possibly be applicable, involvement of multiple experts with different background and available human and structure for implementation [14]. A desirable standard setting must have straightforward criterion to provide evidence based evidence for pass and fail on the test [14]. The significant difference between Angoff and fixed methods in our study is in harmony with other study nationally and internationally [5,15-17]. We have two courses (course 3,5) has no significance difference in their standard setting and the fix or arbitrary and this could be explained as follows: differences in student class levels, difference in the judges in an integrated course might be the reason. Despite the fact that standard setting is not applied regularly because it is one of the innovative procedures in medical education, however our tutors cope with it without any inconveniences. Many authorities like van der Vleuten and Swanson pointed out that basing test result should fallow credible standard setting like Angoff [18]. In fact since subject experts were committed in setting the standard setting through Angoff methods, made it more significant because they well aware of the level of their students.

Strength

The strength of this paper is that: it addresses standard setting in an innovative curriculum, where Angoff standard setting can be proceed not necessarily by subject experts; however our experience in this paper showed possibility of doing standard setting in integrated curriculum without all the rater to be subject expert. Some researchers considered it possible when no enough experts are available for performance of standards setting to do it through post graduate students [7].

Weakness

Still the raters' subjectivity, unawareness of the raters of students depth of knowledge to some extent might limit the proper utilization of standard setting using Angoff for instance, however setting criterion will move standard setting to more objectivity.

Conclusion

Faculty development program is needed to evaluate success of standard setting regularly. The current study showed that Angoff differed from other standard setting significantly, there for should be hilly recommended to justify graduation of safe graduates.

Acknowledge

We would like to acknowledge the great effort and help from our colleges (University of Bisha).

References

1. Ben-David MF. AMEE Guide No. 18: Standard setting in student assessment. Med Teach. 2000;22:120-130.
2. Abbas M, Salih K, Ishag  MEI,  Eleragi AMS, Ohaj M, et al. Comparison of different standard settings in introduction to medicine and medical education course at Bisha Medical College. Intern J Sci Eng Res. 2018; 9:1009-1019.
3. Norcini JJ. Setting standards on educational tests. Med Educ. 2003;37:464-469.
4. George S, Haque MS, Oyebode F. Standard setting: comparison of two methods. BMC Med Educ. 2006;6:46.
5. Elfaki OA, Salih KM. Comparison of Two Standard Setting Methods in a Medical Students MCQs Exam in Internal Medicine. Am J of Med Medical Sci. 2015; 5:164-167.
6. Cusimano MD. Standard setting in medical education. Acad Med. 1996;71:S112-120.
7. Mubuuke A, Mwesigwa C, Kiguli S. Implementing the Angoff method of standard setting using postgraduate students: Practical and affordable in resource-limited settings. Afr J Health Prof Educ. 2017; 9:171-175.
8. Norcini J, Guille R. Combining tests and setting standards. Interna Handbook Res Med Educ. 2002;811-834.
9. Hejri SM, Jalili M. Standard setting in medical education: fundamental concepts and emerging challenges. Med J Islam Repub Iran. 2014; 28:34.
10. Bandaranayake RC. Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37. Med Teach. 2008;30:836-845.
11. Cizek GJ, Bunch MB. Standard setting: A guide to establishing and evaluating performance standards on tests. Am Psychol Educ. 2007.
12. Angoff WH. Scales, norms, and equivalent scores: Educational Testing Service. 1984.
13. Gagnon R, Charlin B, Coletti M, Sauvé E, Van der Vleuten C. Assessment in the context of uncertainty: how many members are needed on the panel of reference of a script concordance test? Med Educ. 2005; 39:284-291.
14. Liu M, Liu K-M. Setting pass scores for clinical skills assessment. Kaohsiung J Med Sci. 2008;24:656-663.
15. Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med. 2006;18:50-57.
16. Humphrey-Murto S, MacFadyen JC. Standard setting: a comparison of case�author and modified borderline�group methods in a small�scale OSCE. Acad Med. 2002; 77:729-732.
17. Impara JC, Plake BS. Standard setting: an alternative approach. J Educ Meas. 1997; 34:353-366.
18. Kaufman DM, Mann KV, Muijtjens AM, Van der Vleuten CP: A comparison of standard�setting procedures for an OSCE in undergraduate medical education. Acad Med. 2000;75:267-271.