Topic: Designing a New Curriculum. Sub-topic: Assessment (Progress Test)

Article: Wrigley, W., Van Der Vleuten, C. P., Freeman, A., & Muijtjens, A. (2012). A systemic framework for the progress test: strengths, constraints and issues: AMEE Guide No. 71. Medical teacher, 34(9), 683-697.

For this week I chose an article about one of the assessment tools that I want to include in the curriculum I am designing: the progress test (a longitudinal, multiple choice question assessment tool). This is an assessment instrument conceived and applied at the level of the whole program (not at a course’s level). It is thus a good fit for a problem-based-learning approach. I deliberately looked for an article pointing to weaknesses of progress testing (and hopefully ideas to overcome limitations), so the attached article seemed like a good choice.  

The article offers an “empirically-based, systemic framework for progress test practices and processes” (p. 684). The authors show that after 30 years of its inception, progress tests are used by higher education institutions in all continents. However, they are used mainly medical schools. More recently, progress tests have been implemented in other health-related educational programs. The authors stressed the positive role of institutional consortia that allow economies of scale for the creation and administration of the tests.

The framework for progress test includes four components: (i) Test Construction, (ii) Test Administration, (iii) Results Analysis and Review, and (iv) Feedback to Stakeholders. The authors elaborate on each component, including phases and products, required resources, mechanisms of quality control, and actors involved.

Test Construction starts with a blueprint and follows with the item authoring phase. It demands a review committee and appointed coordinators.

Test Administration includes decisions about the test purpose, frequency, duration, synchronicity, and delivery method.

The stage of Results Analysis and Review includes the score calculation methods, reviews by national/international committees, item evaluation by students, and decisions about the standard-setting method.

Finally, the Feedback to Stakeholders stage is conceived as quality control for teaching and learning, providing information relevant to students, item authors, faculty members, the school/institution, and local or national/international overview committees.

Take aways for My Big Idea

The framework is extremely useful for designing one of the assessment instruments in my curriculum.  I can use ideas from every component of the framework. Particularly, the fourth component (feedback) sparked ideas that I will include in my design, combined with the use of “degrees of certainty” given by the students for every answer (for example, I will provide automatized feedback about items in which the student gave a “dangerous” answer (wrong and highly confident) or a low-confidence answer (right but unsure). In order to lower the burden of item creation, perhaps the feedback won’t include the actual item text, but rather a description of the involved content(s) and mental process(es) -according to revised Bloom’s taxonomy , for example. Contents and mental processes will be established in the blueprint of the test, in the Test Construction stage.   

Some of the issues or limitations that the authors mention are related to measurement decisions that impact the test scores (particularly the penalty for guessing and the introduction of an “I don’t know” option). By including the “degrees of certainty” (Leclercq), I will address these issues. Instead of penalizing students for guessing, or to giving them the omission option (“I don’t know”), you ask them to “guess”; i.e., to chose one alternative and indicate “5% sure”, as a way to detect partial knowledge. Other weaknesses are related to available resources; for this project I will assume I have enough budget.

I will recommend that the school imparting the curriculum I’m proposing establish all possible alliances with other schools using progress test, hopefully in several fields and disciplines, in order to benefit from economies of scale.

Article: Wrigley_et_al_(2012)

One thought on “Topic: Designing a New Curriculum. Sub-topic: Assessment (Progress Test)”

Leave a Reply