Do the Guideline Violations Influence Test Difficulty of High-stake Test?: An Investigation on University Entrance Examination in Turkey

Abstract

Multiple-choice (MC) items are commonly used in high-stake tests. Thus, each item of such tests should be meticulously constructed to increase the accuracy of decisions based on test results. Haladyna and his colleagues (2002) addressed the valid item-writing guidelines to construct high quality MC items in order to increase test reliability and validity. However, violating these guidelines is very common in high-stake tests. This study addressed two of these guidelines:, “AVOID the complex MC (Type K) format” and “Word the stem positively, avoid negatives such as NOT or EXCEPT”, respectively. After reviewing a total of 2336 MC items extracted from university entrance examination (UEE) in Turkey administered over the past 15 years, we investigated impact of the violations of item-writing guidelines on test difficulty using multiple regression analysis. The findings showed that test difficulty was not statistically changed when MC items with negative stem were used in a test. They, however, indicated that the use of complex MC items has a statistically negative influence on the test difficulty. The paper concludes with possible results of the cases whereby items constructed by violating item-writing gudielines are eliminated from the test, and directions for future studies.