Identifying the Relationships Between Response Behaviors for Reading Comprehension Items and Student Characteristics

The study aimed to identify the students’ characteristics (perception of self-efficacy, resistance, persistence, taking responsibility and competence for their cognitive processes) that might affect the response behaviors of fifth grade students for open ended questions written in different cognitive levels. This study was designed as relational survey model research. Sample consisted of 392 5th grade students selected by purposeful sampling from Yenimahalle, Mamak and Cankaya districts in Ankara city. Data was collected with reading comprehension achievement test, response behaviors form and student information survey. Analysis was conducted with 366 students’ data after missing values were removed. MPlus package program was utilized to analyze data and structural equation modelling was used. The findings of the research revealed that response behaviors for open ended questions written in different cognitive levels in Turkish learning area were found to be predicted low and at a significant level by students’ perception of self-efficacy, persistence, resistance, taking responsibility and cognitive skills. It might be concluded that improving children’s perception for cognitive and metacognitive skills can enhance their thinking process, lead them to perform complex response behaviors and affect positively their higher order thinking process. In other words, students with high cognitive and metacognitive skills show higher level response behaviors such as putting themselves into author’s shoes, relating their own life to the event in the text or using imagination. Therefore, they can comprehend what they read easily and express it back.


Introduction
Reading comprehension is one of the most fundamental steps in acquiring knowledge. Meneghetti, Carretti, & De Beni (2006) define reading as a process to infer meaning and construct knowledge with the help of prior knowledge. Demirel (1999) states that reading is a process of inferring from written symbols in cooperation with cognitive processes and psychomotor skills. Reading comprehension is major skill needed for many aspects of life, including relating facts, evaluating social events, obtaining self-fulfillment, and engaging in education (Mete, 2012). Mullis, Martin, Kennedy, and Foy (2007) indicate that possessing the skills of reading comprehension helps individuals increase their success in daily life and provides them with intellectual information; moreover, people who read and comprehend well can contribute to a nations' socio-economic wealth. According to Kutlu, Yildirim, and Bilican (2011), listening comprehension and reading comprehension have a positive effect on student achievement not only in the Turkish language class but also in other subject areas. If these skills are not sufficiently developed, students will face difficulties in effective reading and listening.
There has been substantial research focusing on defining and developing theories connected to this skill since listening and reading comprehension has been proved to be important for cognitive activities. Theoretical and operational definitions have been made within various reading comprehension theories. The differences in the definitions cause changes to the data collection tools used over the years. Thus, changes in the item format and passage types used in measurement tools can be easily observed theories concerning measurement tools. Farr (1970) reported that items, such as multiple choice, true false, and fill in the gaps were the most preferred until 1960.
Later, educators started to use open-ended questions because close-ended questions provided limited information regarding reading comprehension skills. Furthermore, close-ended questions could often be answered as true without the student even reading the passage item that the question was intended to measure. This type of items is criticized for forcing the respondents to respond in the same way (Buros, 1965). Open-ended questions, which require written expression, have been widely used recently and they prompt students to construct their own responses. Open-ended reading comprehension items developed based on a passage require students to organize information and decide which knowledge should be used in what order and for what purpose to construct the true answer (Gronlund, 1977). Open-ended questions contribute to life skills by measuring the processes used in students' daily lives; for example, deciding on the related information and organizing it so that it can be used. This is how measurement connects to the students' daily lives and starts making sense both in and out of the school context for students.
The aim to equip students with the competences that the world requires reveals that school learning should be related to real life; thus, the curriculum should be constructed with this need in mind. The term authentic assessment was proposed to refer to educational activities designed for real life situations. Authentic assessment is centered on real life tasks that a person may face (Kutlu, Karakaya, & Dogan 2014;Newmann, King, & Carmicheal 2007). Authentic reading assessment, moreover, aims to measure people's reading comprehension skill by reading texts and items centered on daily life (Valencia 1998). This advance shifts the focus of reading comprehension measurement from product to process, and the importance of inference making and response construction becomes clear (Wixson, Peters, Weber, & Roeber 1987).
When the reading comprehension studies are reviewed, it is seen that researchers ignore the response construct process and focus on the responses. In such case, it can be advocated that reading comprehension is considered and evaluated as a product rather than a process. In fact, reading comprehension is a process involving investigation of the passage as well as skills of association and comparison (Kutlu, 1999;Ozdemir, 2007). These skills help individuals shape their prior knowledge and make inferences (Adams, 1990;Anderson, 1984;Small, & Arnone 2011). In other words, individuals make inferences between the passage and the daily life or past events during reading. The responses that students write are constructed as a result of such a process. Fletcher (2006) criticized studies that measure reading comprehension as they focus on the product and ignore the process of making inferences. He stated that constructing a response is an important process in the measurement reading comprehension; yet, it is often disregarded.
Questions such as which route a student follows to answer an item and which cognitive skills are engaged in this answering process can be investigated through the analysis of response behaviors. These behaviors can be defined as the process in which an answer is constructed in the mind or a mind map which is followed to answer an item. Response behavior covers the cognitive activities between seeing an item and giving an answering. According to the literature, researchers tend to pay attention to open-ended item writing but ignore the answering process of posing open-ended questions. Schwarz, and Oyserman (2001) emphasized that the first condition to answer an item correctly is to perform the appropriate response behavior to the construct that the item is intended to measure. The analysis of response behaviors provides information regarding both students' cognitive competences for open-ended questions and the power of an item to measure what it was developed for.
There is limited amount of research in the literature regarding response behaviors. Umay (1997) applied both multiple-choice and open-ended items to measure the same cognitive levels and determined response behaviors. According to the results, students scored higher in multiple-choice items. The ratio of wrong answer behaviors and non-response behaviors was equal in multiple-choice items while the non-response behaviors were more often observed in open-ended items than wrong answer behaviors. Another study conducted by Umay (1998) investigated the response behaviors in multiple-choice items to study the functionality of chance attenuation formula. The findings revealed that the students preferred to mark the choice they considered to be true. If they did not know the true answer, they took a guess instead of ruling out the choices and trying to mark the true answer. Kadioglu (2002) aimed to study the change in response behaviors for multiple-choice items with regards to item features and student characteristics. In the research, 30 multiple-choice items were given and the students were asked to mark the response behavior on a given form after answering each item. The research findings determined that as the items became more difficult, the students were keener to leave the items unanswered, answer the item by analyzing the options, eliminate the options and answer the item, or answer the question without performing at the expected cognitive level. No differences were observed in the response behaviors according to the item discrimination. As the total score from the test decreased, the frequency of leaving the item unanswered, eliminating or analyzing the options, and answering the question behaviors increased. Tokat (2006) investigated whether response behaviors for multiple-choice items differed in terms of sub-tests, gender, and conscious and unconscious answers. The results showed no difference in the response behaviors in terms of conscious and unconscious answers, and gender. Male and female students showed similar response behaviors. On the other hand, the students were found to read the root of the item first when there were tables or figures in the item. Later, they looked at the tables or figures and options, read the root again with options in mind, and marked the option. The response behaviors were reading the paragraph first and then root of the item for reading comprehension items, and reading the paragraph again, analyzing the options, and marking the answer for paragraph-based items. Furthermore, Tokat (2006) observed the common response behaviors for different courses as follows: 'reading the case and then the root, analyzing the options and marking the answer', 'reading the root and then the case, viewing the options and marking the answer', and 'reading the case and then the root, reading the case again, viewing the options and marking the answer' for the Turkish and social studies courses; ' reading the case and the root, investigating the figure or tables, viewing the options, reading the case and the root again with the options in mind and marking the answer' for the mathematics course; and 'investigating the table or figure, reading the case and the root, viewing the options, investigating the table or figures with the options in mind and marking the answer' for the science course.
Pehlivan Tunc, and Kutlu (2014) investigated whether eighth-grade students' response behaviors for multiple-choice items differed according to the sub-dimensions, psychometric features of an item, students' achievement level, and gender. Students were asked to answer multiple-choice items and complete the response behavior form. The research findings revealed that when item difficulty decreased and the item discriminant increased, the behavior 'I read the questions and options, I marked the option I consider to be true' was often observed. As total scores and item discrimination increased, the behavior 'marking the option by eliminating the options, marking the options with guess and leaving the question unanswered' decreased.
In the literature, there is limited research focusing on response behaviors. Moreover, no research has been found that investigates the response behaviors for reading comprehension with open-ended questions and related cognitive skills. Studying response behaviors is considered to be important to support students' academic development, understand students' reading comprehension skills, and provide a better interpretation for students' scores. Exploring the cognitive processes, a student experiences to answer an open-ended item may help educators develop more reliable and valid measurement tools. It is thought that determination of response behaviors for open-ended questions written at different cognitive levels and the effect of student characteristics for response behaviors may help experts in the field of measurement and evaluation and item writers to create more valid and reliable questions and allow teachers to evaluate students' scores more effectively. In this context, this study aimed to investigate the response behaviors of fifth-grade students in relation to open-ended questions written at different cognitive levels and the relation between response behaviors and student characteristics.

Participant (Subject) Characteristics
For the selection of the participants for this study, the aim was to ensure a variety of socioeconomic and sociocultural features. To select the students, purposeful sampling was used since the researcher aimed to obtain rich situations in terms of the determined variables (Buyukozturk, Kilic-Cakmak, Akgun, Karadeniz, & Demirel 2014;Fraenkel, & Wallen 2006). As there has been no study than can lead the researcher to determine a more specific sample group, a starting group was selected via pusposeful sampling which was easy to implement and wise to start with. The participants were fifth-grade students attending schools in the Cankaya, Yenimahalle and Mamak districts in Ankara province, Turkey in the 2016-1017 academic year during the fall semester. Information of the schools and classroom sizes are presented in Table 1.  Table 1 shows that the data collection process was completed with 392 students. Missing values (n=26) were removed from the data; thus, analyses were conducted with the data of 366 students.

Measures and Covariates
The data collection tools used in this study were a reading comprehension achievement test, response behaviors forms, and a student questionnaire. These tools are detailed in the following sub-sections.

Reading Comprehension Achievement Test
The reading comprehension achievement test used in this research was developed by Kutlu, and Aslanoglu (2008;. It consists of eight open-ended items based on a reading passage, which was selected on the basis that it covers an everyday topic. The items were developed based on the four levels in the reading skill taxonomy of the Progress in International Reading Literacy (PIRLS) (MEB, 2003;PIRLS, 2003). There were two items on stating the clearly expressed ideas and making direct deductions, two items on interpreting, two items on the use of knowledge and experiences, and two items on investigating the elements, content, and language of the text. An instruction was prepared that stressed the aim of the research and what was expected from the students. The students were asked to read the instruction prior to the application. Three experts from the field of measurement and evaluation in education, four Turkish language teachers, and two Turkish education academicians were asked to review the text, items, and the instructions since it had been a while since the test was first developed. The evaluators were asked to assess the appropriateness of the text, items, and the instruction for classroom level and vocabulary variety. The feedback provided by the evaluators was examined and applied. This feedback was also considered as a validity proof. The reliability of the test was estimated using Kendall's concordance coefficient by applying it to 70 students. The text, students' responses, scoring rubric, and instruction for raters were sent to three different raters, who were asked to grade the students' responses abiding by the scoring rubric and note the grades on a separate form. The concordance value was found to be statistically significant at the p<.01 level.

Response Behaviors Form
The response behaviors form developed by Simsek, and Kutlu (2017) was used to determine the thinking processes of the fifth-grade students when answering open-ended questions concerning Turkish language learning. There are 14 response behaviors under three themes, which are organized in a hierarchical structure from ordinary to complex. Each response behavior has the characteristics of the theme to which it belongs.
The first theme presents the responses constructed directly from the text. These response behaviors cover the basic cognitive processes. The second theme includes the responses constructed by interpreting the text. The third theme covers the response behaviors constructed by linking real life to the text or daily life. Response behaviors, such as students' putting themselves into author's shoes, relating their own life to the event in the text or using imagination are compiled under this theme. The response behaviors are structured from ordinary to complex; in other words, from lower-order to higher-order thinking processes. It was seen that the students did not perform active cognitive processes when faced with items for the lower order thinking process. Yet, it can be concluded that students use their cognitive abilities when they answer items measuring inference making. They shape their own cognitive process by organizing their prior knowledge and new information. In other words, they activate their higher order thinking skills and construct their responses.

Student Questionnaire
The student questionnaire was developed by the researchers to obtain data related to student characteristics. The questionnaires used in the Trends in International Mathematics and Science Study (TIMSS), PIRLS, and the Programme for International Student Assessment (PISA) were investigated, and various sources from the International Association for the Evaluation of Educational Achievement (IEA 2006;2011;2015), The Organization for Economic Co-operation and Development (OECD 2012), articles, and a thesis in the context of reading comprehension were also examined (Bal, 2010;Kumandas, & Kutlu 2009).
The questionnaire consists of 11 items to determine students' perception of self-efficacy, resistance, persistence, taking responsibility, and competence for their cognitive processes. Marzano and Heflebower (2012) pointed out that these aspects were effective in measuring people's self-understanding, self-control, and self-regulation. In other words, the questionnaire aimed to measure students' perception through their own cognitive and meta-cognitive processes. A draft of the questionnaire was sent to three experts in the measurement and evaluation field for validity concerns. The experts were asked to evaluate the items for what the items were formed to measure and were also asked to review the language of the items. The items used to form student questionnaire were written with items of international examinations in mind. Although the items are not directly from the questionnaires of other exams, the context was quite similar. That's why, three experts were considered enough for items originated from international student examinations. Their feedback was discussed and applied when necessary. The items in the questionnaire are presented in Table 2. When the items are examined it can be seen that the first four items were designed to measure students' meta-cognitive processes while the last seven items were meant to determine students' cognitive skills based on Marzano's cognitive and conative skills taxonomy.

Research Design
This study is designed as relational survey research, which aims to determine the existence or extent of a relation between at least two variables (Karasar, 2012). In keeping with this type of research, this study revealed the extent of relations between response behaviors and related factors.

Data Analysis
Within the scope of this study, the relation between students' response behaviors and student characteristics were investigated using structural equation modelling. The model that was tested is presented in Figure 1.

Figure 1. Model to be tested for response behaviors and student characteristic
The presented model investigates the relationship between students' perception of self-efficacy, resistance, persistence, taking responsibility, and competence for their cognitive processes and their response behaviors. The student characteristics were based on the Marzano's cognitive and meta-cognitive processes taxonomy, comprising self-efficacy, taking responsibility, persistence and resistance, and sub-dimensions of cognitive process, which are recalling information, interpreting/understanding information, using information in daily life, solving basic problems, solving complex problems, relating different information about events and deciding, and producing genuine (new, different, creative) works. These elements were the observable variables of the predictor latent variable, which consisted of the student characteristics.
The measurement models were tested first. The fit indices for the student characteristics model were found to be χ2=102.584; df=44; RMSEA=0.085, and CFI=0.978. In the analysis of the second measurement model, the first response behavior variable (b1) was not significantly related with the model, and thus was removed from the model. Then, the fit indices for this measurement model was calculated as χ2 =14.382; df=14; RMSEA=0.012, and CFI=0.998.

Response Behaviors
The fit indices for both measurement models were considered to show a medium level good fit (Browne, & Cudeck 1993). The structural model was tested on the data obtained and the results are given below.

Results
The study revealed the relationship between the response behaviors of fifth-grade students for the reading comprehension test and the student characteristics. The relationship between the students' perception of self-efficacy, resistance, persistence, taking responsibility, and competence for cognitive processes and response behaviors were examined by structural equation modelling. The structural model is presented in Figure 2.  Figure 2 shows the standardized coefficients for the model-data fit. All the standardized coefficients were significant. Table 3 provides the criteria suggested by Schumacher and Lomax (2016) and the values obtained from the tested model. The model fit indices of the structural pattern (χ 2 /df) showed a good fit between the model and the data. The Root Mean Square Error of Approximation (RMSEA) and the Comparative Fit Index (CFI) were found to be 0.039 and 0.986, respectively. These values indicated a perfect fit.
The predictor latent variable of student characteristics (stdnt), observed variables factor loadings (lambda χ), t values, error values (delta δ), and observed variables for response behaviors (respns), factor loadings (lambda y), t values and error values (epsilon ε) are presented in Table 4. As shown in Table 4, the factor loadings between student characteristics (stdnt) and self-efficacy (self), resistance (rest), persistence (pers), taking responsibility (resp), recalling information (rec), interpreting/understanding information (und), using information in daily life (use), solving basic problems (basic), solving complex problems (complex), relating different information about events and deciding (rlt), and producing genuine (new, different, creative) works (prod) were found to be positive and statistically significant. The most loaded variable from observed variables were recalling and interpreting/understanding information.
The relationship between the dependent latent variable (respns) and its observed variables, which were the response behaviors for eight open-ended items from the reading comprehension achievement test, can be observed in the model. Response behavior 1 (b1) was excluded from the model as it failed to provide a significant relationship with the dependent latent variable. The factor loading for other variables were found to be positive and statistically significant. The highest factor loading with the latent variable was observed in response behavior 3 (b3).
In this study, the relationship between student characteristics and response behaviors were also examined. The correlation coefficient between the predictor latent variable (stdnt) and the dependent latent variable (respns) was found to be low, positive, and statistically significant.

Discussion
Bloom (1995) stated that the recalling and understanding processes provided in his taxonomy are fundamental for further learning; that is, deficient learning in these steps may block or complicate the higher order level learning process. To attain a higher order level performance, students must have the necessary knowledge and be able to understand it. Hence, it is meaningful that the two most important factors related to students' response behaviors are their perception regarding the competence of recalling and understanding information. On the basis of the hierarchical structure of taxonomy in mind when basic learning in recalling and understanding information levels are not completed, then students are likely to fail to engage in more complex processes in terms of cognitive abilities. The variables examined in the context of this study include complex processes and require higher order performance; therefore, the development of these variables; i.e. skills yield complex response behaviors. Thus, in the current study, the students' self-perceptions of cognitive and metacognitive processes were found to be high and positive.
Response behavior 3 was observed to be the variable that had the highest loading and made the greatest contribution to the total score. This variable included those behaviors employing direct inference in answering the items. This high correlation may be due to having been frequently practiced by the students. Furthermore, as stated by Bloom (1995), recalling and understanding information stages of his taxonomy are the most vital for further learning processes.
Another finding of this study was that the relationship between student characteristics and response behaviors was determined as low, positive, and statistically significant. According to this finding, efforts to enhance students' self-perception of self-efficacy, resistance, persistence, taking responsibility, and competence for cognitive processes may produce an increase in the students' performance at the higher-order level and in complex behaviors. Carrell, Gadjusek, and Wise (1998) emphasized that reading comprehension is both a cognitive and metacognitive process. Hence, encouraging students to engage in metacognitive process can promote the development of students' reading and reading comprehension skills. Huang, Ricci, and Mnatsakanian (2016) remarked that if students acquire metacognitive skills, they are more likely to perform reasoning skill which is a higher order skill. Flavell (1976) concluded that higher order thinking skills are the most important factors explaining achievement in the problem solving. Paris, Wasik, and Turner (1991) stated that students with self-regulation skill are more active in pre and post reading processes. Tobias and Everson (1997) commented that students are unable to notice whether they possess knowledge if they are not skilled in metacognitive skills. Therefore, they cannot organize their learning and raise their performance to higher levels. Myers and Paris (1978) investigated the difference between poor and good readers. The research findings revealed a difference between groups in terms of reading metacognitive skills. Students with weak metacognitive skills were found to be insufficient in basic and complex processes regarding reading. Artelt, Schiefele, and Scheider (2001) examined the variables related to secondary school students' reading comprehension skills. Knowledge of metacognitive strategies was determined as a significant predictor of the level of reading comprehension. Another research supporting the literature was conducted by Neuenhaus, Lingel, Schneider, and Artelt (2011). Their findings advocated that the reading comprehension skill and metacognitive knowledge are related. In accordance with the findings of this study, more metacognitively aware students performing higher order response behaviors indicates that they are better at understanding and interpreting the text as well as transforming it and using it in their daily lives. Tregaskes and Daines (2010) examined whether the reading comprehension skill can be assisted by metacognitive strategies. This research revealed an increase in students' reading comprehension when they are educated to use metacognitive strategies.
It is concluded that in the current study, the relationship between students' perception of self-efficacy, resistance, persistence and taking responsibility, and competence for cognitive processes and response behaviors was poor, positive, and statistically significant. This finding implies that an increase in the perception of self-efficacy, resistance, persistence, taking responsibility, and competence for cognitive processes will lead to an increase in performing higher-order response behaviors.
Considering the findings from this research, to improve students' response behaviors towards open-ended questions and encourage them to perform higher-order behaviors, classroom-teaching activities should concentrate on students' personal and cognitive perceptions. Open ended questions are useful as they provide students with more opportunities to show their thinking and comprehension performances. Teachers should create environments for students to realize their own competences, to present their real performances and help them to improve these competences to perform higher order response behaviors. In terms of future research, it would be useful to conduct a study to investigate response behaviors and related factors at different grade levels.
This study was conducted with 366 students in four different state schools in Ankara, Turkey. Further research could include private schools and investigate the differences between schools. In the current study, only the student characteristics variable that might affect response behaviors were examined. Other variables that might be related to response behaviors towards open-ended questions should be investigated in future. The student questionnaire used in this study to collect data was developed considering the age group of students. Hence, it provided data only on a nominal scale level, which limited data analysis techniques and weakened the power of statistics. For this reason, it is suggested that further research is undertaken using equal interval scales.