The Effectiveness of the Flipped Classroom in a 300 Level Terrorism Class

This study compares a 300 level terrorism class taught using a flipped classroom to the same class taught one year earlier in the traditional, face to face method. While one examination score was better for the face to face class, the flipped class did better than the traditional class on the final examination. Every other point of comparison showed no difference between the classes, except for the end of semester student evaluations, which were significantly higher for the face to face class than the flipped classroom.


Introduction
The flipped classroom, or inverted classroom , was discussed in pedagogical research as early as the year 2000. However, due to the increased use of technology in the classroom, as well as video production tools, the use of the flipped classroom has increased in recent years (Cheng et al, 2019), and has been described by some as "one of the new generation teaching models" (Bicen & Taspolat, 2019, p. 115). The flipped classroom changes the traditional approach to teaching. In a flipped classroom, students learn the course content by watching instructional videos (presumably made by the course instructor) outside of the classroom, which frees up the class period for interactive activities. This structure allows students to learn the course material at their own pace (Bergmann & Sams, 2012). According to Bishop and Verleger, a flipped classroom is "a new pedagogical method, which employs asynchronous video lectures and practice problems as homework, and active, group based problem-solving activities in the classroom" (2013, p. 2). They add that out of class video lectures are not a necessary component of a flipped classroom, and a flipped classroom can include "assigning reading outside of class and having discussions in class" (2013, p.5). In a flipped classroom the primary delivery of content is online, which is not the same as students doing homework problems online, which differentiates the flipped classroom from blended learning (Lai & Hwang, 2016;Staker & Horn, 2012). Therefore, the flipped classroom and blended learning "are not always interchangeable and should not be equated" (Cheng et al., 2019, p. 796).
The flipped classroom gained prominence as schools (K-12 as well as colleges and universities) shifted to student-centered learning from teacher-centered learning (Davies et al., 2013;Missildine et al., 2013). Teachers moved "the knowledge delivery outside of formal class time," leaving the class time for students to, "actively and extensively interact with peers and teachers" (Cheng et al., 2019, p. 796). The idea is that rather than sit passively and listen to a lecture, class time could be used for "whole class brainstorming, group based hands on assignments and peer review, feedback exchange, and remedial help" (Kong, 2014, p. 161).The notion of a flipped classroom is that students can listen to lectures on their own time and at their own pace (Bicen & Taspolat, 2019;Kugler et al., 2019), leaving the class period for activities like large and small group discussions and personalized learning (Francl, 2014), changing the role of the instructor from the sage on the stage to the guide on the side (Bicen & Taspolat, 2019;Cheng et al, 2019).
Of course, this raises some questions. One, what is necessarily passive about listening to a lecture? Students have to go to class, pay attention, and take thorough notes. Second, is a flipped classroom really conducted in a self-paced way? After all, there are still deadlines for tests and quizzes. Also, there is the presumption in a flipped classroom that "prior to class foundational knowledge is absorbed by the student," (Kugler et al., 2019(Kugler et al., , p. 1760, which means there is a deadline for watching the instructional videos created by the course instructor. How is this any different than a teacher assigning textbook chapters to be read before the lecture on said material? Kugler and colleagues state that students completing assigned readings prior to class is an "unrealistic expectation" (2019, p. 1759), but what makes anyone think that students will watch instructional videos prior to class is a realistic expectation? While one stated advantage of the flipped classroom is that students can watch the videos more than once (Mok, 2014;Turan & Gotkas, 2015), one could also argue that students could read a textbook chapter repeatedly. Finally, and perhaps most important, is the flipped classroom a more effective way for students to learn than a traditional classroom?

Review of the Literature
Cheng and colleagues (2019) conducted a meta-analysis that examined studies that compared flipped classrooms and those that did not (what they call the traditional classroom). Their purpose was to examine the effect of the flipped classroom on student learning, and the variables that moderated those effects. They searched 17 databases and found 55 publications that met their criteria for inclusion, which were (1) the study had to be published in a peer-reviewed journal, presented at a conference, and a dissertation or thesis; (2) studies that compared flipped classrooms to traditional classrooms (not just on the implementation of a flipped classroom); (3) studies that provided adequate data to determine effect size; and (4) were published between 2000 and 2016 (Cheng et al., 2019).
Cheng and colleagues focused solely on studies that used cognitive student learning outcomes (e.g. exam scores) and not affective (e.g. student evaluations) or behavioral (e.g. retention rates). Their sample included students were taught in a traditional classroom (n=4185) and those who were taught in a flipped classroom (n=3727) for a total sample size of 7912. The greatest number of students (N=5990) were undergraduate college students, but their analysis included studies that examined graduate students (N=789) and K-12 students (N=1133).
What Cheng and colleagues found was that while the flipped classroom had a positive effect on student outcomes, "the effect size is trivial" (2019, p. 813). There were not significant differences between K-12 and college students, but the effect for graduate students was "inconclusive" (2019, p. 814), which is most likely due to the fact that most graduate programs do not rely on lecture and most of the work outside of class is spent reviewing scholarly works and not listening to or watching lectures. Of the disciplines studied only Engineering saw a negative (but insignificant) effect from the flipped classroom, when compared to the other disciplines examined which included Mathematics, Arts and Humanities, and Social Sciences which saw medium to large effect sizes (Cheng et al., 2019). Based on their findings, Cheng and colleagues concluded that flipped classrooms "may not have the same effect on cognitive student learning as other forms of blended learning" (2019, p. 815).
Anne Kugler and colleagues studied the impact of using a flipped classroom on Doctor of Pharmacy (PharmD) students (2019). In this study, an instructor presented half of their material in a flipped format and the second half in a traditional lecture format in the first four weeks of a course taught in a block format. The following year (2015), the same instructor switched the formats for the same topics. The flipped part of the class consisted of the students watching required videos on the subject matter, and they consisted of slides with an audio track of the instructor's voice. Students were released from class one hour early the day before the flipped topic to facilitate completion of the assignment (Kugler et al., 2019). Classroom time was devoted to discussion, application exercises, and quizzes. The rest of the material for the course was taught using the traditional classroom lecture. Student learning was assessed by comparing the final course scores for the 2014 class and the 2015 class, and no significant difference was found. The overall class average for the 2014 class was 84.5%, and the overall class average for the 2015 class was 83.6% (Kugler et al, 2019). However, when examining the four topics in this study separately, student performance on formal assessments was higher for the topics taught using the flipped classroom, and student performance was "significantly higher" for three of those four topics (Kugler et al, 2019(Kugler et al, , p. 1762).
Students were also given a pre-post course survey on the first day of class and one month after the final examination. The survey consisted of five agree/disagree questions that included "I welcome opportunities to use new technologies in my learning," and "I consider the use of a flipped classroom method to be beneficial over the traditional teaching method" (Kugler et al., 2019(Kugler et al., , p. 1763. Students in each class overwhelmingly disagreed with the first question (86.2% in 2014 and 93.8% 2015), and half of each class disagreed that the flipped classroom was more beneficial than the traditional classroom (50.0% and 51.9%, respectively). Therefore, while the flipped classroom demonstrated better assessment results, "students preferred live lectures over prerecorded lectures" (Kugler et al., 2019(Kugler et al., , p. 1759. These findings are similar to Koo and colleagues who found that while the flipped classroom improved student performance, students preferred the traditional classroom (2016).
Some of the research in this field has focused on student thoughts about the flipped classroom. Bicen and Taspolat conducted qualitative research to learn student opinions through interviews (2019). A computer science class, Programming Languages, was taught using the flipped method that utilized a Facebook group and a YouTube channel that was created specifically for the video lectures which were shared, along other course content, via the social media group (Bicen & Taspolat, 2019). Semi-structured interviews were conducted with students before the start of the class and again at the end of the course.
Initially, most of the students were apprehensive about the course, with 12 of the 20 students saying that they were surprised that the course would be taught in this manner (n=7) or that they expected the course to be challenging (n=5). Three of the students said that they were afraid to take a flipped class (Bicen & Taspolat, 2019). However, at the end of the class, half of the class (n=10) said that the class was better than they initially thought it would be, and three students said that this method could be used in their other classes. Two of the students described the class as useless (Bicen & Taspolat, 2019). While Bicen and Taspolat report that the flipped classroom increased student motivation, it is unknown of it also increased their learning.
Lanqin Zheng and colleagues (2020) conducted a meta-analysis to examine the effectiveness of the flipped classroom. They synthesized 95 studies with over 15,000 participants (N=15,386) that were published between 2013 and 2019 to determine the effect size of the flipped classroom, and also to determine the role of moderating variables. Some of these moderators included sample levels (primary school, junior and senior high school, and higher education), sample size (1-50, 51-100, 101-300, more than 300), and learning domains (Engineering and Technological Science, Medical Science, Natural Science, and Social Science). Unlike the meta-analysis by Cheng et al. (2019), Zheng et al. only looked at studies that were published in peer reviewed journals.
Zheng et al. found that the flipped classroom had an "overall positive effect on students' learning achievement and motivation" (2020, p. 11). This positive effect was found among all of the learning domains, which would indicate that the flipped classroom is appropriate in any class. A positive effect was not found in the studies on primary schools, but studies on flipped classrooms in primary schools only made up 3.16% (n=3) of their sample. In addition, no positive effect was found for samples of more than 300, which implies that the flipped classroom might only be appropriate for junior high school, high school, and college classes with fewer than 300 students.
The purpose of this study is to assess the effectiveness of the flipped classroom. A 300 level course, Examining Terrorism, was taught in the fall of 2019 using the traditional lecture format and again in the fall of 2020 using the flipped format. Everything about the content and course setup was identical, except that in the fall of 2020 the course content was delivered via video lectures that the students viewed outside of class, while class time was spent on activities and discussions.

Methods
Examining Terrorism, CJA 322, is a criminal justice elective at the college where the author is a professor of Criminology and Criminal Justice. The course is taught once a year, in the fall, on Mondays, Wednesdays and Fridays, for 50 minutes a class. Class assessments are conducted through four essay examinations, group presentations on a terrorist group that the students choose from a list provided by the instructor (although they can choose a different group with approval), team debates, and a multiple choice final examination. Classes always start with a multiple choice pretest to gauge student knowledge of the subject on the first day of class, and the final examination serves as the posttest.
In the fall of 2019 the course was delivered using the traditional lecture format which is how it had been taught since the first year it was offered (2010), and in the fall of 2020 it was taught using a flipped classroom. The course content, textbooks, examinations, assignments, and pre and posttests were exactly the same. Even the syllabus for each class is identical, except for a paragraph in the fall 2020 syllabus informing students that the course content would be delivered via video lectures that the instructor created using VidGrid and placed on the college's Learning Management System (Canvas) that the students would watch outside of class. The videos consisted of lecture slides accompanied by the voice of the instructor.
The points of comparison for the two classes are cognitive and affective. To assess any differences in learning the course content independent samples t-tests were conducted for each essay examination, the final examination (posttest), and overall course grade. Affective assessment was conducted by comparing the end of semester course evaluations for each class, also via an independent samples t-test.

Participants
The sample for this study consisted of 57 students. In 2019, 29 students took Examining Terrorism and in 2020 there were 28 students. The 2019 class was predominately male (62.1%), criminal justice majors (89.7%), who were seniors (55.2%). The 2020 class was majority male (60.7%), with fewer criminal justice majors (78.6%), and mostly made up of juniors (64.3%). The demographic variables for the sample can be found in Table 1. An independent samples t-test was conducted to compare the pretest scores on the content of the class for each class and no significant difference was found (t(55)=.337, p>.05). The mean pretest score (out of 25 points) for the 2019 class (m=7.97, sd=3.00) was not significantly different than the mean pretest score for the 2020 class (m=7.68, sd=3.41). Therefore any differences in the cognitive aspects of the classes cannot be attributed to a difference in prior course knowledge.

Results
First, the examination scores for each class were compared. The four examinations are worth 50 points each and are made up of six (examination 1) and seven (examinations 2-4) essay questions where the students have to answer five questions that they choose. Examinations in 2020 were identical to the examinations in 2019 and each covered the exact same material. Independent samples t-tests were conducted for each examination. A significant difference was found for the first examination (t(55)=3.84, p<.05), and the 2019 class (m=44.41, sd=7.38) outperformed the 2020 class (m=32.57, sd=14.84) by over 10 points. There were no significant differences in the scores for examination 2 (t(55)=-.875, p>.05) and the scores for 2019 (m=40.59, sd=11.20) were slightly less than the 2020 class (m=42.75, sd=6.87). The scores for examination 3 were also not significantly different (t(55)=.442, p>.05) and were almost identical in 2019 (m=45.86, sd=6.92) and 2020 (m=45.00, sd=7.79), which is also the case for examination 4 (t(55)=.005, p>.05), where the mean score in 2019 was 42.66 (sd=8.49), and the mean score in 2020 was 42.65 (sd=8.82).
To further explore any cognitive differences between the face to face (2019) and flipped (2020) classes, an independent samples t-test was conducted to compare differences between the posttest measure for each class. Here a significant difference was found (t(55)=-6.06, p<.05), with the flipped classroom (m=22.61, sd=2.99) outperforming the face to face class (m=17.93, sd=2.84) on the 25 point multiple choice final examination.
Two more independent samples t-tests were conducted. One was on the overall course grade, to evaluate performance in the class and not just examinations. The overall class grade point average 1 for the face to face class (m=3.28, sd=.76), was slightly higher than the flipped classroom (m=3.11, sd=.53), but the difference was not statistically significant (t(55)=.965, p>.05). Finally, to assess any affective differences in the classes an independent samples t-test was conducted using the student end of course evaluations, and this is where the biggest difference between the classes was found. The student evaluations 2 for the face to face class (m=4.79, sd=.42) were much higher than the student evaluations for the flipped classroom (m=3.11, sd=1.17) (t(26)=5.64, p.05). The most frequent comment from the evaluations in the flipped classroom was that students felt like they were teaching themselves the material. However, these findings should be viewed with caution. Fewer students (n=28) filled out end of the course evaluations than were in both classes (n=57), since completing student evaluations at this college is voluntary.

Discussion
The goal of the present study is to examine the effectiveness of the flipped classroom. To do this the author compared a class that was taught using the traditional face to face format in the fall of 2019 to the same class taught in the fall of 2020 using a flipped classroom. The first point of comparison was cognitive, where the grades for four examinations, the grades for the final examination (posttest), and the overall grades earned in each class were compared. A significant difference was found in only two examinations, the first test and the final. While the face to face class outperformed the flipped classroom on the first exam (by almost 12 points on a 50 point examination), the flipped classroom outperformed the face to face classroom on the final examination (by almost five points on a 25 point test). All other points of comparison for cognitive differences did not demonstrate any significant results.
The other point of comparison was the affective domain of the class, and to assess this component the end of semester student evaluations of the course were compared. In this case it is clear that the students in the face to face enjoyed their experience much more than the students in the flipped classroom. Then what can we conclude about the effectiveness of a flipped classroom compared to a traditional, face to face class? If we are keeping score it would be a draw cognitively, but affectively students in this study preferred the face to face teaching method. These findings are consistent with previous research conducted on flipped classroom that showed no effect (Cheng et al., 2019), or that students prefer live lectures over a flipped course (Koo et al. 2016;Kugler et al., 2019).
The flipped classroom was held during the Covid-19 Pandemic, but the plan to conduct this particular class in a flipped format predates the pandemic. Nonetheless, one advantage to having a flipped classroom is that it limited student exposure to each other and the professor. It would seem that the longer the pandemic lasts higher education will be challenged to find a way to keep students on campus while maintaining social distance and reducing the chances of transmission of the disease. However, if the flipped classroom does not demonstrate benefits over a face to face class, then other avenues of instruction should be explored. Verrecchia and McGlinchey (2021), compared two statistics classes, one face to face and one taught using the HyFlex model, and found no cognitive difference between the two modes of delivery. It seems like the challenge for higher education going forward is to find a way to allow for flexibility in delivering content while also not losing the personal touch that makes education worthwhile.

Limitations
This is a quasi-experimental study with methodological limitations. This study was conducted at one college in South-Central Pennsylvania, so the findings cannot be generalized to a larger population. In addition, it was conducted with two classes and therefore the sample size is rather small (n=57). Another limitation was that this study did not examine a behavioral component of class (see Cheng et al, 2019), simply because only one student dropped the 2019 class, and only one student dropped the 2020 class, leaving no basis for comparison.
This study was conducted using the same exact class at two different points in time (the fall of 2019 and the fall of 2020). While content and assignment wise this is true, the makeup of the classes was different, specifically, the fall 2019 face to face class was made up of more seniors than freshmen, sophomores or juniors than the flipped classroom (X 2 =8.49, p<.05), which leaves unanswered the question, would these results have been the same if the year in school for each class were the same?

Conclusions and Future Research
Future research should include a qualitative component when assessing the effectiveness of a flipped classroom (see Turan & Gotkas, 2015). Anecdotally, the author of this study noticed that three students in the flipped classroom did much better on the second examination than the first (one student went from a 3 out of 50 on the first test to a 49 out of 50 on the second). When these students were asked (separately) why they think they improved so much, they all said a variation of the same thing: they did not read the book or watch any videos until the day before the exam. In other words, they did not do the necessary work. It is too bad, but not unexpected, that it was a poor performance on the first test that motivated them to do the work outside of class that is required in all classes, but is especially important in a flipped classroom when the onus for learning the course content is placed squarely on the students. Future research would be enhanced with surveys or focus group interviews of students, specifically designed to get feedback on the flipped classroom.
Another area of future research should include how college professors feel about teaching a flipped class. Most of the research in this area focuses on student outcomes, and this is understandable. However, due to the restrictions of Covid-19 a number of colleges and universities have had to adjust their teaching modalities to accommodate social distancing requirements while keeping students on campus. How well professors can adjust to new ways of teaching should be a consideration.