Review of A Qualitative Investigation of Student Perceptions of Game Elements in a Gamified Course
Below is the result of the final assignment in ETEC 500: Research Methodology in Education, a Master’s level course in UBC’s MET program. The task was as follows:
This assignment asks you to write a research analysis and critique of a published research study of your choice, to give feedback to two of your classmates on their draft critiques and to make use of classmates’ feedback to refine your own final draft.
Review your selected research study using the most relevant framework and guidelines offered in this Module as a guide to write a thoughtful research analysis and critique. Your review should be 1,500-2,000 words in length. (Note that while these frameworks are useful guides, you need not be too rigid in following them if selected items are not relevant to your chosen study).
Review of A Qualitative Investigation of Student Perceptions of Game Elements in a Gamified Course
The aim of A Qualitative Investigation of Student Perceptions of Game Elements in a Gamified Course was to evaluate student perceptions of a number of game elements in a gamified teacher education course (p. 235). The focus was on how students reacted to specific game elements as opposed to gamification as a whole.
Literature
Prior to this study, limited work had been done on specific game elements in an education context and even then, research was mostly done following a quantitative method which limits gaining insight from the participants about their gamified experiences. Previous research had shown that gamification had a positive influence on motivation, participation, achievement, learning, attendance, satisfaction, and enjoyment (p. 236). The researchers also pointed out that some argue that what is described as gamification is simply “pointsification” since points and badges are the primary vehicles that many use to gamify their courses. By only focusing on points, some argue that the actual value of games is lost (p. 236). Some literature also speaks to the negative impact that competition and leaderboards can have on learning when the learner does not see themselves present or at the bottom of the standings (p. 237). Previous literature has not tended to focus on the effectiveness of game elements themselves and rather has chosen to test gamification against non-gamification.
As we continue to enhance the delivery of educational content in gamified contexts, studies such as this provide needed insights into how specific game elements are perceived by students. The feedback gained can guide instructional design as courses are built and developed with gamification in mind. This study also specifically spoke to the iterative nature of gamification by modifying the game elements while in the middle of the study to improve the effectiveness of the experience (p. 237).
Study Design
Using a phenomenological design method, researchers observed, interviewed, and analyzed students participating in a gamified learning environment. The design of the gamification was drawn from the work of Werbach and Hunter in For the Win: The Power of Gamification and Game Thinking in Business, Education, Government, and Social Impact (Werbach & Hunter, 2012) and contains six key elements:
Define business objectives (in this case, ‘define learning objectives’)
Delineate target behavior
Describe your players
Devise activity loops
Don’t forget the fun
Deploy appropriate tools
The nine game elements studied were:
Challenge
Narrative
Leaderboards
Reward
Badge
Teams
Win-state
Points
Constraint
It is clear that the researchers believe that gamification has beneficial effects on learning. The gap existing in the literature was in understanding which elements had the greatest effect and to what extent that effect was (p. 241). Gamification is not simply adding points to learning or delivering badges for behavior (though those are gamified elements) there are many other game elements to be used in educational contexts. By gaining a deeper understanding of student perception of game methods and their use in the classroom, the researchers believe a gamification framework can be established to guide future instructional design practices.
Three methods of data collection were utilized: observation, interviews, and documents. The documents reviewed included examples of student work, email logs, and comments in an online community (p. 242). Observation and document study was conducted on the entire group of students. Interviews were conducted with a sample of the group. Observation was critical to the study as it added evaluator context to responses to questions asked in the interviews. In one such case, a student commented that being high on a leaderboard caused her team to relax. It was observed that this student’s team was in fact in first place at one point in the study and subsequently fell to finish eventually as the last-place team (p. 245).
Sampling
Participants were collected using both Convenience Sampling and Purposeful Sampling methods. The case and educator were selected through convenience sampling. Convenience sampling was critical to the success of the study as they needed a partner educator who was comfortable with educational games and willing to experiment in her own class with new techniques. Purposeful sampling was used to select interview participants as they wanted to ensure their interview subjects had thoughts and comments they were in fact willing to share (p. 242). There was no control sample or non-gamified group since this study was not seeking to compare gamification against non-gamification but to assess perceptions of gamification techniques.
Participants in the study, 118 in total, are described as being from across the post-secondary academic spectrum. Because the selection of the case was conducted through convenience sampling, and the interview participants selected through purposeful sampling, its possible that researchers were getting a positively biased sample group containing an educator comfortable in the teaching environment, and students who are interested in talking about their experiences in the learning environment. The study was conducted on two groups, leaving more room for further, long-term study in order to achieve saturation.
Data Collection
While this study’s purpose was not to tie perceptions of game elements to outcomes, there was an opportunity here to determine if outcomes had any impact on student perception. This study does not refer to learning results from tests or assignments at any point in the paper. While it would have dramatically increased the workload and scope of the study, we would have learned a great deal about how the perception of various game elements is tied to outcomes. Is it possible to have a negative academic outcome but a positive learning experience? Is the inverse possible as well? Despite this missed opportunity, the study was successful at meeting its goal to gather perceptions on game elements.
Because this study was qualitative in nature and its data was not reliant on numerical outcomes, the researchers did take advantage of mid-study interviews and data collection from students to conduct their own formative assessment on the study design, the nature of the game elements, and how they were being implemented. Changes were made between the first group of subjects and the second in order to enhance the feedback obtained.
Data Analysis
The coded feedback provided meaningful results by which you could gather perspectives about game elements in gamified environments and mainly reflected what was already perceived in previous literature, that some game elements are more popular than others. Leaderboards and reward systems were shown to be incredibly popular with a majority of the students. 71.4% of participants liked having leaderboards, while 90.5% of students liked having a reward system of some sort (pp. 244-245). Game elements such as wrapping a fictional narrative around learning were not as popular, 54.8% of participants disliked narrative elements in their learning (p. 244).
Both the researchers and chosen experts coded the feedback obtained by students. The notes from both sets of coding were compared with a final form of codes accepted by both parties (p. 242). Student feedback was coded into categories for each game element depending on the source of the feedback: observation in class, observations online, interview or email. The number of times a piece of feedback was tied to a category was recorded. The categories for coding the results were different for each game element and the same number of codes was not used across the game elements. The Challenge game element had 13 categories for coding feedback while the narrative game element only had three categories for coding feedback (pp. 244-245).
During the study, researchers had the opportunity to revise their approach based on the feedback of participants. This formative approach to their study design meant that the framework being used was continuously being altered and improved. An example of continuous improvement came in the form of the development of a master document to explain terms being used and how points were applied and shared. The study provided a number of useful insights on game elements to use or not use in a gamified classroom. The feedback gathered can inform the structure of future gamified classrooms to increase the effectiveness of the elements used.
Auditability
The credibility of data was obtained through the use of constant observation and review of email or online notes in concert with interviews. There was a prolonged engagement not only with the concepts but with the subjects under study as well. By purposefully selecting participants, the researchers were able to gather data to support their questions about the effectiveness of game elements in gamified learning environments. This data produced results that are clearly transferable to other studies and contexts in gamification research and application. The study framework was passed through a number of reviewers in order to appear dependable (p. 242). I feel confident that this study can be replicated by other researchers who will, with a high degree of certainty, reach similar results.
Conclusions
The results of the study made conclusive the opinion that there are a number of specific game elements that students enjoy working in. It also made clear that some elements of gamification require nuance and finesse in order to implement them successfully. The discussion portion of this study aims to guide future instructional desires down a path of success in gamified learning. The authors warn of potential pitfalls and remind educators that a gamified experience is not one that can simply come out of a box and be effective in every scenario. This study reinforces the notion that not all game elements are equally engaging and all have unique challenges in implementation and delivery. It also drives home the notion that gamified elements have to be handled with care and throttled in their use depending on the environment and audience.
In the future, studying the same gamified elements from the perspective of the educator would be a strong next step. we must understand how educators, experienced or not, can apply gamified elements with success in their classroom. This expanded scope should be part of a longer study including gathering more student perceptions of gamified elements. As explained before, another unexplored area is the impact of the game elements on outcomes. This topic was not addressed at all in the study and remains an area deserving of future analysis. Additionally, the effectiveness of gamification should be studied through the lens of students’ past experiences. Do students who identify themselves as “gamers” experience better outcomes than students who do not? Finally, there is incredible potential in the realm of gamified learning management systems. Currently, most common LMS applications have badging and points systems but few have depth in terms of offering a more holistic gamified learning experience. By moving this type of study to an online platform, new research could explore the efficacy of game elements in alternative formats.
References
Aldemir, T., Celik, B., & Kaplan, G. (2018). A qualitative investigation of student perceptions of game elements in a gamified course. Computers in Human Behavior, 78, 235-254. 10.1016/j.chb.2017.10.001
Werbach, K., & Hunter, D. (2012). For the Win: The Power of Gamification and Game Thinking in Business, Education, Government, and Social Impact. Wharton School Press.