Tuesday, December 23, 2014

Games as an Assessment Tool - Reviewing State of the Art and Identifying some of the Core Challenges for Using Games as Part of Assessment of Learning


Serious Games, IMT4102
                                                                   Høgskolen i Gjøvik

Faiza Iqbal, 130204


Abstract

There is now a common consent that games can be used as a good learning tool for various skills but now there is need to think and use these games differently. The learning of a student in particular subject can be measured by using these games as an assessment tool instead of instruction tool. This area of study is understudied because of the complexity involved in assessing skills especially 21st century skills. There is need to make some systematic approaches and guidelines to improve the design of serious games to achieve this goal of assessment. Like other educational tools, serious games can help to do performance assessment because it has an important features of showing the learning progress and its outcomes. An appropriate and regular feedback system in serious games makes it more appropriate tool for assessment especially when assessment is stealth assessment. In this paper the main focus is on: 1) to figure out the possible solutions to use games for assessment and 2) what are learning assessment types and how they can be used for games. With the help of related literature, two possible solutions have been presented and explained to use games for assessments with the help of examples. First one is to use existing serious games for assessment and the other one is to design new games. In this paper second solution is concluded as a better option in terms of assessment validity. It is also stated with the help of some game examples that which types of games can be used for assessment of knowledge and which can be used for skills and abilities. After discussing all assessment types (formative, summative, etc.) with examples, it have been concluded that it depends on purpose of assessment that which type is suitable for particular assessment process. There are some limitations and challenges which were figured out during the study are also presented in the end.

Keywords: assessment, game as an assessment tool, assessments in games, challenges of assessment in games,
Game as an Assessment Tool - Reviewing State of the Art and Identifying some of the Core Challenges for Using Games as Part of Assessment of Learning

To succeed in this complex world and to acquire twenty-first-century competencies, students need to think critically and creatively. The problem is not to make student think differently but also to increase their interest towards learning. Games specifically build and used to learn any kind of skill, are considered as serious games. These are more attractive for both students and teachers as compared to traditional boring teaching and learning methods. So now trend is changing, games are being used inside and outside of the classrooms to make learning process interesting and interactive. Games can also be used as an assessment tool at the end or during a study program to check the actual understanding of a student for a subject. Any tool like these could be a practical demonstration that a student has successfully completed a program or training. In the classroom with the help of traditional teaching methods, it is very difficult to teach student 21st century competences. These competencies include thinking, teamwork, creative problem solving, time management, and identity management, self and social awareness [1]. Serious games can be the best and easy way to teach these skills. Not only for learning purposes these games can be used to assess these skills because it would be hard for a human to assess these skills of many student at one time. When we are doing assessments then we have to assume several things, for example, the methods of teaching are actually helping in learning process, learning or skill attributes of player can be measured from this game and player is receiving feedback in some ways during the game.
In this paper the focus will be on how to use serious games with some supporting technologies as vehicle to asses a player’s cognitive skills and competencies, but not to asses any game itself. Digital games particularly provide an advantage in the assessment by keeping the record of every move and decision that a player makes during the play.

Methodology

To achieve the goal of this paper, a detailed literature review was required and this process started from reviewing the purpose of serious games and process of learning from serious games. Some games can be used for both learning and assessing purposes which means now we have to assume that games are helping in learning the skills which we want to assess. There was need to answer some questions to make some conclusions like learning happens with serious games or not, what does learning mean here, what is assessment, what are types of assessments, how these can be used for assessing the learning with games, what are traditional methods of learning assessments and how these methods can assist in developing new games. After this review, two possible solutions are proposed to use serious games for assessments. These solutions are discussed with examples and there was need to figure out which solution is better. With the help of some reasons, one solution is concluded as a better option in terms of assessment validity.
One of the main objective of this paper is to figure out the challenges for using the games for assessment. Considering all assessment types and methods that are discussed in literature review, some possible challenges have been proposed. These challenges require more in detail study or research.

Literature Review

Is learning possible from games?

There are number of studies have been done to answer this question (e.g. [2]), and many of them have concluded their results in the favor of games. In a research paper “A Literature Review of Gaming in Education”, they have made five claims regarding the learning in serious games. These claims include: digital games teach 21st century skills, provide more engagement for learners, these are built on learning principles, provide personalized learning opportunities and also provide authentic and relevant assessment [3]. They have also provided some evidences for these claims in their research paper.  A literature survey done to check the effect and effectiveness of serious games for educational purpose and they also concluded with positive results [4].

Assessment and its types

The main objective of assessments is to help students in their learning process. Sometimes students can ignore the teachings and waste their time. To avoid this situation assessment are done [5]. It is important to test whether the goals of education are being met or not. Assessments can help to make changes in curriculum, grades, advancement, instructional needs and etc. It can also give confidence to student to advance in the world which is continually changing. Now a days teachers are trying to make learning process interesting by using simulations and games. Similarly games can also be used to do assessment in an interactive and better way. Assessment through games can also help to assess those skills which are not easy to assess through traditional methods.

Stealth assessment

 Student’s abilities and competencies can be diagnosed efficiently and accurately with the help of stealth assessment in games. Stealth assessment system is quiet but very powerful way of gathering the student’s performance data and analyze it to infer the student’s level of competences for some skillsets. It is known as an assessment directly woven in the fabric of learning environment. It uses artificial algorithms to analyze the student performance as compared to predefined skill levels and give feedback according to results. A student unknowingly gets in an activity which is actually a goal oriented. In this way he lost his condition of self-consciousness and forget about the sense of time lost.
New technologies now can easily help us to embed assessments in this new learning method. It would be hard for human to do assessment in a way which these games can do because estimation of competency levels for various skillsets can be easily done by machine-based reasoning techniques. 
While discussing the learning assessments we have to know about important assessment types which can help to design the game according to its goal. Below are some assessment types which are commonly used for learning assessments [6]. These types can help to make some suggestions on how and when to use different assessment methods in serious games.

Formative assessment

Formative assessment occurs as a part of daily instructions. It provides information to both teacher and student about what student has understood through a bidirectional feedback system. Teacher can check misconceptions of students and where they are struggling, on the other hand, student can know about their strengths and weaknesses. This assessment method can be used in games to check the learning of students on each level by giving them feedback or student can be asked to write a few lines about what they have learned today or from this level. Student’s engagement with that particular game play can also be assessed through formative assessment. 'Steven Anderson' the firsthand account of a teacher who implemented Formative assessment in his classroom. He used sticky notes version of an exit ticket to elicit the student learning evidence. He says "What a difference that made". "Pol Everywhere"- Teacher can get feedback pol in real time by getting students response in various ways. Open ended questions are asked to capture the data and spin up tag clouds to aggregate response. It helps teacher to assess the student's approach toward problem solution. "Think Builder"- This tool allows students to ask questions and discuss topics in group. It enables students to work in group, share and create new ideas. It helps teacher to broad the student's thinking level. "AnswerGarden"- It’s an online brainstorming or polling tool. In which questions are asked frequently and response of students are judged. Purpose of all these applications to enhance the student’s positive approach towards problem solving.
All these tools described above can be used for assessment when students are playing games for learning purpose. These tools can be integrated with in games or can be used with games for assessments. This assessment type can be used in games which will be built to keep an eye on student progress during the course, in other words, this assessment type is suitable for in-process assessments.

Summative assessment

Formative assessment is assessment for learning but summative assessment is assessment of learning. It is to evaluate student’s success and proficiency at the end of an instructional unit. For example, how a student played a game and what score he had at the end of game. It can help teacher in assigning the final grade to the student. It makes clear idea to a teacher that how much his teaching was effective and how much students absorbed his point of views.

Diagnostic assessment

Like formative assessment, it is to improve the learner experience but it more looks backward rather looking forward. It is necessary to find the difficulties that a student had in last levels before starting new one, otherwise it can limit his engagement in new learning. "Pennsylvania Department of education" introduced class diagnostic tool CDT as online assessment. It assists educator to identify student's strengths and areas in need of improvement by providing links to classroom resources. CDT reports are designed to picture or snapshot of how students are performing in relation to assessment. These reports are helpful for teachers to pinpoint where students are struggling and provide suggestion for "Next Step" in student academic development.

Dynamic assessment

Dynamic assessment is used to check the student general learning potential especially when they do not have any background knowledge related to that topic. Here games can be used a good assessment tool. For example, student from other countries having no prior knowledge of Norwegian language is asked to play a game teaching some Norwegian phrases or words then the score at the end can give a good guess about general learning potential of that student.
Which assessment type should be used for game is totally depends on the main purpose of the game, for example, if a teacher wants to know about weak areas of student in some subjects then he can use diagnostic assessment and if he want to use for daily assessments then developers should follow formative assessment techniques.

Assessment of learning in games

When we talk about assessment of learning then it means assessment of knowledge, skills and abilities. If a teacher wants to assess the learning with the help of games then he should make sure that it is assessing these all three elements. There could be separate games, some can help to access the knowledge like quiz games and other can assess skills and abilities like epistemic and system thinking games.
Assessing the knowledge means testing a person that how much he knows about a topic or subject. For example, if a person have knowledge of how to play football then it does not mean that he is a football player or he can play football. This knowledge can be assessed with the help of serious game scores and levels. On the other hand, skill comes with practice which can give him ability to perform in real world. Skills includes thinking, reasoning, planning, communicating, managing, teamwork and etc. Most of these are 21st century skills which are difficult to assess in classrooms with traditional methods. Serious games can help to assess these skills by providing them a virtual environment to do experiments. Ability is utilizing any of these skills in a meaningful way to perform any task.

Traditional methods of learning assessments

While developing a new game for assessment, it is better to know about traditional assessment methods used inside and outside of classrooms. These methods can help to design a new game with less work and time required. We can follow the same concepts, build a game and replace them. It would be convenient for teachers and interesting for students. Below are four traditional assessment methods commonly used in classrooms. Each method is discussed with examples that how we can use the concept and replace these assessments with games.

In-Process assessment

Teachers use this method to check which options student has chosen or what decisions he has made during the performance based assessments. It helps to evaluate player’s cognitive skills like thinking, reasoning and decision making abilities. Games can be a best tool to do in process assessment because it can save important moves and decisions made by player during the play. It can also help to find out player’s ability to face pressure by noting the points where they change their minds or tracks. It also can help to check if player is actually understanding the topic or just passing the tests. Assessment during the game can be done by adding some pop quizzes or questions which player has to answer correctly to proceed to next level. These quizzes can be added in between two levels rather than adding during the level because it can reduce the interactivity and hence players interest in game.
One of the good examples of serious games that are using in-process assessment is “Immune Attack”. The objective of this game is to teach students about immune system in good and entertaining way. It does not test the player during the game but it only allow a player to progress in game when he has learned new facts or information about immune system. Players have to perform different tasks for example, identify infected blood vessels in body, train defensive cells of body like macrophages to identify its allies and enemies and countering the attacks from bacteria [7]. To proceed the next level, student must have sufficient knowledge. Features like these can help to do in-process assessments.

Completion assessment

Skills or abilities of students can be assessed by checking that if he/she was able to complete a task. The same concept can be used for assessment through games. To pass the test or course, student has to reach a certain level of game. This is a simple method of assessment for teachers because each level of game can describe the level of learning and understanding of student for that particular subject. Tiagia Park game which is described in next section, could be an example of completion assessment. In this game players has to complete the task by investigating problem, make hypothesis, extract the information and made some conclusions. A player would be able to complete the task if he has good knowledge and understanding of the topic.

Self-assessment

This is usually in the form of questionnaire having the questions about how much they have learned and understood the course but in the case of serious games, students or players can assess their own learning from games in the form of game points or the progress they have made. After that they could be asked to rate their understanding in the topic. The game score or their progress can help them in self-assessment. Self-assessment increases student’s engagement, interest and attention. Not only this, student learn more when they know they have the responsibility of assessment of what they have learned [8].
In a research article “The Reliability, Validity, and Utility of Self-Assessment” they have concluded that self-assessment is reliable method because it produces consistent results for tasks and items over short periods of time due to its psychometric properties. On the other hand, evidences for validity of self-assessment are mixed. They also stated that for self-assessment, a proper student training is required in which they should learn self-assessment skills [8]. For serious games, self-assessment of skills can be taught in the form of a video tutorial after game but before assessment.
CancerSpace is a serious game especially designed for professionals who are working in health care centers. It encourages the self-directed learning and assessing by dealing with real-world situations where they have to work and make decisions just like they have to make in clinics. In this game they can evaluate their own literature learning by helping their staff, make important decisions related to cancer treatment and change their plans according to cancer-screening delivery. Cancer screening rates are the main indicator of learning because the player has to make decision and observe the screening rates which will decide that which course of actions is better. Another feature of this game which helps to do self-assessment is to convince patients through interactions who are reluctant to screening process.  This game has preprogramed decision trees which lead to success or failure of this talk. Success means that player was able to convince patient to get screened and failure means player needs to work more hard. CancerSpace game is using an award system to encourage the players who are having best screening rates. To help in self-assessment, at the end of the game there is a screen display having a summary of decisions that players has made and screening rate that clinic had during the play [9].

Teacher Evaluation

This kind of assessment could be surveys, focus–group interviews, midcourse or periodic student feedbacks and teachers own observations. MCQs (Multiple choice questions) is also used for this purpose which is a very convenient option to assess student’s learning because teacher can check the results easily but it is not suitable for all kinds of assessments, for example in advance mathematics it is not enough to just check if the answer is correct or not. It is more important to check if the student has used right method to solve the question or he just guessed the answer.
Serious games can help teachers to do evaluations in simpler way because games can record the moves made by student during the play. Most importantly, the player’s score can help teachers to evaluate. Many games provide the performance record from the start of game till end in the form of tables and graphs making assessment process for teacher easier.
“SimVenture” is a serious game especially designed for business student’s learning and evaluation. In this game the goal can be set by teachers and students themselves. These goals could be to start a company, deal with issues of finance, sales and market, production and organization. These goals are predefined and can be loaded in game. Performance of player can be assessed by a parameter named “Company value”. Every game session has a limited time to achieve the goal and expressed in the form of months. At the end of each month, summary is displayed having the information about the problems that were encountered and the problems that are most likely to be faced. Not only this, this game can provide a straightforward comparison of all players [2].

Possible solutions to use games for assessment

After reviewing the literature about learning assessments and some serious games, a new question raised: how to use these assessment techniques in serious games? One obvious way is design new games for assessments and other is to alter the existing games in some way that they can meet our assessment requirements. To design a new game for assessment what is important to consider? How we can use existing games for assessment? And which option is better? To answer these questions these two solutions are discussed in detail.

Use existing games for assessment

Feedbacks are an important feature of games which make it suitable for assessment. Other parameters like score, levels and player’s health and other conditions can help to evaluate the engagement and understanding of a player in particular game. A teacher can assume about the knowledge of a student with the help of these parameters but to assess various skills and abilities, especially 21st century skills, there is need to take, think and use these differently. There could be some events in games which can assist in assessments. Serious games already supports assessments in some ways as described above but a little alteration and addition of some supportive technologies can make these games a better and reliable tool for assessment.
In this section we will try to figure out some possible ways to use these instructional serious games for assessments. On the basis of learning, serious games can be categorize in to different types:

1) Civilization Games, 2) System Thinking, 3) Epistemic Games, 4) Science Content Learning, [10].

 These kind of digital games help students to learn deeply and to acquire twenty first century competencies. To assess the learning from game, the same categorization can be used. Each category is discussed with some recent examples of serious games and suggestions on how games like these can be used for learning assessment.

Civilization Games

These kind of games are used to teach history to students by giving them deep understanding instead of just memorizing the past events. With the help of these games, students can learn about more than one civilization by playing game again and again. They can easily learn different attributes of particular civilization which otherwise is difficult to memorize in classrooms. Students are asked to build a civilization and take some important decisions to keep it alive. In this way these games not only teach about history but also some important subjects like economy, culture, geography, war, democracy, technological advancement, importance of population control and sometimes archeology.
Games like Age of Empire and Civilization are famous civilization games which can teach about economy, trade, technological advancement and war. “Civilization” has greater scope as compared to other games. For example it starts before Bronze Age in about 400 BC and ends at Space Age which is covering some future technologies. To learn and test the knowledge of history there are many games like American Conquest,Dig Deeper, The Great fire of London, and etc.
These games provide a wealth of information on how a student has played, what strategies he has chosen and what were his weaknesses and strengths. These games give a detailed analysis of performance of player in the form of numbers as well as graphs. Not only this, these games can compare performance of each player with others. Games can be modified for teachers and curriculum creators so that they can produce the customized scenarios for particular topic. After which teacher just has to ask students to play the game in classroom, which will be actually a test of how well they have understood. Assessment can also be done by asking students to customize the game according to given scenarios.

Systems Thinking

These games can be used to check the system thinking skills of student because in these games, players have to consider a lot of parameters, elements and relationship between these elements. If player will not consider them all, he will not be able to win or proceed in the game. These games can ask students to create and understand the relationship between isolated facts and events which were taught in classroom which can help teacher in assessing the thinking skills of students.
Gamestar Mechanics is a game where player can create their own games from start and for this purpose they have to make strategies, some rules and define some parameters. To create a functioning and challenging game, in which student has to think hard about these elements and parameters otherwise game will not work properly. This game can be used to assess the game designing skills of students.
System thinking is an important fundamental principle of a good game [11]. That is the reason many institutes are creating digital tools to assess the thinking skills of students, for example, Institue of Play. There are several more examples of system thinking games like “Gamestar Mechanics” which can help teachers to access thinking skills of students for different subjects.

Epistemic Games

In this era of global competition students need to learn to think like professionals. For this purpose epistemic games have been designed where players learn being innovative and creative about resolving real world problems. Students engage themselves in a particular way for working out the problem [12].These games teach students to solve real world problems by practicing concepts and principles in similar way that professionals do. Urban Science is an epistemic role playing game for professional level urban planning. Players are provided with necessary information such as city budget city, revenue, pollution and other issues. They can interact with expert planners using a special mapping tool iPlan. Changes can be proposed for the land using geographic information system (GIS) for example students’ can turn a parking lot into a playground .They take into consideration city’s pattern of development, resources ,demographic trends and physical features. They work as a team to prepare proposals by visiting sites and conducting interviews for the landscape they want to work on. Further they gather valuable information to construct proposal for redevelopment [10].
Epistemic games provide people with the opportunity to learn by helping them to do experiments which they cannot do in real world due to failure risks. Moreover the cost of failure is reduced by working in a simulated world which encourages a person to work in a risk free environment [13].
Similar games can be used by companies to test an employee before hiring whether he will be able to perform in real world scenarios or not. Not only this, companies can ask their employees to play this game so that they can keep the track of learning experiences of each employee. It can also be used by teachers to assess their understanding at the end of the course by making it complementary to pass the course.

Science Content Learning

Science content learning games help students to experiment through trial and error by observing the effects. Engaging students in science process can motivate learning of science [14]. Users can visualize laws and theories of science and implement the knowledge. Tiagia Park is an example of such games developed by Barab et al. at Indiana University [10]. In this game the player engages himself in some scientific investigations, for example, to find out the cause of death of fishes in a pound. On the basis of the information extracted, the player concludes some hypothesis and presents a solution for control and treatment of fishes. For tackling such situations, students take into consideration the correlation of certain social, ecological and economical factors for applying concepts of science. Such games increase the learning capability of students which is helpful for memorizing information in a better way. If feedback is provided to the students then it enhances their learning capability.
The game described above can be used as a completion assessment.  These can be used as a good assessment tool for school students to assess their understanding by giving the different problems to solve and make some conclusions.

Business Simulation Games

The key mechanism of business games is to train players in business skills and evaluate their performances. These games basically focus on management, finance, human resource and etc.
An example of business game is SimVenture in which a player has to supervise and handle a company taking into account some major issues like finance, market, sales, organization and production. A player has variety of options and opportunities to operate in these areas. The ‘’company value’’ defines performance of a player. The player has to deal with the balance sheets, profit, loss and cash flow. The purpose of the game is to maximize profit or cash flow taking into consideration the element of bankruptcy. Students are assessed at different levels in this game during which they have to face some critical cases.
PIXELearning’s Enterprise Game is another example of web based single player business game that trains and encourages enterprise. It makes players aware of the basics of enterprise and how to run your own business. Using proper market strategies the product produced should meet the market demand both in quality and price. The player can keep an eye on the performance of Competitor Company so that the player can perform better [15].
Teacher can recommend games like these to students to play so that they can improve and assess their business skills. Business games can be used for assessment by business companies to hire new people just like described in epistemic games.

Games for Health

Special games have been designed that help medical students to retain necessary information. These games have been specifically designed for physical rehabilitation, psychological therapy, exergaming, etc.  An example of such game is Immune Attack that does not directly assesses a player, rather helps him memorize information which is necessary for further progress in the game. In this way progress in the game could be a good way to assess the knowledge of player. Moreover a player needs to identify an infected blood vessels, diagnose an ailment, enable white blood cells to detect and fight against bacteria [16]. In this way it is helping the player in self-assessing their knowledge these subjects.
Health games can be used in medical centers to assess the knowledge and skills of medical staff and students. It allows them to do self-assessment by solving some problem and doing some experiments and see some after effects. Medical and health related jobs are very crucial where even a small risk could not be taken. Here these games can play a very important role in term of assessment.

Design new games for assessment

Designing a new game from scratch is costly and time consuming process and also have some risks. You have to take care of all game elements otherwise it will not work properly. There is need to do a lot of brainstorming for example, figure out elements of games, what game type it will be, who will play this game, etc. While working on these points you should not lose focus from the main inspiration, which is assessment in this case. After brainstorming, there is need to design the game which include to check the game constraints and careful selection of different components for example, music, avatar, mechanic, background and etc. There is need to design a proper, meaningful and constructive feedback system in game which is very important for any kind of assessment and learning. Game development is a very time taking process and needs a lot of resources because after completion there is always need to reiterate the game. After getting the feedback from players there is need to change game features according to player’s feedback and this cycle goes on.
There is need to take some special considerations when you are designing a game for assessment purpose. One of the important consideration is assessment validity for which there is need to do a critical analysis, legal judgment and follow particular scientific process. We have to be confident and sure about assessment validity, otherwise the whole game would be useless. For example, for designing a game for entrepreneurs to assess business skills there is need to do a proper analysis on how assessment can be done(e.g. self-assessment), which assessment method is suitable for this case (e.g. formative or summative), method used for assessment will actually give valid results and etc. This validity analysis is very important before developing the game because a single wrong decision can spoil the actual purpose of game. Legal judgment is also very important and it should be done by game itself with the help of some artificial intelligent algorithms. In other words, it is formal decision made by game about learning of player. Teacher have to do research about evaluation of particular skill and discuss it with developers. There is need to test these algorithms carefully because a minor error can directly affect the validity of assessment. Assessment of some skills will be valid only if it is following proper scientific or real life process, for example, if we want to assess game designing skills of some students then game should follow the actual game designing process including all those steps which are discussed earlier. If game is not following actual process then assessment from this game could not be valid. A game should be designed by keeping in mind the importance of assessment validity.
The development of games for assessment purpose could be complex as compared to other serious games which are used only for learning purposes because it would have some extra requirements. This could be extending the game features in some manner, for example, the feature of game to save every activity of player can be used to do analysis for some skill but for that there is need to write extra artificial intelligent algorithms. These additional requirements can increase the cost of the game because more work needed to be done, so more resources are required.
After developing a new game for assessment purpose, there is need to make claims about its effectiveness and also prove it. The market for these games are not big enough which makes it more risky for any company to develop a game like this. Serious games are designed to target a specific group of people who want to learn or assess certain skills. The main goal serious games is education not entertainment but it can be designed to be attractive and entertaining so that a large group of people can be targeted just like commercial games without losing the focus on main objective of game. 

Indirect measures of Assessment

 The first proposed method is simple and easy but not stealth. It cannot provide solid results about learning but it can help to evaluate player’s learning from the game. The measure of student engagement can guide us about his skills level, more engagement means strong knowledge and skills [11]. It is hard to measure the player’s level of engagement and enjoyment by guessing. There are already tools designed to measure student engagement in games including “Game Development Questionnaires” [17] and “Game experience Questionnaires” [18]. Another example is “EGameFlow” which is basically designed to measure the player enjoyment from e-learning game [19]. With the help of these tools, player’s engagement can be measured and hence learning but filling the questionnaires is not interesting and attractive way of assessment. This concept can be used to build the games for assessment. This method may not give the accurate results about student learning but can be used to for in-progress assessment to keep an eye on student’s progress.
One other method to measure the engagement of a player in a game is by investigating the psychophysical responses like Facial electromyography (EMG), Galvanic skin response (GSR), Electroencephalography (EEG) and cardio vascular measures [20]. Plenty of work in remaining in this field but it will not help developers to design better games but also help in various assessments in future.
To assist the teachers in student’s evaluation with the help games, different audio and video recording tools can be used. There are several tools available which can record screen and save it in the form of video files. Video cameras of the computers can be used to make videos to record actions and microphone can be used to record their voices when they are thinking aloud during the gameplay. It is now not difficult to extract the information from these audio and video files as there are many convertor tools which can transform it from one to another useful form. This information can be used by teacher in debriefings and in assessment of some skills.
There are numerous tools have been designed for assessment purpose for different subjects and field of studies [21]. The idea and information from these tools be used to create games. Game can increase interactivity, entertainment and interest of student in an assessment. In this way less work and time required to build the games for assessment because all required research and information for particular domain would be already available.

Challenges

There are some challenges that teachers can face while using games for assessment purpose. In this section all possible challenges and issues are discussed in detail.
As described above, there are several different ways of assessment. Some are suitable for some subjects and skills but others are not. So the first challenge for examiner could be to decide which assessment method could be most suitable for that particular assessment. For some skills, self-assessment could be enough for evaluation but for some skills teacher evaluation is also required. After deciding the assessment method, teacher has to know which game type is suitable for assessment. It depends on the skills which he want to evaluate. If he wants to evaluate decision making and critical thinking skills of a student’s then there are many games which he can choose like system thinking games, epistemic games and etc. But if the students are business students then it’s better to choose business simulated games.
The simpler and traditional way of assessment from games is to check the player score and progress in game, but sometimes these parameters are not enough to reveal actual learning. To get more accurate results, a teacher has to observe the moves of students during the play. The decisions he made and the tracks which he chooses can be saved in game but it is not an easy job for a teacher to review these activities of all students and then make conclusion about learning. There are some games which makes this teacher’s job easy with the help of some artificial algorithms by making summary reports. Some examples of such games are already discussed above. But still it requires teacher to give attention to individual students.
It is always easy to test knowledge as compare to test the skills, especially abstract skills like cognitive skills. There are many games already designed or are in process of development which claims that they can teach and assess these cognitive skills [22]. There have been done many researches on these games that proves that learning happens with these games (e.g. [2], [3]) but less work is done on if these games are also good for assessment. It is difficult the trust any game right now which claims that it provide actual measure of learning.
When players are playing the games for entertainment or even for learning purposes, cheating can be ignored. But in assessments, cheating and cheat codes are not affordable, otherwise it can destroy the whole assessment system and main objective of that game. A minority of cheaters can destroy the game for all others. This is a bitter reality that cheating in games have a very colorful history. So while designing or using any game as an assessment tool, it is very important to check all possible ways of cheating and make it insusceptible from cheating and cheat codes.
Stealth assessment system is quiet but very powerful way of gathering the student’s performance data and analyze it to infer the student’s level of competences for some skillsets. But sometimes it can effect assessment in a negative way. If a player do not know about the assessment then he may not take game seriously. Instead he can take it as a source of entertainment which can affect the assessment results. If this happens then it is hard to guess which player played the game seriously and who did not.
On the other hand, if the assessment is not stealth then the results could have hawthorne effect. In harthorne effect subject changes its behavior according to environment. This happens when people know that they are under observation. It results in increased attention and hence increase temporary productivity which can affect the actual assessment. To solve these problems there is need to find a middle way for assessment purpose.
When we want to assess 21st century skills through games like critical thinking, reasoning, teamwork, collaboration, management and etc, then it would not be a simple or a small game. It will be a game with a huge game environment and tasks. Sometimes it is very difficult and time taking to understand or learn the game in the start. Here it is a challenge to make these serious games easy to understand by using some techniques or by adding some features which can help to learn game in the start.
In-process assessment can affect the player’s experience of playing games. It could be in the form of pop-up questions during the play or questionnaire at the end of each level. These pauses can affect game interactivity and hence could be a reason of losing the interest of player in game. There is need to find a more appropriate way to do in-process assessment.

Discussion

Technology assisted techniques to help in assessment are being used for many years in the form of some simulations and online questionnaires but now it has improved and simplified whole process. It has solved a lot of teacher’s problems by calculating the accurate scores and formulating the student’s performance reports. Serious games are now being used as a learning tool but less work has be done to use them for learning assessment. It can help to assess the areas of learning that traditional methods cannot address. For example, if a teacher or trainer want to give a test to evaluate the understanding of tennis rules then he can actually test his knowledge of game rules but he cannot assess player’s ability to apply these rules to actual tennis game. In other words, he cannot assess his skills and abilities. Similarly, when a company is going to hire an employee then it is difficult to evaluate his practical skills on the basis of written test results and interviews. Here serious games can help to evaluate skills. 
The next question is how to use these instructional serious games for assessment? To answer this questions two solutions have been proposed. One is to use existing games with little alteration or with some assistive technology and the other is to design and build the game for assessment from scratch. Serious games already have a lot of features that can help in simple assessments like scores, levels or progress in game, health, etc. But to assess 21st century skills, these parameters are not enough. The purpose of these games can be changed by thinking and using these games differently. In some games there is need to customize these games for assessments. This is a good way of assessment but it has some drawbacks. The first one is clear that you have to modify a commercial game which needs a lot of time and resources. It is just implementing that game again in a different way which needs resources. The second one is, limited number of skills can be assessed if we are using the existing games. In other words, we will have freedom to assess only those skills which a player can learn from that game. This is the reason that the games have been categorized in paper on the basis of learning outcomes (knowledge, skills and abilities) which can be assessed from these existing games. The third drawback is that there will be no solid surety about validity and reliability of results because assessment was not the actual purpose of these games.
If a game is planned to create for assessment from the scratch, there is need to do a proper analysis about assessment validity, game design, market place, development cost and additional assessment requirements. Assessment validity is very important when you are building a game especially for assessment purpose because if the assessment results are not valid then game and all work done is useless. Market place for such games is also not bigger as compared to common commercial games. To solve this problem the best way is to design a game, develop its engine and contact educational institutes and suggest them that this is a modern and better assessment tool.
The former solution is simple but not reliable in terms of assessment validity. If there is need to alter existing game for assessment purpose then it’s better to design a new game by keeping in mind the main assessment requirements. One big drawback of second solution is, it is expensive and time taking. Not only this, right now market for these games is not big enough.
It is very difficult task to figure out if somebody has learned something but it is possible if we use serious games. When we talk about learning then it means knowledge, skills and abilities. There are several games which can assist to evaluate the learning but not all of them can help to assess skills and abilities. A game which is designed to assess the knowledge only, cannot evaluate skills and abilities. On the other hand, a game designed to assess abilities or skills can also help to assess the knowledge because knowledge is just the information which a person has, it cannot describe his skill or abilities.  Games like Civilization and Age of Empires can help to assess the knowledge of different civilizations and some cognitive skills like critical thinking. Gamestar Mechanice can help to assess game designing skills and Urban Science can help to assess regional planning skills. But games like SimVenture and CancerSpace can help to assess the abilities of players because in these games player has to use his previous knowledge and skills (thinking, management and teamwork) to perform a task. The results will define their abilities to perform certain tasks in real world.
There are some types of learning assessments which can assist to design and use games according to our purpose of assessment. For example, formative, summative, dynamic, and diagnostic assessment. Which assessment type should be used for game is totally depends on the main purpose of the game, for example, if a teacher wants to know about weak areas of student in some subjects then he can use diagnostic assessment and if he want to do general and more detailed in-process assessment then formative assessment would be best option. And if he wants to know about general learning abilities of a students at the time of admission then he can use games following the concept of dynamic assessment. On the other hand he wants a supportive tool to assist him to assign a final grade to student then he can use summative assessment.
Traditional assessment methods used in classrooms like completion assessment, in-process assessment, and self-assessment and teacher evaluation, can be replaced by games following the same concept and purpose of assessment. For example, if a teacher want to do in-process assessment or simple teacher evaluation then formative assessment techniques could be used to design the game. Because in formative assessment there is a two way feedback system and it is used during the learning of course. For completion and self-assessment, summative assessment techniques can be used because summative assessment techniques help to assess at the end of learning process. CancerSpace can be used for both self and completion assessment because it is using summative assessment techniques by displaying the summary of performance at the end of the game. It is also using award system on the basis of overall performance of player.
Assessments needing more teacher involvement would not be a better option. The results of assessments would be more reliable if it is more processed and analyzed by assessment algorithms. Here our objective is to design a game to assess 21st century skills which is not an easy task and cannot done accurately in classrooms. Less teacher involvement means less human errors. This can also help to reduce the effect of teacher favoritism on assessment. It will increase the trust of students on assessment methods and will make them more responsible, independent and serious towards these assessments. Not only this, it will also favor teachers by reducing their workload and allowing them to give time to other important teaching related activities.
There are some indirect methods of learning assessments for example, by measuring the student engagement in game, by investigating the psychophysical responses and by recording student physical activities while playing game. These methods are not reliable because these are understudied and requires a lot of research work. If these methods are actually useful for learning assessment then these can be used for in process assessments.
It is very challenging and time consuming to design an in-game assessment. There are many challenges for using games as an assessment tool for example, cheating, assessment type selection, assessment method selection, checking the assessment validity of a game and etc. One important challenge that could be faced is that if we use stealth assessment to remove time and exam pressure from a student and keep the assessment hidden then there are chances that students will not take them seriously which could effects the results. On the other hand if assessment is not stealth then the results could have hawthorne effect.
With the passage of time, it is expected that these problems can be solved when more and more studies would be done on this topic because now assessment is new emerging topic in field of serious games.

Conclusion

Kevin Corti of PIXELearning stated, “Assessment is future of serious games. [Serious Games] will not grow as an industry unless the learning experience is definable, quantifiable and measurable. Assessment is the future of serious games” [23]. There is a lot of work done to prove that serious games actually supports learning, but to use it for assessment, a lot of research work is still required. One big advantage of serious games is that it already provides some kind of self-assessment in the form of scores, level and progress reports, but to assess cognitive skills, these parameters are not enough. Serious games are designed to teach 21st century skills but it is also important to measure these skills and it is possible with the help of serious games. To achieve this goal, there is need to think and design serious games differently.  There a lot of examples of serious games discussed in paper to reiterate these games which can be and should be used for assessments. And to design a game from scratch there is need consider assessment validity, game design and assessment requirements. Designing a new game is better option than reiterating the existing games because in this way the main focus will be on the purpose of game.  Traditional assessment methods that are used in classrooms are not sufficient to assess cognitive skills. By using the same concepts of traditional methods, new games can be designed and replaced with these methods. It can increase the interest of students in assessments and assist teachers by simplifying their assessment jobs.
To use games for assessment there are some challenges which need to be addressed, for example, cheating, choosing best game for assessment, accuracy of results for cognitive skills, etc. To solve the problems like these, more research work is required.


References


  1. “About Eye Wire, A Game to Map the Brain” from eye wire blog. Retrieved from: http://blog.eyewire.org/about/
  2. “Authentic Business Simulation” from Semi Venture- The ultimate business simulation game blog. Retrieved from: http://simventure.co.uk/index.php
  3. Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. Computer games and instruction, 55(2), 503-524. 
  4. Backlund, P., & Hendrix, M. (2013, September). Educational games-are they worth the effort? A literature survey of the effectiveness of serious games. InGames and Virtual Worlds for Serious Applications (VS-GAMES), 2013 5th International Conference on (pp. 1-8). IEEE. 
  5. Brown, S. (2004). Assessment for learning. Learning and teaching in higher education, 1(1), 81-89. 
  6. “Types of assessment - some definitions” by University of Exeter. Retrieved from: http://as.exeter.ac.uk/support/staffdevelopment/aspectsofacademicpractice/assessmentandfeedback/principlesofassessment/typesofassessment-definitions/
  7. Kelly, H., Howell, K., Glinert, E., Holding, L., Swain, C., Burrowbridge, A., & Roper, M. (2007). How to build serious games. Communications of the ACM, 50(7), 44-49. 
  8. Ross, J. A. (2006). The reliability, validity, and utility of self-assessment. 
  9. Swarz, J., Ousley, A., Magro, A., Rienzo, M., Burns, D., Lindsey, A. M., ... & Bolcar, S. (2010). CancerSpace: A simulation-based game for improving cancer-screening rates. Computer Graphics and Applications, IEEE, 30(1), 90-94. 
  10. Shute, V. J., & Ke, F. (2012). Games, learning, and assessment. In Assessment in Game-Based Learning (pp. 43-58). Springer New York. 
  11. Sweetser, P., & Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment (CIE), 3(3), 3-3. 
  12. “Learning from epistemic games” from National Network for IT-Research and Competence in Education (ITU) website. Retrieved from: http://www.itu.no/Learning+from+epistemic+games.9UFRnGYy.ips
  13. Shaffer, D. W. (2007). Epistemic games to improve professional skills and values. In meeting of the Organisation for Economic Co-operation and Development (Center for Education and Research Innovation) Expert Meeting on Videogames and Education, Santiago, Chile. Retrieved November (Vol. 16, p. 2008). 
  14. Honey, M. A., & Hilton, M. (Eds.). (2011). Learning science through computer games and simulations. National Academies Press. 
  15. “PIXELearning's Blog – Serious Games and more-Serious Games and Immersive Learning Simulations for learning and development”. Retrieved from: http://pixelearning.wordpress.com/about/
  16. “A History of Immune Attack” from immune attack- federation of American scientist’s webpage. Retrieved from: http://immuneattack.org/2009/12/a_history_of_immune_attack.html
  17. Brockmyer, J. H., Fox, C. M., Curtiss, K. A., McBroom, E., Burkhart, K. M., & Pidruzny, J. N. (2009). The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology, 45(4), 624-634. 
  18. IJsselsteijn, W., van den Hoogen, W., Klimmt, C., de Kort, Y., Lindley, C., Mathiak, K., & Vorderer, P. (2008, August). Measuring the experience of digital game enjoyment. In Proceedings of Measuring Behavior (pp. 88-89). Netherlands: Maastricht. 
  19. Fu, F. L., Su, R. C., & Yu, S. C. (2009). EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52(1), 101-112. 
  20. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of Serious Games: an overview. Advances in Human-Computer Interaction, 2013, 1. 
  21. Zoanetti, N. (2011). Software for online testing and quizzes. assessmentfocus. com, Retrieved from: http://www. assessmentfocus. com/online-testing. php. 
  22. “GlassLab: Transforming Learning and Assessment through Digital Games” from Institute of Play webpage. Retrieved from: http://www.instituteofplay.org/2012/06/glass-lab-transforming-learning-and-assessment-through-digital-games/
  23. Ritterfeld, U., Cody, M., & Vorderer, P. (Eds.). (2009). Serious games: Mechanisms and effects. Routledge.


0 comments:

Post a Comment