3.1 . Design of a Virtual World-based Learning Environment

We adopt a constructivist view of learning which argues that meaningful learning is considered as a collaborative and active engagement in an authentically rich and realistic environment (Jonassen, 1999). There is strong empirical evidence that, in TML contexts, collaborative learning has superior learning effects in comparison to individual learning (Lim, Ward, & Benbasat, 1997; Lou, Abrami, & D’Apollonia, 2001). This is because group learning facilitates social interactions among learners and forces them to engage in deeper-level thinking (Alavi & Leidner, 2001). Virtual worlds facilitate constructivist learning through their ability to simulate a more realistic context and to support learning-by-doing (Leidner & Jarvenpaa, 1995). Such simulated learning environments are particularly useful for information security learning because they allow learners to make mistakes (e.g., an insecure router setup, a weak password choice, or purchases from insecure Web sites) without risking negative consequences when those same mistakes occur in the real environment. Figure 1 illustrates the design of a virtual world-based, simulated, collaborative learning environment in the context of information security learning in Second Life.

새창으로 보기

Fig. 1

A virtual world-based simulated, collaborative learning environment

3.1.1. Virtual World Features

Research concludes that two features distinguishing virtual environments from other technologies are: immersion and interactivity (Steuer, 1992; Walsh & Pawlowski, 2002; Witmer & Singer, 1998). Immersion refers to the extent of representational richness and the range of sensory modalities employed (Steuer, 1992). For example, a virtual world that supports both textual and audio chat capabilities is more immersive than another virtual world that supports only a textual chat capability. It is important to recognize that these immersive features describe technology characteristics, not an individual’s responses to the technology (Slater, 1999). The terms system immersion and psychological immersion are often used to distinguish the former concept from the latter. Despite their distinctiveness, the system immersion and psychological immersion concepts are strongly related. In particular, system immersion promotes a sense of psychological immersion or “a psychological state characterized by perceiving oneself to be enveloped by, included in, and interacting with an environment that provides a continuous stream of stimuli and experiences” (Witmer & Singer, 1998). Table 1 illustrates our application of immersive features in Second Life. These include team avatars and custom-made physical structures and objects that provide high fidelity with the scenario described in Figure 1. Figures 2-4 show a Second Life store, a message board, portals, and an information security readiness scoreboard, respectively. These are key objects to facilitate learning in our Second Life environment. Figure 5 shows an online store from which the teams can purchase their products.

새창으로 보기

Fig. 2

A team Second Life store

새창으로 보기

Fig. 3

A team message board

새창으로 보기

Fig. 4

An information security readiness scoreboard (left) and portals (right)

새창으로 보기

Fig. 5

An online store

새창으로 보기

Table 1.

Virtual World Features: Immersion and Interactivity

Interactivity refers to the extent to which a user can participate in a virtual environment by modifying the form and content of the environment in real time (Steuer, 1992). Interactivity is a prominent feature of a virtual environment. More specifically, it enables users to maintain active involvement and adjust the mediated environment according to their interests. Suh and Lee (2005), for example, found that a virtual environment enhances consumer learning about products. In this study, we incorporated interactivity through avatar customization and several forms of avatar-object interactions (see Table 1).

3.1.2. Self-Regulation

Self-regulation refers to a general skill to direct and sustain efforts toward the attainment of one’s goals through goal setting, progress evaluation, and goal and strategy adaptation to ensure success (Zimmerman, 2000). There is a strong body of evidence suggesting that learners who self-regulate their learning have higher academic achievements (Pintrinch & Degroot, 1990; Zimmerman, Bandura, & Martinez-Pons, 1992). This study adopts the social cognitive perspective on self-regulation in which learners play an active role in enacting self-regulation towards learning (e.g., setting learning goals, choosing learning strategies, having sufficient motivation to learn). The social cognitive theory (Bandura, 1986) emphasizes a learners’ personal agency and suggests that, in addition to cognitive engagement, self-efficacy and motivation are critical components in self-regulatory processes and learning outcomes.

We propose that instructional strategies in virtual worlds should include interventions that guide learners to apply self-regulation in their learning. Some of the interventions include, among other things, formulating appropriate goals, enhancing their motivation to learn, and using feedback to adjust learning goals and strategies. In particular, we draw on Zimmerman’s (1998) process of self-regulation model in which three cyclical phases of self-regulation are distinguished: (1) the forethought phase, (2) the performance control phase, and (3) the self-reflection phase.

The forethought phase occurs before learning and provides opportunities to prepare learners to set realistic goals and outcome expectations, plan strategies, and develop a positive mindset towards learning. The performance control phase involves processes during learning. During this phase, self-evaluation and external feedback are critical to encourage learners to continue their learning efforts. The self-reflection phase involves reflection or evaluation of outcomes compared to goals after performance. We implemented a number of interventions across all three self-regulated learning phases as summarized in Table 2.

새창으로 보기

Table 2.

Self-Regulatory Interventions

3.2. Evaluation of a Virtual World-based Learning Environment

Following the proposed design framework of a virtual world-based learning environment, we developed an information security learning environment in Second Life. Next, we evaluated the effectiveness of the design framework on learning outcomes and investigated the extent to which different psychological traits of students (e.g., learning goal orientation, computer learning self-efficacy) affect learning experience and outcomes.

Both learner performance and satisfaction have been widely used as measures to evaluate the effectiveness of learning environments (Alavi, 1994; Piccoli, Ahmad, & Ives, 2001). However, previous research in IT training suggests that performance does not offer a complete explanation on future behaviors. Self-efficacy (Bandura, 1986) which refers to individuals’ belief in their ability to perform behavior has been found to be an important predictor of behaviors when confronting a task that requires individuals to apply the acquired IT skills. Therefore, in this research, effectiveness is measured by learning outcomes (performance and self-efficacy) and satisfaction. We discuss the hypotheses put forth in our research next.

Virtual world-based learning environments provide rich information and enable more interactions with learning contents and other learners. Research reported that rich, interactive, and engaging presentations of information enhance learning among consumers (Li, Daugherty, & Biocca, 2001; Suh & Lee, 2005). The high degree of interactivity enables learners to have higher control of the learning environment and facilitates active learning. Higher degrees of learner control improve learner performance and lead to higher satisfaction (Merrill, 1994). As discussed earlier, there is strong evidence that self-regulatory learning through goal setting, feedback, and self-reflection affects learning outcomes (Schunk, 1991; Schunk & Ertmer, 1999). In some cases, virtual world features can promote self-regulatory learning processes leading to positive learning outcomes. For example, the information security scoreboard allows learners to view their own team’s score as well as those from other teams. This enables learners to monitor their progress against their goals as well as evaluate their relative progress with other teams. Perceived progress promotes self-efficacy and motivation, which enhances learning. Hence, we propose the following hypothesis:

H1: The design of a virtual world-based learning environment that leverages unique virtual world features and incorporates design interventions to promote self-regulatory learning will enhance learning outcomes.

Learning goal orientation is a motivational variable that explains how individuals respond to new learning environments (Brett & VandeWalle, 1999). In other words, learning goal orientation relates to the use of learning strategies in new learning situations (i.e., virtual world-based learning environments). The concept of learning goal orientation is critical in high-order skills learning (such as information security learning) because it requires the use of more complex learning strategies in which learners associate the new information with other more familiar materials (Fisher & Ford, 1998). Individuals with high learning goal orientation set challenging goals for themselves and pursue adaptive learning strategies in which they are persistent in their learning e orts. They also report enjoyment with the challenge (Brett & VandeWalle, 1999). Thus, we propose the following hypothesis:

H2a: In a virtual world-based learning environment, learning goal orientation will be positively associated with perceived enjoyment.

Self-efficacy is a critical variable that affects all phases of self-regulation (Schunk & Ertmer, 1999). Students who are familiar with traditional classroom environments were reported to have difficulty in managing different delivery methods (Gall & Hannafin, 1994). Although some may argue that self-efficacy beliefs towards learning in a technology-mediated environment may have a direct influence on learning outcomes, evidence from previous TML studies did not find support for the relationship. In e-learning environments, Santhanam, Sasidharan, and Webster (2008), for example, reported that self-efficacy beliefs towards learning in a technology-mediated environment do not explain differences in changes in declarative knowledge and hands-on performance. In contrast, research evidence seems to suggest that perceived enjoyment mediates the influence of self-efficacy beliefs on learning outcomes. For example, several students reported having difficulty and frustration during their learning process in the new Web-based virtual learning environments (Piccoli, Ahmad, & Ives, 2001). Thus, we propose the following hypothesis:

H2b: In a virtual world-based learning environment, computer learning self-efficacy will be positively associated with perceived enjoyment.

Research has shown that psychological characteristics such as motivation and a degree of effort are positively correlated with learning outcomes (Alavi & Leidner, 2001). It is expected that individuals who experience enjoyment while interacting with a virtual world-based learning environment are motivated to put more efforts towards learning tasks, thus enhancing learning outcomes. Perceived enjoyment is related to the notion of ow (Hoffman & Novak, 1996) and research suggests that flow leads to several positive outcomes including increased learning and satisfaction with the experience. Therefore, we propose the following hypotheses:

H3a: In a virtual world-based learning environment, perceived enjoyment will be positively associated with learning outcomes.

H3b: In a virtual world-based learning environment, perceived enjoyment will be positively associated with satisfaction.

4.1. The Learners

The virtual world-based information security learning module using Second Life was incorporated into an “Introduction to Information Security” course at a major university in Fall 2008. The students (52 males, 7 females) were undergraduate students enrolled in the course at two campuses. The small number of female students is a reflection of the 13% female enrolment at this college during that period. Most of the students are experienced computer users with an average experience of 11.9 years. About one-third (33.3%) of students have no prior experience with Second Life.

4.2. Procedure

All students attended a semester-long course. The course met twice a week for a period of seventy-five minutes. Students were divided into teams of four to work on the virtual world-based information security learning module during the two week period of classes. To control for potential biases, an identical set of procedures, materials, and classroom protocols were used by the instructors of the two classes.

Before their engagement with the module, students were introduced to the virtual world in general and Second Life in particular through a class lecture, a short video clip, and a brief hands-on exercise. A written description and instruction about the activities were used to provide information about learning objectives, the scenario, team assets, and activities. During the rest of the two-week class periods, the teams were given a set of interrelated tasks to open a new virtual store in Second Life.

4.3 . Measures

4.3.1. Dependent variables

Consistent with previous TML research (Piccoli, Ahmad, & Ives, 2001), the dependent variables are measured by learning outcomes (learners’ performance and information security self-efficacy) and satisfaction. Learners’ performance is measured with a conceptual knowledge test and hands-on information security readiness scores (technical security readiness and online purchase behavior scores). In this study, the conceptual knowledge test included thirteen multiple choice questions. Satisfaction was measured through a validated scale (Piccoli, Ahmad, & Ives, 2001). Information security self-efficacy was measured by adapting validated scales from Chai et al. (2006).

4.3.2. Pre-learning survey

A questionnaire capturing the learners’ gender, self-rated computer experience, virtual world experience, learning goal orientation, and computer learning self-efficacy was administered at the beginning of the learning module. Learners were also asked to respond to the questions that test their information security knowledge and capture their self-efficacy before learning. Both learning goal orientation and computer learning self-efficacy were measured through the original scales from Zweig and Webster (2004) that were also used in Santhanam, Sasidharan, and Webster’s (2008) e-learning study.

4.3.3. Post-learning survey

At the end of the learning module, learners were asked to respond to another questionnaire to capture their information security knowledge and self-efficacy after learning, perceived enjoyment, and satisfaction. Perceived enjoyment was measured through a validated scale from Deci et al. (1994).

5.1. Measurement Model Validation

Two related analysis procedures were used to test the effectiveness of the virtual world-based information security learning module. Paired t-test was used to test the pre- and post-learning outcomes. Partial Least Square (PLS) was used to evaluate the influence of psychological learning variables on learning outcomes. To determine the appropriateness of analysis at an individual level, we calculated the Intra-Class Correlation (ICC) (Shrout & Fleiss, 1979) and evaluated the possible team-level effects. The ICC is derived from a nested ANOVA test that assesses whether or not members in the same team produce more similar outcomes. The ICC score of 1.0 indicates that all the team members have the same score, and the ICC score of 0.0 indicates that people within a team are no more similar than people from different teams. In other words, the lower value of the ICC indicates that the team level does not have any significant effect and the analyses can be conducted at an individual level. The ICC values for this study are from 0.007 to 0.42 with an average of 0.18 which were below the suggested theoretical guideline of 0.5 for considering the team effects (McGraw & Wong, 1996). Thus, all the analyses were conducted at an individual level.

Construct validity was evaluated before conducting further analysis. Since all of the constructs with the exception of information security performance are reflective constructs, the conventional construct validity assessment using a common factor analysis is used. The convergent validity and reliability were evaluated by examining item loadings and composite reliability. Among the twenty eight measures, an item loading for one measure is significant at p < 0:10 and the other twenty one and six measures are significant at p < 0:01 and p < 0:05 respectively. The composite reliability values are high, ranging from 0.84 (information security self-efficacy) to 0.96 (perceived enjoyment).

Discriminant validity was evaluated by examining the loadings and cross-loadings of item-construct loadings, and average variance extracted (AVE). Discriminant validity is established when items load higher on their hypothesized constructs than on other constructs and when the square root of a construct’s AVE is larger than its correlations with other constructs (Gefen & Straub, 2005). All items load higher on their constructs than on other constructs, and the square root of each construct’s AVE is higher than its correlations with other constructs.

5.2. Results

We first performed paired t-test to evaluate Hypothesis 1 concerning the effectiveness of the virtual world-based information security learning on learning outcomes. As shown in Table 3, the results indicated significant improvement in both information security knowledge (p < :01) and information security self-efficacy (p < :05) between pre- and post-test measures. This suggests that the virtual world-based learning environment that integrates unique virtual world features with the self-regulatory learning interventions enhances information security learning.

새창으로 보기

Table 3.

Second Life Security Learning Module: Paired t-test of Learning Outcomes

To better understand the psychological learning process through which learning occurs, we performed PLS analysis to evaluate learner traits on learning outcomes. In particular, we performed a bootstrap analysis with 500 subsamples using PLS Graph 3.0 to estimate the path coefficients and their significance. Figure 6 presents the path coefficients and the explained variances.

새창으로 보기

Fig. 6

Results of the influence of psychological learning variables on learning outcomes

The significance levels are: * p < :10, ** p < :05, *** p < :01

The results support all the hypothesis except H2a - Learning goal orientation is positively associated with perceived enjoyment. Computer learning self-efficacy is positively related to perceived enjoyment ( = 0:49, p < :01), supporting H2b. Perceived enjoyment is positively related to information security performance ( = 0:37, p < :01) and satisfaction ( = 0:79, p < :01), supporting H3a and H3c. Overall, computer learning self-efficacy predicts perceived enjoyment (R² = 0:27). Perceived enjoyment predicts information security performance (R² = 0:35) and satisfaction (R² = 0:63). These results lead us to conclude that learner traits have an important influence on learning outcomes. In other words, learners who have higher self-efficacy with the virtual world learning environment are likely to have enjoyment experience during the learning process, which in turn enhances learning and overall satisfaction.