The Pedagogical Implications of Implementing New Technologies to Enhance Student Engagement and Learning Outcomes

doi:10.4236/ce.2014.52017

Paper Menu >>

Journal Menu >>

Creat ive Educati on

2014. Vo l.5, No.2, 104-113

Published Online February 2014 in SciRes (http://www.scirp.org/journal/ce) ht tp://dx.d oi.org/10.4236/ce.2014.520 17

The Pedagogical Implications of Implementing New Technologies

to Enhance Student Engagement and Learni ng Outcomes

Alison Sammel, Ka tie Weir, Christopher Klopper

School of Education and Professional Studies, Griffith University, Queensland, Australia

Email: [email protected]

Received Septem ber 20 th, 201 3; revi sed Octobe r 20th, 2013; ac cepted October 2 7th, 2013

Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the

original work is properly cited. In accordance of the Creative Commons Attribution License all Copyrights ©

2014 are guarded by law and by SCIRP as a guardian.

This paper reports the findings of research conducted by three teacher educators about the effects on

teac hing and lear ning from imp lementing a variety of di gital tec hnologies i n their undergr aduate c ourses.

The aim of t his study was to assess t he degree to whic h certai n universit y sup ported digital tec hnologies

assi sted in pr omoti ng student e ngagement a nd part icip ation in col laborati ve learning. The data are based

on the s emester long exper iences o f the thr ee lectur ers a nd their s tudents . From this data emer ged an ho-

listic picture tha t highli ghts which of the i mp lemented digit al technologi es constrains or enables parti cular

pedagogical aspects such as communication of course requirements; student engagement, meaningful

formative feedback; and deep connections between course elements. This picture assisted the authors in

generating a matrix f or implementing certai n digital technologies that cater f or diverse l earning styles, and

diversely experienc e an i nteres t in us ing tec hnolo gy. The t heoreti ca l framewor k for buildi ng the mat rix is

based on Collins, Brown and Newman’s (1990) Cognitive Apprenticeship Model. It is also underpinned

by the sugges tions tha t as “teachers” we too often over look whet her or not our st udents ha ve the requi sit e

skills to engage with technologies because of tacit assumptions about how this generation of students

wants to learn. Likewise, the same can be said of those who provide professional development sessions

for staff who are learning how to use new technologies and who often appear to make similar assump-

tions.

Key words: Digital Technologies; Online Education; Pre-Servic e Teacher E ducation

Introduction

Walking into any university course in Australia, or indeed

around the world, you will find students who have different

reasons for attending, different levels of motivation, different

attitudes and expectations towards their learning, and different

responses to technology. Understanding these differences not

only allows teaching staff to better cater to the diverse learning

needs of their students, but can also increase student engage-

ment, satisfaction and retention (Felder & Brent, 2005). Sey-

mour and Hewitt (1997) showed that many of the students who

drop out of university courses or programs do so not because

they are th e weaker stud ents, but because they are n ot engaged

with their learning and feel dissatisfaction with their education-

al instruction. To advance the reputation of the university, and

to increase student retention, efforts to increase student en-

gagement with learning materials and improving student satis-

faction with courses have been an important goal within uni-

versities for some time (Moosmayer & Siems, 2012). By re-

flecting on university courses that students find successful,

patterns endorsed by universities emerge. Patterns such as

creating supportive places where students can easily and effec-

tively interact with one another, the university teaching staff

and the course materials. In these successful courses, high le-

vels of student engagement are achieved. Students who are

engaged per ceive that they learn more, and become more satis-

fied with the learning experience (Arbaugh, 2008; Boston et al.,

2009; Garrison & Arbaugh, 2007; Garrison & Cleveland-Innes,

2005; Richardson & Swan, 2003; Rubin et al., 2013; Swan,

2002). Research has found strong links between successful

courses, effective use of technology and student achievement

and contentment (Liaw, 2008; Levy, 2008). Specifically, tech-

nology used to support university courses can directly affect

student satisfaction and success through promoting ease of

interaction and understanding of course materials, increase

facilitation between students and faculty, improve feedback,

and encourage interactive and independent learning (Selim,

2007). For this reason, this study will examine how three aca-

demic staff members and their students assess certain digital

technologies in the promotion of student engagement, and staff

and student satisfaction.

Theoretica l Back grou nd : Task-Technology Fit

To gain a greater understanding of the factors that influence

technology use, student engagement, and staff and student sa-

tisfaction with technology, it is helpful to explore task-tech-

nology fit theory. Goodhue and Thompson (1995) suggest that

OPEN ACCE SS

104

A. SAMMEL ET AL.

in order to have a success outcome with technology usage, the

users need to distinguish the goals for task and the fit between

the task and the technology. The basic argument of this theory

is that for a technology to have a positive impact on outcomes

and satisfaction, the technology must fit with the tasks it is

supposed to support, and the understandings and needs of those

who use it (Rubin et al., 2013). The better the task–technology

fit, the more it is anticipated that there will be positive conse-

quences from using the technology. In the context of using

various digital technologies within the three university courses

being investigated, the academic staff hoped the anticipated

consequences included facilitating a clearer understanding of

course objectives and assessment items, and promoting a dee-

per, more engaging and satisfying learning experience. As such,

task-technology fit refers to the ability of the various digital

technologies to support students in the range of learning activi-

ties they engage in, whilst accommodating the variety of stu-

dent abilities. These activities include promoting ease and

comfort communicating with other students and the academic

staff, accessing and promoting a deeper understanding of

learning materials and encouraging interaction and discussions

in lectures and tutorials. McGill & Klobas (2009) advocate that

by increasing awareness and understanding of the relationship

between the technology and the needs and goals of its user(s),

task-technology fit will have a positive influence on satisfac-

tion.

This theory suggests that for a digital technology to achieve a

certain mission, in this case increase student engagement, the

technology must first meet the needs of the user. Therefore, the

needs and understandings of the user must be explored. This

includes attitude and beliefs towards technology use, social

norms around the use of technology, prior comfort with tech-

nology use, experiences with other form of technology, and any

facilitating conditions. Further, technology-task fit also neces-

sitates that there is an understanding of functionality and cha-

racteristics of the technology. By addressing both the needs and

goals of the users, and the features of the technology, this study

explores the factors that influence technology-task fit such as

the perceived usefulness of the technology, communication

quality and changes in understanding knowledge, as well as

staff and student engagement, self-efficacy, computer literacy

and situational understandings. All these factors have been

shown to influence both the use of technology and their per-

formance impacts.

Research Design and Methodology

Design-based research (Brown, 1992; Collins, 1992) is an

emerging paradigm for the study of learning in contexts

through the systematic design and study of instructional strate-

gies and tools. The Design-Based Research Collective (2003)

argues th at design-based research can assi st in the creati on and

extension of knowledge “about developing, enacting, and sus-

taining innovative learning environments” (p. 5). There is no

single design-based research method, but rather an explicit

overarching concern for using methods that link processes of

enactment to outcomes. This overarching concern is seen to

have the po ten tial to gen erate kn owledg e th at direct ly appl ies t o

educational practice. “The value of attending to context is not

simple that it produces a better understanding of an intervention,

but also that it can lead to improved theoretical accounts of

teachin g and lear ning” (Th e Desi gn-B ased R esearch Col lecti ve,

2003: p. 7). This approach in which the context and interven-

tion are problematized sets this research method apart from

traditional evaluation where the intervention is evaluated

against a set of pre-determined standards (Worthenm, Sanders,

& Fitzpatrick, 1996). The intention of design-based research in

educational settings is to generate models of successful innova-

tion rather than particular artifacts or programs (Brown &

Cam pione , 19 96) .

The Procedure

The study was conducted in Semester one, 2012 at a large

multi-campus university in Queensland. Three academic staff

members teaching Science, Assessment and Communication

within a School of Education, reflected on the aim, goals and

practices of their instruction to understand what they would like

to improve upon. Increasing student engagement in lectures,

within the online course community and with the course con-

tent emerged as the main theme. The three academics worked

with their blended learning support team to investigate the dif-

ferent university supported digital technologies that would

promote their agenda. Each academic choose a few digital

technologies to learn and implement in their Semester one

course. During and after the implementation semester, each of

the academics wrote a case study explaining their experiences

and findings. Portions of these case studies are presented in the

data secti on.

There were a 204 students enrolled in the first year commu-

nication course, 151 in the third year science course and 160

students in the forth year assessment course. This gave a total

of 515 students who were asked to participate in this study. The

students were asked to complete an online survey at the start of

semester one. An academic who was not teaching into that

specific course gave each cohort of students instructions about

how to complete the survey and its importance. Students were

told that it was voluntary and anonymous. The online survey

asked questions about students’ previous experience with tech-

nologies in “learning” (etc. classrooms, lectures, tutorials) situ-

ations. They were also asked questions about their expectation

of face-to-face learning and online means of learning, their

expectations of technology and how they prefer to learn.

Twe nty-seven students participated in the online survey; the

result s are presented in the data section.

To further explore student how the students engaged with the

digital technologies offered in their courses, three focus group

interviews were conducted. Each group had around 25 - 30

participating students. The students were asked open-ended

questions about the what digital technologies they used in their

course; which ones they preferred and why; their motivations,

expectatio ns an d experi ences using t he techno logy; whet her th e

technology improved their learning outcomes; whether it en-

gaged th em; chal lenges and issu ed encountered an d if they had

any specific feedback about the digital technologies utilized in

their course. These focus group interviews were taped and then

transcribed. The results of these interviews are highlighted in

the data section.

Exploring the Task and the Technology: Three

Case Studies

The Staff Experience

Each staff member on the research team produced a case

OPEN ACCE SS 1 05

A. SAMMEL ET AL.

study that recorded their experience with implementing new

digital technologies with a goal to enhance student engagement

and learning in their respective courses. These are presented

below with the associated di scipline iden tified in bracket s . Each

case stud y was an alysed for any recurri ng or significan t themes

that were categorised under two subheadings: “Professional

Development” (improving their technological skill level) and

“Course Implementation” (the outcomes of implementing digi-

tal technologies in their respective courses). The findings from

these analyses are summarised under these same headings in

the right hand column of the Data Summary Table at the end of

this section.

Staff Case Studies

1) Lecturer 1 (Assessing Primary Students’ Learning)

This particular course was chosen to be put under the spot-

light of th is study becau se of the low scores and negati ve feed-

back in the student experience survey data the previous year.

This situation was seen as an opportunity to implement new

technologies for enhancing the delivery of the curriculum and

for improving communication with students with a goal of en-

hancin g the qu ality of their learni ng exper ience. Apart from the

major aspects of the course that were revised such as the as-

sessment items, the technologies I decided to implement this

semester included the following: an online assessment submis-

sion process (Safe Assign) for greater efficiency and for its

capacity to provide better feedback; Lecture Capture to record

the lectures that focused on the two assessment tasks; Personal

Lecture Capture (Echo360) to ensure consistent delivery of key

course mes sages; and a co llabor ative onli ne classroo m (Wimba)

to enhance communication. Other digital technologies in place

such as the course website and the online discussion board were

also revamped to improve curriculum delivery and course

communication.

Once I had selected these new digital technologies I under-

took professional development provided by our Technology

Advisor. I also undertook further learning about the technolo-

gies I was already familiar with e.g., Safe Assign and the Dis-

cussion Board where my learning focused on improving their

functionality. Wimba was the technology I was least familiar

with after experiencing immense frustration trying to access

and utilise a Wimba classroom successfully from off-campus I

rejected this technology out of fear that students would expe-

rience the same access and communication issues. I was simi-

larly unfamiliar with Personal Lecture Capture (Echo360) and

this turned out to be another option I rejected because of time

constraints and the lack of resources for preparing quality re-

cordings, both technical and human.

I utilized Lecture Capture selectively to record the lecture

content directly related to the assessment tasks. Student atten-

dance dwindled to less than half the enrolled number by the

final lecture. The tracking dat a from the university’s blackbo ard

site in dicate that 80 % of students accessed Lecture Cap ture.

Using Safe Assign for assignment submission created some

unforeseen challenges for online marking. One of the reasons

this technology was implemented was for its plagiarism analyt-

ics as the assessment task is recycled each year. Safe Assign

allows only one file to be submitted which created issues for

students who had saved their task in different formats and in

different documents (despite explicit instructions requesting

their submission as one Word document). For the course teach-

ing staff this method of providing feedback to students “in situ”

using the “Track Changes” function was preferred over tradi-

tional handwritten comments on paper-based assignments but if

students saved their files in other formats they were prevented

from providing this sort of targeted and feedback on their work.

How stud ents named their files was pro blematic and it was ti me

consuming to remediate this by re-naming each file. The

download function in Safe Assign did not work off-campus,

which was another glitch in the system especially when the

sessional staff were already under confident with using this

online assessment system. More problems emerged when as-

signments were returned, as it was not clear to students how to

access their marked work. None of these sorts of pragmatic

issues were addressed in the PD sessions I attended but now I

am aware o f them they can be ea sily remediated with approp ri-

ate and explicit instruction for staff and students.

The Discussion Board was utilised more around the due

dates for ass essment and the focus of the discussions was pure-

ly on details regarding the asses sment tasks.

2) Lecturer 2 (Primary Science Educat ion)

The course I chose to investigate was a primary science

course. My goal was to implement a range of digital technolo-

gies that I had not used before and to investigate what I found

to be practical and useful, and to explore how the students re-

sponded to each of them. This course was the second core

science course the Bachelor of Education (Primary) students

needed to take. It runs in the first semester of their third of four

years. There were 151 students ranging in age from school

leavers to mature age students. Attendance at the lectures was

nearly 100% at the start but on average became around 70% of

the student cohort.

The first digital technology initiative I implemented was

Echo360 which I used to record detailed explanations of the

assessment tasks. I went through both the assessment outline

and the corresponding marking rubric. I wanted to ensure that

my students had a consistent and clear message around their

assessment items. Each recording was around 11 minutes long

and consisted of specific details and helpful tips and advice. I

used prior student examples displayed on the computer screen

to address major aspects of the assessment task. The students

commented that they appreciated this information; especially

the verbal explanations using prior students work to demon-

strate my point. They said this assisted them in undertaking the

assessment task. However, in response to further questioning,

the majority of students tended to focus on the practical exam-

ples provided rather than also seeking to understanding how

their work would be evaluated using the rubric. This lead to

students forgetting to do certain small tasks, like cor rectly refe-

rencing. The student tended to not access the assessment infor-

mation at the start of the course but when i t was needed . There

were quite a few students who came to me for assessment ad-

vice and when asked if they had listened to the specific

Echo360 recordings under assessment at the Learning at Grif-

fith site, said they had forgotten about it. These were students

who did attend the lectures but usually arrived late, hence

missing the verbal reminder to access these recordings, howev-

er a reminder was usually written on the white board in the

lectures. I would have liked the tutors to have used and pro-

moted using the Echo360 recordings more. This could have

been done by listening to it themselves or directing students to

listen to it when they had assessment questions, or displaying it

on the whiteboard when answering specific questions. Some of

the four tutors gave assessment information that was different

to what was in my recordings, leading to confusion amongst the

OPEN ACCE SS

106

A. SAMMEL ET AL.

students. As this was a new tool to this course, I found tutors

and students were not in the habit of listening to the assessment

recording, and so did not use it as the supportive tool that it was

intended to be.

Lecture Capture was also implemented in this course to

record each of my nine, 2-hour lectures which were then up-

loaded into the relevant weekly folder on the course Blackboard

site within 24 hours of each lect ur e. An an no uncement was also

circulat ed to the students when this lecture material was availa-

ble. The students could access the lecture content through Lec-

ture Capture whenever they chose to. Statistical tracking

showed the majority of students listened to all nine recordings,

but not within a week of the given lecture, rather around the

week before their end of the course exam. When asked, the

students who regularly attended lectures explained that it was

only if they had missed the lecture or did not understand a point,

would they listen to the recorded 2-hour lecture within a week

of the lect ure. The majorit y said that this tool was an important

part of their exam study process as they would re-listen to spe-

cific lectures that they found the content more difficult to study.

Rarely did the students listen to all recorded lectures, but nearly

each student explained they would listen to 2 or more record-

ings out of the nine. There was no specific pattern to their re-

listening, however, the chemistry and physics lectures were

mostly replayed.

The formal university course and teaching evaluations al-

lowed only for comments arou nd the Lecture C apture asp ect of

the course. It showed there was a mixed reaction to this tool.

When asked i f the reco rd ed lectu res assi sted th eir l earn in g, on a

scale of low, medium, high and neutral, 30% of the student

responded with the “high”, while 40% replied with a “medium”

response and the final 30% identified as “neutral ”. There were

no written comments relating to any of the digital technology

implementations. These mixed reviews may have been due to

the fact that students could not specifically skip to certain

points in the recording and did not want to re-listen to the

whole of the 2-hour lecture. New Lecture Capture programs

now allow for movement to specific places in the recording,

and statistical tracking in my more recent courses have shown

students are accessing Lecture Capture more frequently.

Another reason for the mixed reviews may involve my interac-

tive lecturing method, which involves many student-centered

activities that Lecture Capture cannot fully record. The visual

and sound quality for these student based experiments and ac-

tivities would not have been at the same high level as when I

was talkin g to th e specific Po werP oi nt images on the screen . To

this end, the Lecture Capture capabilities only really support a

“chalk and talk” style of lecture, and not the process of interac-

tive demonstrations, activities or discussions that allow for the

joint development of knowledge. To this end, Lecture Capture

does not translate well for my lectures. However, students do

have the po tential to acces s what infor mation is r ecorded at any

time.

The third digital technology initiative I implemented was

JPoll. JPoll supported my desire for more interactive pedagogy

as it enables students to anonymously articulate their prior

knowledge o f a scien ce to p ic and displ ays the co llect ive resu lts .

These results initiate discussions around science content and

pedagogical strategies related to teaching that science topic. I

used this technology to replace a more general PowerPoint

discussion of key alternative conceptions primary aged students

have about science topics. JPoll allowed me to interactively

investigated science misconceptions. To assist students’ use of

the tool I developed a slide to explain what JPoll was, how to

connect to the university’s wireless Internet, and then connect

to the JPoll site. Any student with a smart phone or mobile

computer device could access JPoll. Those students who did

not have a computer or a smart phone completed the task in

pairs or groups with students who did have this technology

(about 50% of students). This organisation process involved

developing 15 true or false questions before the lecture. The

answers the students provided to the 15 questions were ano-

nymous, thus no individual student answer could be identified.

Once answers are entered into a mobile device, the statistical

information instantaneously displays on the front screen. This

instant, visual statistical information dramatically highlights

any science inconsistencies student have without revealing

individual responses. This resulted increased engagement,

promoting deeper discussions compared with previous years. I

displayed the information in a bar graph of either true or false,

so everyone could quickly see what percentage of the students

thought the question to be either true or false. From this infor-

mation a large group discussion was started. When asked about

the experience of using this technology to promote engagement

and lear nin g, th e stud ent s were ful l o f praise for ever y aspect o f

this process. No one complained that it was not individually

accessible, as they did not have the smart device available to

them personally. I assumed this would be more of a problem. I

believe JPoll worked well, however instead of only having

access to questions formulated before the lecture, it would be

advantageous to be able to type in questions during the lecture.

Being limited to only using re-registered questions I found a bit

restrict ive for my preferred ped agogical st yle.

Implementing these three learning technologies was reward-

ing for enhancing my science pedagogy and allowed me to

model using these technologies for pre-service teachers.

3) Lecturer 3 (Primary Mu sic Education)

The course was implemented according to the University’s

mandatory three hours per week (one hour lecture and two hour

tutorial) model of delivery, with a further seven hours expecta-

tion allocated for course related self-study. The lectures were

designed to present the theoretical content through transmission

mode while the tutorials provided avenues for interactive prac-

tical application of the l ecture material. The Blackb oard course

site was identified as the platform best suited to harness the

potential of a range of instructional delivery technologies to

augment best features of face-to-face interaction. On the site

course content was offered from the beginning of the semester

in weekly folders consisting of lecture notes; lecture Power-

Point slides, and recommended readings. Each lecture was

captured weekly using Lecture Capture operating system and

uploaded to the course site within ten minutes of the lecture

concluding. This allowed students to repeatedly access the lec-

ture asynchronously for further engagement, reflection, and

clarification. It also afforded absent students access to the lec-

ture presentation.

I used an assessment Vodcast provided explanations of the

text-b ased assessment task which p roved valuable i n offerin g a

consistent explanation of assessment expectations, which both

students and sessional teaching staff could refer to throughout

the semester .

The virtual learning environment, Wimba classroom, was

opened for synchronous interaction five times during the seme-

ster of 50 minutes duration. This occurred after an overview of

OPEN ACCE SS 1 07

A. SAMMEL ET AL.

operation was presented during the tutorial of week one. Fur-

ther text-based support materials were provided on Blackboard

site to guide the student through installing and running the

Wimba Wizard on their personal computers. The intention of

the Wimba classroom was to assist students to undertake the

written task successfully through support offered via alternate

ways. Sessions focused on academic writing skills, unpacking

the task, making sense of the prescribed readings and writing

the essay. The centrality of critical thinking underscored the

sessions. While this virtual learning environment offered syn-

chronous interact ion it was decided to archive al l the sessions to

allow asynchronous membership. This proved valuable for

students who were unable to actively participate during syn-

chronous availability.

Other resources uploaded to the Blackboard site include ex-

emplars of learning materials for music classrooms, suggested

further readings, tips and hints for assessment tasks, and shared

findings where students could upload resources suitable for the

teaching of music education in the primary school. As part of

the written assessment task, students were required to evaluate

one of these resources. To this end, www.artsmmadd. com was

provided as an open source conten t webpage for use.

Remote assignment submission was selected to reinforce the

use of ICTs within this course. This submission mechanism

allowed students to submit their assignment remotely from an

off-campus location.

A representation of access by source is presented in the dia-

gram below. It does not argue for any one source to be better or

superior to another, but rather provides the backdrop to discuss

the potential of each source to support the achievement of the

learning outcomes through scaffold learning, as shown in Fig-

ure 1.

Not surprisingly the Course Content received the highest

access. As l ectures wer e the pri ncipal mode of con tent del ivery

the 84% access of source suggests the importance and value

that students place on having access to the course content. In

the attempt to revamp the instructional delivery technologies

(IDT) with this course to cater for the ‘new’ learner by inte-

grating IDTs into the classroom, I expected a much higher

access percentage associated with the Lecture Capture and

Wimba sou rces. I can r atio n alize t he l ow acces s rate o f 55 % for

Lecture Capture through the high attendance rate of lectures,

but initially I could not fathom the low access rate of 10.9% for

the Wimba session. This required a closer examination.

Few students could grasp how to access the Wimba virtual

learning environment. Many spoke of the difficulties expe-

rienced trying to access Wimba from an off-campus location. I

recall during the first week of the semester having three stu-

dents arrive at my office and ask: “Where is the Wimba class-

room? You have not advertised the location.” I naively thought

that this was just the sentiment of a few students, but learned

that I was wrong. My assumption and expectation of a genera-

tion whom have been exposed since birth to the Internet and

hypertext to think and process information differently from

previous generations was inaccu rate. Advances in teaching and

learning come through evolution not revolution. I realized that

the students required scaffold learning to access and use the

learning technologies effectively to enhance their learning out-

comes in this course.

Student E xper ience

The following results are drawn from an analysis of three

data sources. The first is from an online survey that students

voluntarily completed at the beginning of the course. The sur-

vey contained a mix of closed and open questions that asked for

students’ opinions and comments on their prior learning expe-

riences at un iversi ty as well as their exp ectati on s ab ou t learning

in higher education. The second data source was a series of

focus group interviews conducted at the end of the course.

During these sessions students were asked a series of open-

ended questions designed to gather feedback on their expe-

rience with the digital tech nolo gies employed in each cou rse. A

thematic analysis of the data was conducted and these results

are described in detail below and summarised in the first two

columns of the data summary table at the end of this section.

The third and final data source was t he for mal uni versit y cou rse

and teach in g evalu ation surve ys (o r stud ent experi ence s urveys)

to which the authors added an open question that asked students

to provide feedback on the employment of digital technologies

in each course. The response to these surveys was poor and

nature of the responses was of limited use and so it is omitted

from the Data Summary Table that gives an overview of the

findings from the combined data sources.

1) Survey data: Students Prior Learning Experiences

The quantitative survey data regarding students’ exp er i ences

Figure 1.

Repre sentation of acces s by source.

OPEN ACCE SS

108

A. SAMMEL ET AL.

with online learning indicates that a significant majority (>75%)

of the students surveyed had previous experience with access-

ing resources and other learning materials from the internet,

viewing PowerPoint presentations, participating in discussion

boards. A slightly smaller majority (65%) had experienced

some form of assessment online, while less th an half had expe-

rience with Wikis or audio-visual lecture-capture. Only two

students provided information about “other” online learning

experiences they had with Wimba and with Library-referenc-

ing.

When students spoke of their lecture experiences some key

themes emerged from the survey data. The first theme de-

scribed lectures in terms of a traditional mode of information

delivery that is teacher-centered. This is manifest in phrases

such as “a lot of chalk and talk”, being “talked at”, “listening”,

“note-taking” and, “little interaction”. A number of students

mentioned PowerPoint presentations as a common AV aid for

lectures that focused on theory and whose goal was to present

“as much information as possible”. Some students did ascribe

positive attributes to lectures/lecturers claiming they enjoyed

listening to an expert who could provide alternative perspec-

tives to the recommended readings and these students tended to

seek motivation to learn more and get some direction to attend

to further study.

When students explain how they expect learning to occur in

lectures and tutorials a significant number stated that lecturers

should be an “expert ” and see their role as providing the es-

sence of each week’s lecture topic in an interesting and engag-

ing manner. The y generall y desi r e more in teract io n between t he

speaker and the audience. They expect the lecturer to have on-

going communication with tutorial staff so the delivery of key

curriculum messages is consistent. Students clearly distinguish

tutorials from lectures as they describe more discussion and

interaction with tutors and peers taking place in tutorials. The

expected role of the tutor is to create a supportive learning en-

vironment where students can ask questions and bounce ideas

around to clarify their interpretation and gain a deeper under-

standing of the course content. Students also mentioned tuto-

rials as places to network with other students. They expect their

tutors to have deep knowledge of the subject matter.

When asked more specific questions about the use of tech-

nology in lectures and tutorials students stressed the importance

of teaching staff having the confidence and know-how to use

technology effectively and in appropriate/relevant ways. They

also identified audio-visual aids in lectures as assisting in

meaning-making by complimenting or elaborating upon any

written or print-based text. They would prefer a variety of dif-

ferent technologies be employed as they also identified their

use in catering for diverse learning styles. Apart fro m the Dis-

cussion Board, few students mentioned using other technology

to communicate with staff whilst social media (such as Face-

book) was the preferred contact with other students to discuss

more general aspects about the course delivery and content.

Nearly all students commented that quality technology was

vital for them to access resources and information. Other stu-

dents made indirect reference to wanting easy access to infor-

mation and resources through the organisation of the course

content.

2) Focus Group Interviews: Students Learning Experiences

Post Implementation of New learning Technologies

Three focus group interviews were conducted with students

and the responses from both groups were similarly positive

about the employment of digital technologies in the courses

involved in the study. However one group claimed a persistent

preference for traditional modes of teaching and learning

through personal communication in face-to-face contexts with

print-based materials. All students interviewed agreed that the

technology employed did cater for different learning styles.

During one focus group discussion many of the students de-

clared their only prior experience with online engagement had

been through social media platforms. These platforms are dy-

namic and interactive and hold appeal for students. These same

students expressed that the design of learning technologies

promoted at Griffith University to be linear, static, and not par-

ticularly interactive.

The most significant finding from both group interviews is a

preference for three types of ICTs-Lecture Capture, Echo360

and the Discussion Board due to their ease of access and use,

and the assistance to learning they provide. There were differ-

ent reasons cited for using each of these technologies and the

number o f ti mes the y wer e accessed also differed. For instance

Lecture Cap ture was claimed to beneficial for clari fying content

interpretation and how ideas linked together although students

tended to access this only if they missed the lecture or if the

lecture focused on the upcoming assessment. The Discussion

Board was predominantly used to communicate with their peers

and teach ing staff to pursue alternat ive perspectives and clarify

instructions around assessment. Students claimed they preferred

the Discussion Board for asking for asking questions because it

allowed anonymity in the sense that they were “faceless” in a

cohort where students were not that familiar with each other

(compared with a tutorial where, over time, students became

more familiar with others in the cohort). Students reported that

they also accessed an unofficial Facebook site for the course

where they could communicate between themselves about as-

pects o f the course and help each other to clarify things in their

own language.

All stud ents claimed regul ar and easy access to th e universi-

ty’s learning platform. They also all claimed preference for a

course s ites th at are well organise d in t erms course co nten t. For

example th ey benefited from course content being arranged into

weekly folders that contained the required weekly readings plus

a selection of optional learning resources. Students indicated

that this arrangement was appreciated as saved them a lot of

library and database search time with one student commenting

that this layout enabled him to “pinpoint the learning outcomes

for each week and tie all of the resources together”. However

all students agreed that they still needed the weekly tutorial

sessions to further tie the content together and align it with the

assessment requirements. Students claimed they were more

motivated to engage with these resources because the lecturer

selected t hem an d therefore th ey assu med these resou rces were

directly aligned with the course learning objectives.

The online assessment processes were well received with few

students reporting difficulties uploading their assignments.

There was a generally positive response to the SafeAssign re-

porting feedback as this highlights where they could improve

their referencing and it meant they felt “more accountable” for

their citation skills. It also meant they could submit a draft at-

tempt and have the time (24 hours) to reflect and revise before

they could re-upload their final document. The interview data

revealed that students appreciated receiving assignment feed-

back from their tutor/marker online as it was easier to read

(when the Track Changes function was utilised land further-

OPEN ACCE SS 1 09

A. SAMMEL ET AL.

more, it was retrievable to reflect on when they were complet-

ing other assignment in the future.

3) The SE C and SET feedback data

The limited data received from one open question in these

surveys supports the focus group interview data with students

claiming the technology employed assisted their learning, spe-

cifically mentioning Lecture Capture and the organisation of

course content material on the course websites. A number of

students suggested all lectures should be recorded but this

strategy should not replace face-to-face lectures. For the data

summar y pleas e see Table 1.

Implications for the School of Education and

Professional Studies and Initial

Teacher Education

The data obtained from the students and staff who partici-

pated in this research project has pedagogical implications for

employing digital learning technologies in higher education.

We belie ve t he findings are not limited to teacher educatio n and

can be generalized for consideration in other higher education

contexts. Therefore the conclusions that we draw from our

research can assist other most lecturers who are considering

using any of these technologies targeted in our research to en-

hance their course learning outcomes. Conducting the initial

survey on student experiences and expectations with learning in

higher education was a prudent move as the findings from this

data source indicate that we cannot assume that students are

motivated by, or know how to use or access the digital tech-

nologies we intend to employ in our courses. The fact that a

number of students still have a preference for face-to-face

communication with teaching staff suggests that we need to be

more explicit about our rationale for employing digital tech-

nologies and how they will assist and enhance student learning.

This finding further highlights the need for staff to be mindful

of the diverse learning styles amongst our student cohorts and

to demonstrate how a variety of digital and traditional peda-

gogical str ategies can cat er for th is diversity.

Our findings show that consideration of the diversity of prior

experience and knowledge about digital learning technologies

amongst the teaching staff is also paramount if they are being

used to enhance course learning outcomes. The data from the

case studies show that even though the staff were motivated to

undertake relevant p rofessional develo pment for each new dig-

ital technology, the pedagogy experienced in these sessions did

not necessarily assist in employing the technology in a range of

learning contexts. The data tells us that direct instruction in a

one-hour timeslot about how to use the technology was not

enough for staff to understand the complexities and challenges

that can arise once students engage with them. A more helpful

approach would be to include some discussions about the sorts

of problems that can and have emerged when employing new

technologies and how to overcome these. The findings from the

staff case studies also point to how essential it is to determine

the skill level of sessional staff employed to tutor students and

to make available timely professional development for these

people. If this is not done our results suggest that attempts to

enhance course learning outcomes will be undermined by in-

consistent or incompetent use of the technology by our session-

al staff. Students expect all teaching staff to be competent and

efficient users of technology and yet the burden of up skilling

in this area is on individual staff members who may not neces-

sarily be rewarded (through workload considerations) or remu-

nerated (in the case of sessional staff) for doing so.

Another significant finding that relates to those outlined

above is the need to explicitly teach students how to use digital

technologies to enhance their learning in addition to explaining

why and for what purpose they are being utilised. While all

students were comfortable with using the Blackboard learning

platform to access course content, there were varying degrees

of access and efficient use of Lecture Capture, the Discussion

Board and online assessment handling. While we acknowledge

that students have to access these tools in order to complete

their courses successfully, our research provides evidence that

explicitly teaching students the learning benefits of these tools

and explaining the different ways they can be utilised should

ensure better course delivery and outcomes. For example, it

was revealing for staff to discover that students use assignment

feedback provided online for assisting them to complete future

assignments in the same and other courses. This example alone

demonstrates the importance in finding out the variety of ways

this and other digital learning tools are utilised by our end-

users.

Wimba proved to be the most problematic for students to

access and for one staff member the frustration experienced

with this learning tool meant it was scrapped before the course

commenced. Another concern for staff that emerged from this

research was student preference for social media sites such as

Twitter and Facebook to have “sideline” discussions about

specific cou rses and staff membe rs. Rath er than seei ng this as a

threat that might undermine our course communication via say,

the Discussion Board, we interpret this preference as something

which staff can capitalize on. For instance one staff member

circumvented any problems this may cause by setting up a

course Facebook site and monitoring the discussions thus pre-

venting the misinformation that can circulate between students

on these s ites.

The research findings outlined above give us some insight

into how students and staff experienced the employment of

some digital learning techn ologies in teach er education courses

at Griffith University. Even if these results are nuanced and

varied depending on staff and student skill level, experience

and motivation, we believe they can be used to assist other

university staff intending to enhance their course learning out-

comes through utilising digital learning technologies. However

in its current format our research findings are not easily trans-

lated into assisting associated pedagogical strategies in other

higher learning contexts. This understanding led us to search

for a model of learning into which we could input our findings

and enable them to be generalized for broader use. The most

appropriate model uncovered for teaching with digital technol-

ogies that caters for diverse learning styles and experience in

using technologies is the Cognitive Apprenticeship Model de-

veloped by Collins, Brown and Newman (1990). Based in con-

structivist approaches to human learning and situated learning

theory (Brown, Collins, & Duguid, 1989), the model explains

how skills are more readily acquired in authentic learning con-

texts and by communicating with experts and others. The model

is also underpinned by the notion that teachers (as “experts”)

often fail to take into account implicit processes involved in

carrying our complex skills when they are teaching novices. In

other words, we too often overlook whether or not our students

have the requisite skills to engage with technologies that we

presume to be relatively simple because we have already

OPEN ACCE SS

110

A. SAMMEL ET AL.

Table 1.

Data summary table.

Student Expectations

Student Experience

Staff Experience

In lectures and of the lecturer’s r ole

•

an “expe rt” whose role is to capture the

essence of the topic in an interesting

and engag ing m anner

•

students desire mor e interaction

between lecturer and the audience

In tutorials and the tutor’s role

•

creatinga supportive learning e

nvironme n t w he re stu dents ca n a sk

questions and bounce ideas around to

clarify their interpretation of the

course content in an effort to gain

a deeper understanding.

•

expect the ir tu tors to hav e deep

knowledge of the subject matter

•

key learning activity should

involve substantive conversations

between all participants

Of digital learning technologi es

•

teac h ing staf f need to be c o nfide nt

and u se t he techno l o gy in app ro priate

and rele vant ways

•

expect audio-visual aids in lectures to

assist meaning-making by

complimenting or e laborating

upon any written or print-base d r e sourc e

•

prefer a variety of different technologies

•

recognise their role in catering

for diverse learning styles

•

communicate wi t h sta ff and peers

through Discussi on Board

•

social media is the preferred contact

point with peers

•

quality technology is vital for

them to access resources

and information

With digital learnin g technologi es

•

positive about some digital technologies

but a significant numbe r claimed preference

for campus-based learning that

involves persona l communication

and some print-base d resources

•

the technologies employed did

cater for different le arning styles

•

a clea r preference for technologi es

with ease of access and assistance t o lea rning

•

Echo360 clarified expectations of

assessment tasks

•

Lecture Capture was beneficial

for clarifying how ideas linked t ogether

•

Disc u ssion bo a rd predomi nantly use d to

communicate wi t h pe e rs and te a ching

staff to pursue alternative perspectives

on ideas and clarify instructions around,

and requirements of the assessment tasks

•

comfortable with the anonymity of the

Discussion Board when as king questions

(described as “faceless” in a cohort

not all students are familiar with na mes)

•

significant number of students also

acces sed the “unoffici al” Faceb ook site

for courses where they could communicate

between themselves about aspects of

the course in their own language

•

org a ni sation of Blackboard site s is impo rtant

for tying the lecture content to the tutorial

activity and other learning resources. This

appears to differ widely across cou rses

but the re is a p refe rence for course

content to be well organised

•

positive about upl o ading required rea d in gs

and other relevant learning mate rials to

Blackboard. This saved time and

students’ assume direct alignment

with learning obje ctives

•

online assessment process well re ceived

especially in terms of submission on due dates

•

positive response to SafeAssign reports that

provide feedback on where students could

improve their referencing—made them fe el

“more accountable” for their citati on skills

•

appreciated receiving online feedback which

was easier to read and retrievable for reflecti on

when completing futur e assignments

Professional development

•

one-size-fits-all approach not appreciated;

needs to cater for diversity in expe rience and

confidence and teaching styles and contexts.

Appears to assume all staff have similar prior

knowledge and experience. Lack of critical

discussion around im plementati on challenges

•

frustration wit h a cc ess to so m e techno l o gies

led to rejection of taking risk

•

lack of support for quality pre-recor ding and

editing of lectures combined with an assumption

that all staff are comfortable with this

mode of de livery.

Course Implementation

•

the course content available from the

Blackboard site was accessed more

than any other learning resources/technology.

•

mixed response to Lecture Capture that

indicated less access with great er lecture

attendance (access ranged from 70% - 80%

of students in one course to less than 4% in another).

•

access to Lecture Capture was highest

within a week of assessment due dates

•

a downside of Lecture Capture is the lack

of editing—student loss of interest and attention

•

tendency for students to only access

info r matio n w hen need e d

•

primary focus of the Discussion B oard

posting around assessment

•

Echo360 used inconsistently with more

attention paid to instructions about

assessment tasks and less in explanations

about the marking criteria

•

JPoll implemented successfully for assessing

students’ prior kn owledge and using the

results as basis of substantive conversation s

about selected topic

•

concerns about students using social media e.g.,

Facebook for discussions about course and

teaching staff. An awareness that misinformation

about assessment ins tructions and othe r key

messa g es bei ng exchange d betwee n stude n ts

•

issues with SafeAssign were numerous but also

easily re solved with better communication

and training of sessional staff and students

•

Wimba was the most difficult technology

for students to access

•

need to assist sessi onal staff to engage with

digital learning technologies in more consistent way

mastered these skills. Likewise, the same can be said of those

who provide professional development sessions for staff who

are learning how to use new technologies and who often appear

to make similar assumptions.

The cognitive apprenticeship model involves a series of pe-

dagogical strategies that run from a highly teacher-centered

approach to enabling full student direction over the learning.

Runni ng parall el to th ese pedago gical p ro cesses are t hree st ages

of learning: from cognitive to associate to autonomous cogni-

tion. The model is illustrated in Table 2.

This diagram indicates the cognitive apprenticeship model to

be a linear series of teaching processes associat ed with t he three

stages of cognition/learning. However, our results indicate that

in reality both the teaching and learning involved in utilising

new technologies is not static and instead may shift between

processes and stages in a d ynamic way dependin g on the exp e-

rience, motivation and disposition of the teacher and the learner.

What makes the model useful is its ability to associate the most

appropriate instructional delivery technology with the cognitive

stage of the learner. In order for the model to be used success-

fully, the learner must have an opportunity to articulate where

they believe th ey are along the scale of cogni tion and so teach-

ers need to provide this opportunity, preferably at the com-

mencement of the course or before the introduction of the deli-

very technology. This can be done by simply determining prior

knowledge and experience with different learning tools and

monitoring use and access of the tools as the course progresses.

Based on our experience detailed in the results of this study,

we have nominated particular instructional delivery technolo-

gies which best suit the cognitive stage of the student. This

OPEN ACCE SS 1 11

A. SAMMEL ET AL.

nomination has formulated the instructional delivery technolo-

gy matrix and but this is not to suggest that this is representa-

tive of all possible available technologies. It is proposed that

the further develo pment of the matrix be organic depending on

learning context. Each instructional delivery technology in the

matrix has a hyperlink, which provides a detailed teachers

guide for use to enhance student learning (Please see Table 3).

Concl usion

This study examined the effects on teaching and learning for

three a cademi c sta ff memb ers an d th eir stud ents from integrating

of a variet y of digital technologies into their university teaching

and learning activities. Goodhue and Thompson (1995) suggest

that in order to have a successful outcome with technology

usage, the user(s) need to distinguish the goals for task and the

fit between the task and the technology. With this in mind, both

the staff and the students involved in this study were asked to

reflect on their goals and expectations around digital technolo-

gies, the degree to which the technology supported the task

asked of it, changes in student engagement within the university

course, and levels of satisfaction with technology. If technology

is going to have a positive impact on outcomes and satisfaction,

then the technology must first meet the needs of the user.

Therefore, this study focused on exploring the needs and un-

derstandings of the user(s) including attitude and beliefs to-

wards technology use, social norms around the use of technol-

ogy, prior comfort with technology use, experiences with other

form of technology, and any facilitating conditions.

Our findings showed diversity of assumptions, prior expe-

rience and knowledge about digital learning technologies

amongst the teaching staff and students. The data from the case

studies showed that even though the staff were willing to un-

dertake relevant professional development for each new digital

Table 2.

Three stages of learning from cognitive to associate to autonomous cognition.

The Cognitive Apprenticeship mode l applied to digital instructional technology usage matrix

Teacher direc te d

Student directed

Modelling

Coaching Scaffolding Articulation Reflection Exploration

Table 3.

Three stages of learning from cognitive to associate to autonomous cogn ition.

The Cognit iv e Apprentic eship mo d el applied to digital instr uctional tech nology usage ma t rix

Modelling Coachin g Scaffolding Ar ticulati on Reflection Exploration

Skype Facebook Blogs Virtual learning

environment: Collaborate Blogs P ersonal le arning

networks

Personal learning

networks Wikis Wikis Social networking

in Education Personal learnin g

networks Social networking

in Education

Google docs Skype Sk ype Personal learning ne tworks Social networking

in Education Google docs

Lecture capture Personal learning

networks Persona l learning

networks Digital portfolios Google docs Flipped classroom

Social networking

in Education Social networking

in Education Twitt er Twitter Twitter

Podcast Lecture capture Virtual learnin g

environment: Colla borate

Vodcast Virtua l learning

environment: Colla borate Podcast

Virtual learning

environment: Colla borate Podcast Twitter

Vodcast JPoll

The cognit ive stag e The asso ciativ e stage The autonomous stage

OPEN ACCE SS

112

A. SAMMEL ET AL.

technology, the pedagogy experienced in these sessions did not

necessarily assist in employing the technology in a range of

learning contexts. To offer more support to staff, professional

development would include discussions of problems that can

and have emerged when employing the specific digital tech-

nologies and how to overcome them. While it is acknowledged

that students have to access a variety of digital technologies in

order to complete their courses successfully, our research pro-

vides evidence that explicitly teaching students the benefits of

these technologies and explaining different ways that students

can utilise these tools for learning will ensure better course

delivery and outco mes.

Our results also indicate that in reality both the teaching and

learning involved in utilising new digital technologies are not

static and instead may shift between processes and stages in a

dynamic way depending on the experience, motivation and

disposition of the teacher and the learner. To help both univer-

sity teaching staff and students understand how to cater for the

diverse learning styles and experience in using digital technol-

ogies, a modified Cognitive Apprenticeship Model was devel-

oped. What makes the model useful is its ability to associate the

most appropriate instructional delivery technology with the

cognit ive stage o f the l earner . B y ad dres sin g both the n eeds and

goals of the users, and the features of the technology, this mod-

el speaks to the factors that influence technology-task fit. Pro-

moting awareness and understanding of the relationship be-

tween the technology and the needs and goals of its user(s) is

essential to increasing engagement and s atisfaction.

REFERENCES

Arbaugh, J. B. (2008). Does the community of inquiry framework

predict outcomes in online MBA courses? International Review of

Research in Open and Distance Learning, 9.

htt p://www.i r rodl.org/index. php/irrodl/artic le/ viewArticle/490 /1045

Boston, W., Diaz, S. R., Gibson, A. M., Ice, P., Richardson, J., & Swan,

K. (2009). An exploration of the relationship between indicators of

the Community of Inquiry framework and retention in online pro-

grams. Journal of Asynchronous Learning Networks, 13, 67-83.

Brown, A. L. (1992). Design experiments: Theoretical and methodo-

logical challenges in creating complex interventions in classroom

settings. Jour nal of th e L e arning Sciences, 2, 141-178.

http://dx.doi.org/10.1207/s15327809jls0202_2

Brown, A. L., & Campione, J. C. (1996). Psychological theory and the

design of innovative learning environments: On procedures, prin-

ciples, and systems. In R. Glaser (Ed.), Innovation s in learn ing: New

environm en t s f or educa tion (pp. 289-325). Mahwah, NJ: Erlbaum.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and

the culture of learning. Educational Researcher, 18, 32-41.

http://dx.doi.org/10.3102/0013 189X01800 1032

Collins, A. (1992). Toward a design science of education. In E. Scanlon,

& T. O’Shea (Eds.), New directions in educational technology (pp.

15-22). New York: Springer-Verlag.

http://dx.doi.org/10.1007/978 -3-642-77750-9_2

Collins, A., Brown, J. S., & Newman, S. E. (1990). Cognitive appren-

ticeship: Teaching the crafts of reading, writing, and mathematics. In

L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in

honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Lawrence Erl-

baum.

Felder, R. M. & Brent, R. (2005). Understanding student differences.

Journal of Engineer ing Education, 94, 57-72.

http://dx.doi.org/10.1002/j.2168-9830.20 05.tb00829.x

Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community

of inquiry framework: Rev iew, i ssues , and fu ture di recti ons. Internet

and Higher Educat ion, 10, 157-172.

http://dx.doi.org/10.1016/j.iheduc.2007.04.001

Garrison, D. R., & Cleveland-Innes, M. (2005). Facilitating cognitive

presence in online learning: Interaction is not enough. American

Journal of Distance Education, 19, 133-148.

http://dx.doi.org/10.1207/s15389286ajd e1903_2

Goodhue, D., & Thompson, R. L. (1995). Task-technology fit and

individual performance. MIS Quarterly, 19 , 213-236.

http://dx.doi.org/10.2307/2496 89

Levy, Y. (2008). An empirical development of critical value factors

(CVF) of online learning activities: An application of activity theory

and cognitive valu e theory. Computers & Education, 51, 1664-1675.

http://dx.doi.org/10.1016/j.compedu.2008.04.003

Liaw, S. (2008). Investigating students’ perceived satisfaction, beha-

vioral intention, and effectiveness of e-learning: A case study of the

Blackboard system. Computers & Education, 51, 864-873.

http://dx.doi.org/10.1016/j.compedu.2007.09.005

McGi ll, J., & Klobas, J. E. (200 9). A task -technology fit view of learn-

ing management system impact. Computers & Education, 52, 496-

508. http ://dx.doi .org/10.1016/j.com pedu.2008 .10.002

Moosmayer, D. C., & Siems, F. U. (2012). Values education and stu-

dent satisfaction: German business students’ perceptions of universi-

ties’ value influences. Journal of Marketing for Higher Education,

22, 257-272. http://dx.doi.org/10.1080/08841241.2012.746254

Richardson, J., & Swan, K. (2003). Examining social presence in online

courses in relation to students’ perceived learning and satisfaction.

Journal of As y nchron ous Learning Networks, 7, 68-88.

Rubin, B., Fernandes, R., & Avgerinou, M. D. (2013). The effects of

technology on the Community of Inquiry and satisfaction with online

courses. Internet and Higher Education, 17, 48-57.

http://dx.doi.org/10.1016/j.iheduc.2012.09.006

Selim, H. M. (2007). Critical success factors for e-learn ing ac ceptan ce:

Confirmatory fact or m odels. Computers & Education, 49, 396-413.

http://dx.doi.org/10.1016/j.compedu.2005.09.004

Seymour, E., & Hewitt, M. N. (1997). Talking about Leaving: Why

Under -graduates leave the Sciences. Boulder, CO, Oxford: West-

view Press.

Swan, K. (2002). Building learning communities in online courses: The

importance of interaction. Education, Communication & Information,

2, 23-49. ht tp://dx. doi.org/10.1080/146 3631022000005016

The Desi gn-Based Research Collective (2003). Design-based research:

An emerging paradigm for educational inquiry. Educational Re-

searcher, 32, 5-8. http://dx.d oi.org/10.3102/00131 89X032001005

Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1996). Program

evaluation: Alternative approaches and practical guidelines (2nd

ed.). New York: Longman.

OPEN ACCE SS 1 13