The National University of Mongolia (NUM) is the country’s oldest and only comprehensive university and a leading center of science, education and culture. It has more than 30 schools, institutions and research centers. There are more than 10,000 students in 16 branch schools. The School of Mathematics and Computer Science (SMCS) of the National University of Mongolia has three branches: Theoretical Mathematics, Application Mathematics and Information Technology.

The School of Mathematics and Computer Science (SMCS) is planning to build infrastructure model for elearning. This paper addresses selection and evaluation of most appropriate e-learning system within the above objective.

Building the infrastructure for online learning requires that many factors be considered, so it is difficult to provide a straight-forward checklist or recipe to follow. All educational endeavors are systems, made up of various interconnected components [1].

Our infrastructure model for e-learning system consists of following interconnected components:

• E-learning system;

• University Management Information System;

• E-library system;

• E-content development;

• Other services to students.

There are two development aspects:

1) Waterfall development [2]. To develop complex information system, which includes all sub-systems;

2) Agile development. To develop system by developing sub-systems separately and integrating them.

Usually developers use a waterfall method when developing e-learning. The most used one is the ADDIE model, where development has five phases: analysis, design, development, implementation and evaluation. It worked for years but it takes a long time to go through all the phases, not really suited for on demand responses. We need to get faster and more iterative [3].

The iterative nature of agile development means features are delivered incrementally, enabling some benefits to be realized early as the product continues to develop. A key principle of agile development is that testing is integrated throughout the lifecycle, enabling regular inspection of the working product as it develops [3].

Therefore we selected Agile development method for building infrastructure model for e-learning system.

The SMCS has been implementing an open source e-learning system based on international standard since 2009. During this period we have implemented several systems [4].

This work addresses development and implementation of systems; comparisons of LMSs; survey on system usage are also presented. The first section describes an in-house development of e-learning system. The second section addresses comparative analysis on LMSs using Edutools; [5] the third—Comparison on adaptation of open source e-learning systems by using evaluation Methodology: Qualitative Weight and Sum Approach [6]; the fourth—Survey on System Usage. The last section comments on the advantages and disadvantages on development and implementation of e-learning system.

We developed e-learning system named Edunet 1.0 [4] on Rails framework based on Ruby programming language [7] with MVC [8] architecture. The system has following modules or functionalities:

• Message: This module supports information flows: Education department-lecturer, lecturer-lecturer and lecturer-student. Also file attachment is available.

• Course: This module allows students to receive all information and lecture materials of specific course.

• File: This module allows to lecturer to upload course related files to the system and to students to download course materials.

• Homework: This module allows the Education department and lecturer to receive tasks and homework in file format within pre-defined period.

• Quiz: Quiz module allows the lecturer to set quiz tests, to set a time period for testing and to export to MS excel file. Quizzes can allow multiple attempts.

• Journal: This module allows the lecturer to insert student’s entry, activity or participation. The marks for quizzes will be inserted automatically to the journal.

• Discussion: This module provides a simple communication method between lecture and students. A student can open a dialogue with a lecturer and ask questions.

This system is important in terms of supporting learning activities and helping to conduct training. However this system does not meet all functionalities of modern learning systems such as discussion, content development and electronic presentation of course materials etc. Thus, the users requested updates.

Due to the lack of human resources in the development team, it was impossible to improve the content development and other technical issues in the system. Therefore, we have decided to implement another e-learning system [6].

The following problems occurred during the two years implemetation of the Edunet 1.0 [4] system:

1) Development of new function, module or application according to each new requirement. Duration of development takes usually 1 - 2 months;

2) Lecturers send lecture materials to students directly;

3) Quiz module supports only few simple types of questions;

4) Impossible to use course materials in the next academic year or backup course materials;

5) Impossible to analyze students activity or participation;

6) Lack of key e-learning modules such as team management, virtual classroom etc.

For comparing LMS products, we used www.edutools.info tools [5]. The comparative analysis on key modules of LMSs, such as Blackboard [9], Angel [10], JoomlaLMS [11], Moodle [12], and ATutor [13] described in Table 1, where “+” marks availability of current function. For example, five “+” for file exchange module of Black-Board LMS and one “+” for Moodle system. It means the Blackboard has a five different method of file exchange and Moodle has only one.

Our study shows that commercial systems have more functionalities than open source free systems. But open source systems have most of key necessary functionalities and possibility for future development according to the user needs and requirement.

Therefore based on our studies we decided to implement Moodle [12] open source system.

One of key issues to consider when developing and implementing e-learning systems is the adaptation of the current system to the cognitive learning characteristics of the students. Implementation of the adaptation is not a simple process, since it implies the study and conjunction of technical and pedagogical issues [14].

Therefore we evaluated an adaptation of systems based on evaluation study of open source e-learning platforms/ Virtual Learning Environment [15]. The most adaptable system is selected on results of evaluation on adaptaion systems using Qualitative Weight and Sum Approach method [6].

There are two main points on evaluation of e-learning systems:

• Selection of modules to be evaluated;

• Selection of evaluation method.

Evaluation was carried out by using the Qualitative Weight and Sum Approach method according to the IEEE LTSA [6] reference model. The qualitative weight and sum (QWS) approach is a well-established approach for the evaluation of software products. It establishes and weights a list of criteria. QWS is based on the use of symbols. There are six qualitative levels of importance for the weights, frequently symbols are used: * = extremely valuable, # = very valuable, + = valuable, | = marginally valuable and 0 = not valuable.

We considered following criterias defined by P. Baumgartner and H. Häfele [16]:

Table 1. Comparative analysis on LMS.

• Support of dynamic communication;

• Sustainability of development;

• Good documentation of the system.

The following modules of the ATutor, Dokeos, ILIAS, Moodle, OpenUSS, Sakai, and Canvas systems have been examined:

• Communication tools;

• Learning objects;

• Management of user data;

• Usability;

• Adaptation;

• Technical aspects;

• Administration;

• Course management.

After defining above modules we divided these modules into sub-modules and carried out weight of each sub-module. Maximum value is a maximum value of current criteria. List of modules and sub-modules by each system are presented in Table 2, where red color is best one. Table 2 shows Canvas-five, Moodle-two and ILIASone best result. Summarized evaluation is presented in Table 3. Figure 1 shows that weights of Canvas [17] and Moodle systems are higher than others.

We started to implement Moodle system since 2009- 2010 academic years and developed 20 course materials to the system. We are still using this system in conjunction with other systems. We also decided to implement the Canvas Learning Management System that was selected because of its:

• clean and user-friendly interface;

• rich collaboration and discussion tools;

• ability to embed multi-media and web-based resources;

• integration with communication tools such as Facebook, GoogleDocs, Skype and Google Apps for Education, and compatibility with other existing teaching technologies [15];

• great course management and organization tools including a calendar that integrates and displays due dates and academic activities for students;

Table 2. Evaluation results of e-learning platforms for each subcategory.

Table 3. Summarized evaluation by using method qualitative weight and sum approach.

• well-designed rubrics and grading tools.

Within the implementation of the system, we:

1) Implemented vitual server at http://lesson.num.edu. mn and uploaded more than 200 hours lecture materials;

2) Carried out the localization of the Canvas system to Mongolian language;

3) Uploaded localized system to the server conducted training on system usage to all lecturers.

According our survey and analysis, learning management systems should have following functionalities [18]:

• Logon/authentication with high security;

• Ability to configure according to user requirements;

• Ability to integrate student activities;

• Curriculum development with course specifics and selected learning methods;

• Course management;

• Student enrollment;

• Communication support (social networking, discussion forms, live chat etc.);

• Support of SCORM 2004.

Table 4 shows that our developed Edunet system has only three of the 12 above mentioned functions.

We conducted a survey on those systems in 2009-2010 and 2010-2011 academic years. The survey covers total of 1426 students in 2009-2010, 756 students in 2010- 2011, and 670 students in 2011-2012 academic years. Survey results are presented in Tables 5 and 6.

Based on our survey, we compared system normal use with all three systems using following formula:

where, X-percentage of quality of current indicator, Ynumber of responses on Мoodle and Canvas systems, Ztotal number of students participated in survey.

Final result is presented in Table 7.

Figure 2 shows that most indicators were unsatisfactory for the in-house Edunet system.

The system response time for Edunet system was higher than Moodle system because of the functionalities of the system are less than Moodle and the system doesn’t require additional libraries. Because of its use of web services for team works and collaboration activities, related indicators for the Moodle were unsatisfactory in our survey. The most indicators were satisfactory for the Canvas system in development mode and approximate to the normal level. System response time can be higher in production mode than development mode.

The success of e-learning in tertiary education may be attributed to the following factors:

• Sustainable finding;

• Total commitment and support from top management;

• Participation, cooperation and support from other major universities and IT communities;

• Sufficient development and support staff with advanced technical skills;

• Strong technical support;

• Expertise in instructional design.

Considering the rapid development of the ICT, especially that of the educational technologies, networks and software, design and implementation of an e-learning system, the customized system in-house by the institute itself is not best option. Because, our lessons and practices show that, the in-house development of learning system by academic institutions (especially in developing countries) may experience following disadvantages:

• Development and implementation costs are high;

• The development team is not fully familiar with the standards of e-learning systems for which in most cases, some training classes are held to educate the team.

Table 4. Comparison of three implemented systems.

Table 5. Survey results: 2010-2011 academic years.

Table 6. Survey results: 2011-2012 academic years.

Table 7. Comparison on system usage.

• System development demands a close collaboration between the IT professionals and the educational experts.

• If the development team’s output is low, the actual overall expenses for this approach would be higher than other solutions [8].

The SMCS uses SISI University information management system. Current open source e-learning systems can receive users and course related information from SISI system using LDAP server in real-time.

We are planning to conduct online trainings using the Moodle and Canvas systems in future, and started to implement these systems, which cover following activities:

1) Conduct application trainings of these systems for

lecturers and students;

2) Conduct trainings for system administrators;

3) Develop user manual of the Moodle and Canvas systems in Mongolian language;

4) Conduct trainings on development of e-courses for lecturers. These trainings will cover applications such as eXe, Courselab, Articulate, Ispring, Adobe Flash which support Scorm standard;

5) Import data of all system users and courses from SISI system.