The framework identifies the skills involved in the certification system. It provides a mapping of the skills to competencies (dimension 2 of DIGCOMP) and to Proficiency Levels (dimension 3 of DIGCOMP).

DigComp 2.2 is used as a basis for designing this framework and as such, it follows the same structure.

1) Dimension 1: Competence Areas

This dimension uses 3 out of 5 Competence areas described in DIGCOMP, namely:

• Competence area 1: information and data literacy

• Competence area 2: communication and collaboration

• Competence area 3: digital content creation

2) Dimension 2: Competencies

This dimension uses 7 out of 13 DigComp Competence descriptors for the 3 Competence Areas covered, namely:

Competence Area 1: Information and data literacy

1.1 Browsing, searching, filtering data, information and digital content

1.2 Evaluating data, information and digital content

1.3 Managing data, information and digital content

Competence Area 2: Communication and collaboration

2.1 Interacting through digital technologies

2.2 Sharing through digital technologies

2.3 Engaging in citizenship through digital technologies

Competence Area 3: Digital content creation

3.1 Developing digital content

There is also a partial use of descriptors from Competence Areas 2 and 3.

3) Dimension 3: Proficiency Levels

Four (4) out of the eight (8) proficiency levels of DigComp are used, namely:

Foundation level, including:

• Level 1: Basic level with guidance

• Level 2: Basic level with autonomy

Intermediate level, including:

• Level 3: Well-defined, routine tasks

• Level 4: Tasks, well-defined and non-routine problems

4) Dimension 4: Knowledge, skills and attitudes applicable to each Competence

This dimension includes ecommerce specific skills that are based on DigComp but are more targeted to the ecommerce scope.

5) Dimension 5: Examples of use

Examples of use are e-commerce-specific.

The operation of the certification framework requires the design of questions that assess the skills of every Competence Area and Competence included in DigComp, as they were detailed in the previous section. Digital skills can be either generic or related to using ecommerce applications or methods. Both categories of skills need to be mapped to the specific competencies of DigComp. There is a one to many (1 − N) relationship between Competence and skills.

Moreover, skills are Proficiency Level—specific; that is, a skill is mapped not only to a Competence but also to a specific Proficiency Level. These skills are called PL-Skills (Proficiency Level Skills). PL-Skills are the generic digital skills which the trainees of e-commerce are going to acquire during a training programme.

Examples of use are presented for the certification of e-commerce skills. They are derived based on skills and are e-commerce specific. There is generally a many-to-one (N − 1) relationship between mastery and examples of use. From the standards of use, e-commerce-specific skills are derived. These are also included in the skills which e-commerce trainees are intended to acquire during the programme. Generic and e-commerce-specific skills are complementary. The Examples of use, along with the PL-Skills are used to derive questions and answers for the certification tools. The skills are described using the specific terms of each Competence and PL, having in mind the goals of each Competence. Terms such as “identify”, “find”, “explain” and others that appear in the descriptors of Competence in DIGCOMP are used to describe the skills and then to derive examples of use.

Skills are different at each Proficiency Level of each Competence. The descriptors provided by DigComp are quite general and they permit design flexibility. However, this also means a lack of practicality. The structure design differentiates the skills between each PL of a Competence in as clear a way as possible. The differentiation of skills between levels 1 and 2 (foundation level) is based largely on the assistance a trainee needs to perform a task. This is directly and clearly incorporated into the examples of use and the answers that correspond to PL 1 vs. the one that corresponds to PL 2 of a question that assess the knowledge of the skill.

Questions need to be designed to uniformly assess each Competence covering as many PL-Skills as possible and all the e-commerce skills. Answers to Test-I (multiple choice quiz for the trainee) are designed to rate multiple Proficiency Levels using the same question. Thus, questions of Test I are multi-levelled. This requires not only a micro-but also a macro-management of each Competence. Answers need to cover all PLs (micro-management related to skills) while questions need to cover groups of skills that are relative to each other, and they are also multi-levelled (they correspond to different PLs). These skills are called HL-skills and their goal is to help design questions that fit to different PLs. This is needed because answers to each question do cover all PLs.

HL-skills are internal to the design process of the certification framework. Questions conceptually group the assessment of skills from different PLs based on the semantic resemblance of the descriptors corresponding to each PL. Grouping is performed at a Competence level only. The semantic resemblance is expressed in the descriptors when the same clause is used for completing tasks and the only difference is the type of the action, e.g. “identify” for PLs 1 and, “explain” for PL 3 and “illustrate” for PL 4.

The process of mapping dimensions to each other requires the analysis of each Competence Area into competencies. This is provided by DigComp. The next step is to process each Competence separately as follows: based on the descriptors of each PL, a set of HL-Skills is derived. They are used to design the first version of the questions of the certification tools. In parallel, HL-Skills and descriptors are used to derive the PL-Skills for the specific Level of Proficiency Level of the Competence. From each PL-Skill (or wherever possible, from multiple PL-Skills), the next step designs Examples of Use that are e-commerce specific. By using PL-Skills and Examples of Use, the process 1) refines the questions previously designed to produce the minimum set of questions that cover most skills with the best possible quality of answers and 2) designs the answers for each question. Questions can be generic or e-commerce-specific, focusing more on the latter. This process is repeated for every Competence included in the certification framework. The design of the mapping between dimensions and the production of the certification tools is depicted in Figure 1.

Some of the High-Level Skills (HL-Skills) presented in Table 1 are used for designing an initial version of the questions of the certification tools. They are loosely formed groups of PL-Skills. They express mainly functional tasks to permit the design of practical questions (questions that assess the skills of a trainee in a specific context). They are mostly used in the design of Test II questions where the trainee must be given a particular task in a specific context (scenario) to perform and be rated by the trainer. Specific scenarios define, in a concrete way, the goals to be achieved and at the same time allow the trainee to focus on what is assessed and not on secondary tasks that are not under assessment. This guided certification method reduces the workload of both the trainee and the trainer. Furthermore, it reduces the bias introduced by the distraction of the trainee by secondary tasks. Test I questions are more detailed (since the set of PL-skills is larger than that of the HL-skills), so HL-skills are used for refining the questions and calibration between Test I and Test II questions.

The goal of this certification tool is to provide detailed information on the Proficiency Level of the trainee by assessing a wide range of PL-Skills according to the mapping of the LCCF. It is a multiple-choice test including questions that are grouped by DigComp’s competencies. Trainees can only give one answer to the multiple-choice questions.

Answers are designed so as to reflect different digital PLs. A question has answers corresponding to all PLs (one for each level) plus answers that are false. Thus, by choosing a specific answer, the trainee is mapped to a PL and awarded specific points (Figure 2).

Ideally, a scoring strategy should correctly measure the knowledge of the trainee, but trainees may attempt (unconsciously or consciously) to guess answers rather than admit ignorance. Thus, the interplay between the scoring strategy and the student’s strategy must be considered. To this end, two additional false answers are added to each question. The score for choosing a false answer is 0. This highlights a need to distinguish between guessing and giving wrong answers in good faith.

This question design strategy has several advantages. Partial knowledge is credited and scoring somewhat straightforward and thus the transformation to a grade on scale for a specific Competence is direct. This means that there is no need to apply calibration as in the case of questions corresponding only to one PL. Furthermore, by assigning all PLs to one question, the test is more compact and less questions are needed in total for the certification. Having in mind the large number of skills identified in the framework and the use of two certification tools (Test I and Test II), this is a significant parameter. Finally, it must be

noted that the scale used is independent of the number and type of questions used in the assessment tools. If a median score was to be used, then calibration should take place to calculate the exact range of each PL for a specific set of questions. If questions were to be changed, the calibration process should be repeated. It is obvious that when calibration is involved (that is, testing with at least 100 participants to derive the ranges) for each specific set of questions the effort of certifying becomes large.

The main disadvantage of using the median method is the need to design answers for each PL. This is a difficult task since the burden of designing different questions for each PL is shifted to designing different answers for each PL. This method is more easily applied to observation tests. When the number of PLs is large, designing answers get more difficult. This obstacle can be overcome by designing questions that correspond to similar PLs (as defined in DigComp 1 vs. 2.1): Foundation Level (PLs 1/2), Intermediate Level (PLS 3/4), Advanced Level (PLs 4/5) and Highly Specialized (PLs 7/8). This is a two-dimensional organization of both questions and answers.

The score of a Competence C which is certified by k questions q_k is calculated as follows:

S c o r e ( C ) = ∑ i = 1 k s c o r e ( q i )

where

s c o r e ( q i ) = { 1 ,         if   answer   is   PL 1 2 ,     if   answer   is   PL 2 3 ,     if   answer   is   PL3 4 ,     if   answer   is   PL 4 0 ,     if   answer   is   false

Thus the maximum points awarded to an answer are 4 points and the minimum, 0 points. If there are k questions assigned to a Competence then the maximum points awarded are k * 4 and the minimum, 0 points.

Each Competence is generally assessed by the same number of questions. A minimum of 4 questions and a maximum of 8 questions per Competence are used in e-commerce. There are six (6) possible answers in each question: four options correspond to the four PLs and two options to false answers.

This is an observation test where the trainee is asked to perform simple tasks. Each task is assessed by one question. Tasks correspond to PL-skills of various PLs. The trainer observes the reaction of the trainee and assesses the way that the latter executes a task. There are various ways of executing a task, organized into patterns of behaviour. These patterns are, similarly to the answers to the questions of Test I, mapped to each one of the 4 PLs. The trainer selects the pattern (thus, the Proficiency Level) of the behaviour (the closest one) the trainee has exhibited when executing a task and a score is assigned. Scores range from 1 (for PL 1 related behaviour to 4 - PL 4 related behaviour). A score is derived for each question. Competence score is calculated in the same way as in Test I and using a predefined scale, it is translated into a PL for the whole Competence. There is thus conformity between the design principles of Test I and Test 2.

For the successful execution of Test II, the trainer needs to be well informed about the differences between Proficiency Levels in DIGCOMP. He must also be able to distinguish between different patterns of behaviour (ways of executing the task) to successfully select the pattern that is closer to the one the trainee actually exhibited. In order to help the trainer in this task, each question contains a brief description of the scenario and context of the task to be performed by the trainee. These scenarios must be designed and tested for practicality (that is, whether they can be performed at some degree even by a Proficiency Level 1 trainee). In contrast to Test I questions, Test II questions are not theoretic but require execution in real conditions.

A question of Test II gives exactly four choices to the trainer. One of the choices is standard and corresponds to Proficiency Level 1 behaviour: “Can do with assistance”. The other three choices depend on the Competence and skills under assessment (Figure 3).

The scoring strategy assigns a numeric score to a Competence, but this must be transformed into a grade on some scale so as to derive a decision about the Proficiency Level. The basic idea behind this transformation is that a threshold of more than 50% of the maximum score (>50%) of a Proficiency Level must be obtained for the Competence to be assigned at that level.

The maximum score of a Proficiency Level i in a set of k questions assigned to a Competence is calculated by the product:

M a x S c o r e ( i ) = k ∗ i , with i = 1 , 2 , 3 , 4

This result is easily calculated since an answer that corresponds to Proficiency Level i is awarded with a score of i points (Table 2).

The threshold of 50% for each level produces a range of values for transforming a Competence score to a Proficiency Level.

The upper limit of the range of level i of a Competence C that is certified by k questions, (where k is an odd number) is:

U p L i m i t ( C i ) = ⌈ k 2 ⌉ i + ⌊ k 2 ⌋ ( i + 1 )

The upper limit of the range of level i of a Competence C that is certified by k questions, (where k is an even number) is:

U p L i m i t ( C i ) = k 2 i + k 2 ( i + 1 ) − 1

For Test I, the minimum score value of a Competence is zero and the maximum is k * 4 (Table 3).

For Test ΙI, the minimum score value of a Competence is k and the maximum is k * 4 since there are no answers awarding 0 points (Table 4).

The threshold for odd numbers practically means that if more than 50% (>50%) of the answers correspond to level i, then the certificate is awarded to

that Proficiency Level. An assumption is made that within the assessment of a Competence, responses usually vary between neighbouring levels (e.g. between Level 1 and 2 or level 2 and 3 etc.). Significant PL variations in answers constitute outlier cases which are difficult to manage.

The threshold for odd numbers practically means that, in the worst case, where half of the answers correspond to level i and the other half to level i + 1, the largest level prevails.

The method presented above deals better with marginal cases (as described above) than the more simplistic one of dividing the 0 − k * 4 range to four equal intervals (0% - 25%, 26% - 50%, 51% - 75% and 76% - 100%) or even the modified four-level Danish scale (0% - 22.5%, 23% - 50%, 51% - 77%, 78% - 100%). Cases of outliers cannot be managed well by any method. For example, when a trainee chooses answers corresponding to all five levels within a Competence, a specific PL cannot be easily derived. In this case, the median is calculated. Outliers are caused when a trainee answers questions randomly (the presence of false answers significantly reduces the appearance of such outliers) or when the questions are not designed correctly. The presence of outliers is more probable to appear in Test I than in Test II (the latter is conducted by an expert and is thus considered more reliable).

The Score Mapping to PL certification table for the 41 questions of the e-commerce Test I is presented along with the Skills-Questions-Answers matrix. The matrix also contains the points awarded for each answer to a question.

The score mapping to the level certification table for the set of 31 questions related to e-commerce Test II is presented within the test, after the questions assessing Competence.

Each one of the tests produces a result for each Competence, used to derive the final Proficiency Level. If the results of both tests for a specific Competence are in agreement, then the Certificate is awarded for the standard PL derived. If the results are in disagreement, then the result of Test II prevails, and the corresponding PL is derived as final.

The reason for choosing the result of Test II over the one of Test I is that, generally, controlled assessments with the participation of experts are more reliable than multiple-choice questions quizzes. Furthermore, it is not safe to combine the results of both tests because a different number of questions may assess Competence in Test I than in Test II. The range of values for calculating the final Proficiency Level will also differ, not only because of the number of questions but because of the different minimum values of the 1st range (Table 3 and Table 4 in Section 4.3).

Some examples of skills and usage scenarios related to the competence “browsing, searching, filtering data, information and digital content mapping” (1.1) for the Foundation PL, are presented in Table 5.

At the Intermediate Proficiency level, the examples become more complicated, following the increased complexity of the relevant skills. Usually, more than one example of use is needed to depict this complexity. The difference from the Foundation level lays on the ability of the trainee to perform tasks individually, without the help of an expert. Some examples of skills and usage scenarios related to the competence “browsing, searching, filtering data, information and digital content mapping” (1.1) for the Intermediate PF, are presented in Table 6.