Participants were learners enrolled in courses delivered within the three pilot institutions. Although all pilots were guided by a shared pedagogical framework, local adaptations were made in instructional design and delivery to reflect institutional contexts. For the IASIS pilot, the dataset includes 255 participants, while the IASIS pre-pilot phase involved 67 participants. In the EASD pilot, a total of 130 participants were recorded, compared to 118 participants in the pre-pilot phase. For UPATRAS, competency assessment data were available for both pre-pilot and pilot phases, while grade data were available only for the pilot phase, which involved 251 participants. All datasets were anonymised prior to analysis and collected in accordance with institutional ethical standards (Table 1).

Table 1.Participants in the pilot and pre-pilot.

Learner competencies were measured using structured rubric-based assessment instruments aligned with the Horizon augMENTOR project’s 4Cs theoretical framework ([19]). These instruments operationalised Creativity, Communication, Critical Thinking, and Collaboration through predefined criteria and performance levels, producing numerical scores on a standardised scale for each competency. The rubrics were integrated into the assessment practices of each pilot context and applied consistently across pre-pilot and pilot phases. The instruments were theoretically grounded in established definitions of the 4Cs and had been previously applied. Internal validation procedures were conducted, including data cleaning, consistency checks, and score standardisation across pilots to ensure comparability of measurements. Each competency was represented by a numerical score on a standardised scale and aggregated at the learner level. Prior to analysis, all datasets underwent cleaning and validation procedures to remove incomplete records and ensure consistency across pilots. Variable names and scoring formats were standardised to enable cross-pilot comparison, and competency scores were aggregated to compute mean values per learner and per phase. Competency, grade, and activity log datasets were then integrated using anonymised learner identifiers. Only learners with valid and matching data across the relevant datasets were included in each analytical step. Longitudinal analysis was conducted to examine changes in the 4Cs between the pre-pilot and pilot phases, except for University of Patras. Mean competency scores were calculated for each phase and compared within each pilot to identify absolute changes over time. This analysis enabled the identification of competencies that improved, remained stable, or declined during pilot implementation, as well as differences in developmental patterns across institutional contexts.

Learner activity logs were analysed using engagement metrics including activity completion frequency, number of interactions within learning activities (e.g., submissions or participatory actions), and time-on-task. These indicators were examined in relation to changes in 4C scores at the pilot level to identify behavioural patterns associated with observed competency gains or declines.

In the IASIS pilot, creativity increased from the pre-pilot phase (M = 70.27, SD =

Table 2. Participants in the pilot and pre-pilot.

25.88) to the pilot phase (M = 72.80, SD = 19.57; Δ = +2.53). Communication showed the largest improvement, increasing from M = 64.23 (SD = 22.95) to M = 75.77 (SD = 16.67; Δ = +11.54). Critical thinking also increased from M = 69.66 (SD = 21.92) to M = 75.38 (SD = 17.09; Δ = +5.72), whereas collaboration showed a small decline from M = 78.85 (SD = 18.34) to M = 76.87 (SD = 16.66; Δ = −1.98).

In the EASD pilot, communication increased from M = 75.01 (SD = 23.57) in the pre-pilot phase to M = 80.18 (SD = 21.07; Δ = +5.17). In contrast, decreases were observed for creativity (Δ = −4.37), critical thinking (Δ = −3.70), and collaboration (Δ = −4.15).

In the UPATRAS pilot, substantial gains were observed in critical thinking, which increased from M = 66.10 (SD = 24.65) in the pre-pilot phase to M = 95.09 (SD = 10.18; Δ = +28.99). Creativity also increased (Δ = +7.05), while collaboration remained largely stable (Δ = +0.50).

Across pilots, communication showed consistent improvement in all contexts where data were available (IASIS and EASD). Critical thinking exhibited the greatest variability, ranging from moderate gains (IASIS) to substantial gains (UPATRAS) and moderate declines (EASD). Creativity outcomes varied by context, with increases observed in IASIS and UPATRAS and a decline in EASD. Collaboration showed the least positive development overall, with decreases observed in IASIS and EASD and only a negligible increase in UPATRAS.

One of the most salient findings is the consistent improvement of communication in all pilot contexts where data were available. In both IASIS and EASD, communication showed clear gains despite divergent trends in the other competencies. This pattern aligns with theoretical perspectives that position communication as a competency that benefits strongly from increased structure, explicit criteria, and frequent opportunities for practice and feedback. As communication skills are often directly embedded in assessed tasks, such as written assignments, presentations, or structured interactions, they are likely to be reinforced during pilot phases characterised by clearer expectations and tighter assessment alignment.

Critical thinking displayed the greatest variability across pilots. The very large gains observed in UPATRAS contrast with moderate gains in IASIS and declines in EASD. From a theoretical standpoint, critical thinking is closely tied to pedagogical designs that emphasise problem identification, exploration, and reflective analysis. The findings suggest that when pilot implementations explicitly scaffold these processes, substantial improvements are possible, particularly when pre-pilot baselines are relatively low. Conversely, when pilot conditions prioritise task completion or delivery efficiency over epistemic exploration, gains in critical thinking may be constrained or even reversed. UPATRAS pilot serves as a concrete illustration of the design sensitivity in the evolution of critical thinking. In the context of UPATRAS, learning activities placed a strong emphasis on problem-driven tasks that required learners to analyse complex scenarios, justify their decisions, and iteratively refine their solutions based on feedback. Assessment criteria explicitly prioritised argumentation quality, evidence use, and reflective reasoning. This alignment between task design, assessment focus, and epistemic demands likely contributed to the substantial gains observed in critical thinking, particularly given the relatively low pre-pilot baseline.

Creativity outcomes were markedly context-dependent. While moderate to strong gains were observed in IASIS and UPATRAS, EASD experienced a decline from pre-pilot to pilot phases. Creativity is commonly understood as emerging from a balance between novelty, effectiveness, and implementability. Pilot phases that impose tighter schedules, predefined outputs, or stronger assessment constraints may inadvertently reduce opportunities for divergent thinking and exploratory problem-solving. The observed variability reinforces the view that creativity is particularly sensitive to instructional flexibility and the availability of open-ended learning tasks but also to the potential of being creative over time, showing not only a creative intention but creative perseverance ([14]).

Collaboration emerged as the most fragile of the four competencies. Two pilots showed declines, while the third demonstrated only marginal stability. This pattern is consistent with theoretical and empirical work suggesting that collaboration depends heavily on sustained interaction, reciprocity, and carefully designed group structures. As learning initiatives scale from pre-pilot to pilot phases, reductions in synchronous interaction, changes in group composition, or increased individual accountability may weaken collaborative dynamics, even when collaboration remains an explicit learning objective.

The divergent trajectories observed across pilots also highlight the importance of contextual factors. Differences in pedagogical emphasis, task design, and implementation strategies appear to play a significant role in shaping competency outcomes. The strong gains in critical thinking observed in UPATRAS, for example, suggest that targeted design choices can yield substantial benefits, while the declines observed in EASD indicate that competency development may be compromised when higher-order or collaborative elements are deprioritised. These variations should not be interpreted as reflecting differences in learner ability. Rather, they point to the central role of learning design, assessment practices, and activity structures in enabling or constraining the development of specific competencies. From this perspective, declines or stagnation in certain competencies are better understood as structural effects of implementation choices rather than as failures of learners or educators.

Several limitations should be acknowledged. The quasi-experimental design does not allow for causal inference, and observed changes in competencies cannot be attributed solely to pilot interventions. Furthermore, collaboration outcomes may be under-represented in grade-based analyses, as assessment practices primarily emphasised individual performance. Variability in data availability across pilots, particularly the absence of communication data for UPATRAS, also limits full cross-context comparability.

Our study findings underscore that improvements in transversal competencies are not automatic outcomes of scaling educational interventions. Instead, each competency responds to distinct pedagogical and assessment conditions. Competencies that are closely aligned with individual performance and assessment criteria, such as communication and aspects of critical thinking, appear more resilient during pilot scaling. In contrast, competencies that rely on social interaction or creative exploration are more vulnerable to structural constraints. These results support the argument that transversal competencies should not be treated as a homogeneous construct ([20]). Aggregating the 4Cs into a single composite indicator risks obscuring important differences in how competencies develop and how they are affected by design decisions. A competency-sensitive approach to both instructional design and evaluation is therefore essential, particularly in modular learning environments where learning activities and assessment practices may vary substantially across modules.