As a hypothesis, we refer to a complex (modular) information system as an Information System (IS), which provides various diverse services organized in modules. The quality assessment of such a system is a complicated process due to the diversity of its components, possibly a multi-faceted approach, considering the individual characteristics of each module [9] .

Each module in a complex information system may have its unique functionalities, user requirements, and potential pitfalls. Consequently, assessing the quality of each individual module can be likened to evaluating multiple distinct systems. Additionally, the interdependencies and interactions between these modules add another layer of complexity to the assessment. When one module malfunctions or underperforms, it may produce ripple effects throughout the entire system, impacting the efficiency and effectiveness of other modules. The metrics and benchmarks used for quality assessment may vary from one module to another. For instance, the module responsible for data storage might be evaluated based on its speed, capacity, and reliability. In contrast, a module designed for user interaction might be assessed based on user-friendliness, responsiveness, and accessibility.

In this paper, we argue that the ISO25000 series standard can be used to assess complex information systems more efficiently if suitable metrics are mapped to individual components. For this research, we use, as a case study, a model of a modular, general-purpose Information System with the following components (sets of services):

· Workflow Management: This service aims to unify the protocol assignment process and facilitate the traceability of protocolled documents throughout their life cycle. It entails a set of functions controlling the inflow and outflow of documents during their circulation. It will include document management and digital signature of documents (possibly using third-party software, the license and installation of which is the responsibility of the contractor).

· User Management: This service aims to offer a secure, consistent, and unified mechanism for managing the users and their roles for the entirety of the system. This service implements a secure access policy to the content and services of the system while providing efficient ways to prevent access to unauthorized users and limit the outcomes of malicious actions.

· Data Warehouse: The objective of this service is the gathering, standardization, organization, and utilizing data and operational knowledge derived from primary data sources. It’s a set of functions mainly targeting internal users, organizing, and managing data originating from various participant registries and historical data. It supports feedback mechanisms and data extraction applications.

· Business Intelligence: The goal of the service is to provide high-quality, uniform, and cohesive data to facilitate complex queries, monitor results, and assist decisions at both a tactical and strategic level. It concerns services which merge data (data fusion) from internal and external repositories and store them in an appropriate format in the Data Warehouse of the IS. This merging combines information stored in various heterogeneous environments, their integration, and presentation in a single, consistent business model. It provides the capability for query submission, conducting research, and producing reports based on dynamic criteria.

· E-commerce Portal: This service aims to enable online purchasing to shoppers and partners via a centralized portal. It aims to assist them in making informed purchasing or partnership decisions, leveraging the latest and most efficient e-commerce technologies.

The architecture of the IS conceptually groups services into subsystems (modules). The purpose of the subsystems is to integrate processes that will be used by the applications that constitute the IS. Each subsystem should be considered an autonomous entity but should cover a range of functions that are characteristic of it. For example, the user management subsystem should not depend on the data warehouse subsystem but only be interconnected. The development, upgrading, and maintenance should not prevent the other’s upgrading, and maintenance. The subsystems should be central to the IS since many applications will depend on them.

The design, analysis, development, testing, evaluation, and support of complex IS, as defined in this work, is relatively ideal for complying with the software quality standard ISO25000 series (SQuaRE—System and Software Quality Requirements and Evaluation) [8] . The standard ISO/IEC 25012:2008 defines the general data quality standard for data stored within a system. It is used to determine the data quality requirements, the data quality measures, and the design and conduct of evaluations of the data quality of IS. The standard categorizes quality characteristics into fifteen characteristics from two perspectives: inherent and system dependent. It is intended for use in conjunction with other parts of the 25,000 series, such as the ISO/IEC25010 standard [10] . The data quality of an IS complies with ISO25012, which belongs to the ISO25000 series of standards [11] . This standard provides a general framework for evaluating the quality of data from various perspectives, including application requirements, product quality, and data quality during data management. The ISO25012 standard is ideal for the IS as it finds broad application in organizations and businesses that handle large amounts of data and want to ensure the high quality of their data [12] . Using the standard can help improve data processing performance, evaluate, and select data-based software or services, and ensure compliance with security and data protection requirements.

The application of the ISO25000 series of standards enables the management of the life cycle phases of an IS (ISO25020) and its data (ISO25012). The primary principles are:

· Effectiveness: This refers to the efficient coverage of all life cycle phases. The standard addresses every phase of an IS life cycle, meaning many processes may be covered in each phase. Although covering the entirety of these processes is an advantage, it also increases the complexity of the IS.

· Completeness: This covers all modules of the IS, irrespective of the type of user, data, architectural specifics, and dissemination means. The standard encompasses two main axes: the operational and the technological.

· Flexibility: The standard allows for various certification levels. Following the proven best practices of the ISO organization, the ISO25000 series is hierarchical and non-overlapping. However, the border between sub-characteristics is often only apparent to specialists.

· Practicality: It responds to real-world needs by defining levels based on economic and operational parameters and the IS’s cost-performance relationship. The practicality of the standard is enhanced by the use of quality metrics to provide tangible performance measurement indicators and best practices that offer more detail or clarification of the standard’s guidelines.

· User-Centric Focus: A critical prerequisite for the successful use of an IS is user participation, both in design processes (through needs analysis) and in improvement (via evaluation). Many standards’ shift towards a user-centric focus is emphatically expressed in various instances, either by adding to existing standards (like the Quality in Use pillar to ISO9126) or by designing the standard entirely based on the user (as with ISO25000).

The philosophy of ISO standards is to use different evaluation approaches in the same standard, depending on the product’s life cycle phase or which part of the product is being evaluated. Each approach corresponds to specific, distinct features and sub-features.

In the new standards of the 25,000 series, Internal and External Quality and Quality in Use are linked to the phases of the product’s life cycle. Based on this definition, quality (hence the components of a standard) can be related to the software lifecycle through the life cycle model. Quality is preferred (according to ISO25010) to three main phases of the product’s life cycle:

· During the phase where the product is under construction: the evaluation refers to Internal Quality.

· During the phase where the product is in the evaluation phase: the evaluation refers to External Quality.

· During the phase where the product is in the usage phase: the evaluation refers to Quality in Use.

The requirements for Quality in Use are of great importance and it determines the required quality level from the user’s perspective. These requirements are used to validate the product by users. They are determined by specific quality metrics in use. External requirements contribute to the recognition and definition of internal quality requirements and are, in turn, used to predict the requirements for quality in use. This creates a cycle where the requirements of one category contribute to determining the requirements of the next. On the other hand, Internal Quality requirements on the quality from the “internal” side of the product. They are used to determine the properties of intermediate products of the production process (software requirements, source code, etc.). They are used to define the properties and non-executable deliverables such as documentation and user manuals. They serve as validation targets at various stages of product development for determining development strategies, validation criteria, and evaluation. These requirements are quantitatively defined in the form of metrics— measures.

The role of standards is to provide guidelines for ensuring the quality of data or software. A standard is an official agreement that details technical specifications that can be used as rules for evaluating a subject [7] .

The structure of ISO organization standards is usually hierarchical. At the top of the hierarchical structure are the quality characteristics. They constitute categories of quality components that do not overlap. Each characteristic contains (or is broken down into) a set of non-overlapping quality sub-features. The non-overlapping nature of the characteristics implies that the relationship between characteristics and sub-features is one to many. These two levels of a standard’s structure describe, in general terms, the quality components to which absolute values cannot be attributed during the evaluation of the subject, only descriptive values. This is necessary to ensure the generality of the standards, meaning their independence from specific techniques or implementation technologies of the evaluated object [10] .

The structure’s third level consists of metrics with also a one-to-many relationship with the sub-features. Metrics can take absolute values and are measures of quality. In many cases, their practical value is significant as they can provide precise information/guidance for the design/construction of quality objects. However, a numerical value cannot accurately reflect reality, as this holds for most absolute measures. Therefore, in quality evaluation, metrics should be used with caution.

Implementation guidelines or usage examples usually accompany standards. These do not constitute part of their structure. It is common for new standards that exclusively contain specifications or application instructions of other standards (that include quality models), management of the standard application processes, or general reference frameworks. If appropriately adapted, these can form the basis for quality evaluation systems or quality specifications.

The Software Quality Measurement Model, as defined in ISO25010, outlines the inherent properties of the software, which can be distinguished quantitatively or qualitatively as characteristics. Quality characteristics are the inherent properties of the software that contribute to its quality. These quality characteristics are categorized into one or more sub-characteristics. Quality characteristics are measured using a measurement method. The result of applying a measurement method is called a quality measurement element. Quality characteristics and sub-features can be quantified by applying measurement functions to these elements. A function is essentially an algorithm used to combine elements. The result of applying a measurement function is called a quality measure. In this way, quality measurement elements become quantified reflections of quality characteristics and sub-characteristics. More than one measure can be used to measure a feature or sub-feature.

The ISO25000 series, also known as Software Product Quality Requirements and Evaluation (SQuaRE), is the newest version of standards for software system quality. It was designed to replace the standards of the 9000 and 10,000 series with the goal of standardization and the elimination of overlaps. The standard’s objective is to replace ISO9126-1 in terms of providing a quality model for evaluating software systems and services. ISO25010 is based on the quality model of ISO9126, has a similar hierarchical structure, and most of its characteristics and sub-characteristics are the same. The new standard does not have defined metrics and relies on the metrics of the ISO9126 standard.

The research methodology uses ISO25010 (which focuses on the quality of software and systems) and ISO25012 (which focuses on data quality). Both are crucial for evaluating and improving software and data quality [12] . ISO25010, entitled “Quality Models for Systems and Software,” offers a comprehensive guide to the quality of software products, defining the main aspects of quality that need to be considered. It covers various elements such as performance, reliability, usability, and more. ISO25012, referred to as the “Data Quality Model,” focuses on the quality of information data. It provides a thorough framework for data quality assessment, considering various perspectives, from application requirements to data management. Each of these characteristics is further broken down into sub-characteristics. This standard defines three categories of data quality: intrinsic data quality, system-dependent data quality, and user-dependent data quality. It evaluates various properties of data, such as accuracy, completeness, consistency, reliability, timeliness, accessibility, etc.

The goal of the ISO25000 series standard is to replace ISO9126-1 in terms of providing a quality model for the evaluation of software systems and services. The new standard does not have predefined metrics and relies on the metrics of the ISO9126 standard. The innovation of the present work is based on this observation. The ISO25010 standard addresses “System and Software Quality Models”. It defines the quality characteristics and sub-characteristics that must be taken into account in the evaluation of a software product. The differences between the two standards are as follows:

· ISO25010 has eight features against 6 of ISO9126 and 39 sub-features.

· Functional completeness has been added as a sub-characteristic, and Interoperability and Safety have been moved as new quality features.

· The Accuracy has been renamed to Functional Correctness and the Suitability to Functional Suitability.

· Efficiency has been renamed to Efficiency Capability. The capacity is added as its sub-characteristic. Compatibility is a new feature that now includes sub-characteristics Coexistence (which moved from Portability) and Interoperability (which moved from Functionality).

· Usability has as new sub-features the User Error Protection and Accessibility (used by individuals with a wide range of characteristics).

· Understandability is renamed to Recognizability, and Attractiveness is renamed to User Interface Aesthetics.

· Reliability has a new sub-feature, that of Availability (when required to be used).

· Security is a new feature with sub-characteristics of Privacy (data accessible only by authorized users), Integrity (protection from unauthorized modification), Non-Repudiation of Responsibility and Authenticity Ability.

· Maintainability has new sub-features of its Extensibility.

· Reusability Replaceability and stability are components of Modifiability.

The ISO/IEC25012: 2008 standard defines the general data quality standard for data stored in a structured format within a computer system, such as an IS [11] . It can be used for defining data quality requirements, for data quality measurements, or for designing and conducting data quality evaluations. It could be used, for instance, to define data quality requirements during production, acquisition, and completion processes, to identify quality assurance criteria that are useful for the reuse, validation, and improvement of data, and for the reorganization, evaluation, and improvement of data, and to assess the compliance of the data with legislation or/and requirements. The standard categorizes quality characteristics into fifteen characteristics from two perspectives: inherent and system dependent. The ISO25012 data quality model defines:

· Internal Data Quality, which refers to:

- Data values

- Data types and sizes

- Data definitions (including metadata)

- Data rules

- Data links

· External Data Quality refers to the ability of the data to meet specified needs under specified conditions within a software system.

Metrics serve as a principal constituent feature of quality standards, positioned at the lowest level of the hierarchy. Nonetheless, their practical value is significant. Metrics are deployed for the appraisal and quantification of properties of the item or information under assessment. They present an empirical, objective assignment of a numerical or symbolic value to an entity, or a component of the assessed object aimed at evaluating a particular attribute of it. The utilization of metrics targets addressing the fundamental challenge of defining measurable quantities. While the notions of quality characteristics and sub-characteristics are marked by flexibility and breadth in interpretation, metrics are determined through measurements. Measurement is the process through which numbers or symbols are aligned with properties of the components comprising the evaluated system or product, describing them based on defined rules. The interpretation of measurements attributed to metrics remains a pivotal area of study, especially their interpretation concerning the quality of the system. The interpretation of a metric lies in determining its degree of correlation with one or more external features of the system.

The metrics of the ISO25000 series can be classified according to the nature of the object they evaluate or the type of feature they address. They can be discerned into process metrics, which pertain to the development process; resource metrics, addressing the available resources for the development of the object; and product metrics, focusing on the characteristics the system possesses for end-user delivery. A fundamental categorization of metrics is executed based on the nature of the features they measure, distinguishing between internal and external metrics.

There are two general categories of metrics. Internal metrics can be applied during the design and development phase. During the development process, interim derivatives can be assessed using internal metrics. The primary goal of internal metrics is to ensure the required external quality of the system and its quality in use. Internal metrics measure the system’s internal features by analyzing the properties of the intermediates or deliverables before they are used under real conditions. Measurements of internal metrics refer to numerical data related to the frequency of appearance of the elements that constitute the object and, for example, refer to the source code, the flow diagram, and the system’s complexity. Characteristic examples of internal metrics for software are the lines of code (LOC) that characterize the size of the software’s source code and the cyclomatic complexity, a measure of the software’s complexity based on the flow chart. Cyclomatic complexity aims to highlight parts of the software that will be difficult to understand, test, and maintain. External metrics on the other hand, are based on the definition of quality that emphasizes user satisfaction and directly measure the desired external features and are grouped by quality sub-characteristic, suggesting a way to apply them. Their key feature is that they require user participation and developer involvement. They capture the system’s external quality in combination with the internal knowledge of the features the system provides. The importance of external metrics is to depict the object’s external quality in relation to the functions and services they offer to the end-user under real usage conditions.

Data Quality metrics can be produced per quality characteristics and be applied horizontally, that is, for all the modules of a complex Information system. Table 1 provides a non-exhaustive list of the data quality metrics and the corresponding sub-characteristic of ISO25012.

A metametric is a metric that combines two or more existing metrics to measure complex or critical features. They provide a higher-level view by combining existing metrics. They can offer insights that might not be immediately apparent from individual metrics [13] . As a case study for the IS at hand, a set of Meta-metrics for the BI component based on the previously mentioned metrics is depicted in Table 7. It must be noted that these meta-metrics offer a holistic view of the BI component’s capabilities and strengths. By blending individual metrics, it is possible to provide to IS stakeholdersa more comprehensive understanding of specific complex aspects of the system. They are especially useful when making strategic decisions or comparing multiple BI systems. However, the weights and formulas can be adjusted depending on the particular emphasis and priorities of the organization.

The utilization of meta-metrics to assess the Business Intelligence (BI) component offers an insightful perspective into the multi-faceted nature of such a module [9] . The right combination of multiple individual metrics, meta-metrics provide a comprehensive lens through which the system’s performance and capabilities can be observed. More specifically, assessing a BI component using meta-metrics facilitates a holistic overview of the system. Rather than navigating the intricacies of each individual metric, decision-makers can swiftly discern the

performance domains of the system. This approach also lends itself well to comparative analysis, enabling organizations to calibrate their BI system’s performance against industry standards or rivals using universally recognized meta-metrics. Furthermore, when confronted with the need to make strategic decisions about areas of investment, meta-metrics provide some guidance on whether the emphasis should be on enhancing performance, user experience, or perhaps security.

However, the adoption of meta-metrics is not devoid of challenges. A notable concern is the potential for over-generalization. The very act of blending several metrics might inadvertently lead to the simplification of specific aspects, creating a blind spot for particular issues. Additionally, the question of weight allocation for each embedded metric within a meta-metric often surfaces, as this distribution might be perceived as subjective and potentially not encapsulate actual significance or implications. The evolving nature of the BI component also necessitates that the meta-metrics be frequently recalibrated or redefined to maintain their relevance. Moreover, the accuracy of individual metrics plays a pivotal role; any miscalculation in these foundational metrics can misguide the interpretation of the meta-metric. Finally, without a deep-rooted understanding of the constituent metrics, stakeholders could misconstrue the ramifications of a specific metameric value.

Despite the shortcomings mentioned before, the benefits of employing meta-metrics are manifold. The streamlined nature of meta-metrics ensures strict reporting, allowing BI managers to present consolidated insights to leadership, aiding in a more efficient grasp of the system’s health and performance. They also pave the way for a standardized modus operandi for evaluating BI components across diverse departments, initiatives, or even organizations. The act of aggregating related metrics might unveil correlations or insights that would be otherwise concealed when examining metrics in isolation. This holistic focus, inherent to meta-metrics, facilitates BI teams to strategize comprehensive improvements as opposed to merely addressing individual anomalies. Furthermore, the very essence of meta-metrics simplifies the communicative process, which is especially beneficial when elucidating intricate BI concepts to non-technical stake-holders.

The metrics mentioned in the previous section may play a positive role during the Validation and Verification (V&V) phase of an IS, presenting objective and quantifiable goals to evaluate system efficacy, performance, and alignment with predetermined standards. Through the application of these metrics, stakeholders can attain an augmented assurance of the system’s deployment readiness and its aptitude to cater to both user and business imperatives.

V&V can be described as a process used to discover software defects and to confirm that the software is of quality (to some extent) in relation to some of its features. It is used to detect errors but also to evaluate quality factors such as reliability, security, usability, etc. Software testing is implemented through test scenarios that employ metrics. The modern approach to designing test scenarios is to generally describe the initial state of the user, the steps to be followed in order to test the software (after setting the appropriate goal for the test), and finally, to describe the expected outcome. Metrics play a pivotal role in defining the goal of test scenarios, how they will be implemented and the success threshold. This process offers flexibility. The basic principles of software testing through test scenarios follow a list of principles:

· Principle 1: The purpose of testing is to discover errors and assess the quality of the software.

· Principle 2: A good test scenario is likely to discover a new error or a quality failure (corresponding metric success value not obtained).

· Principle 3: The results of a test should be meticulously reviewed.

· Principle 4: A test scenario must necessarily include the expected output data. Desired output results are essential but are often overlooked. For software that performs simple operations, the tester calculates the expected values without having previously budgeted them. This is difficult to happen in specialized software or in pieces of software that interact with each other. Metrics facilitate the measurement in such complex cases.

· Principle 5: Test scenarios should be designed for both valid and invalid input data.

· Principle 6: The likelihood of defects in the software is proportional to the number of defects that have already been identified. The greater the number of errors detected in a piece of software, the greater the likelihood that there are others that have not been discovered. This principle is based on the empirical observation that errors occur in clusters.

Metrics for various components, such as BI, E-commerce, and more, offer objective standards that can be effectively harnessed to streamline and enhance the V&V processes. For instance, metrics related to the completeness and correctness of transactions, be it in E-commerce or BI queries, furnish verifiable evidence to confirm that specific functions of the system are operational as envisioned. In particular, metrics like “Number of complete BI queries” can be instrumental in verifying the proficiency of the BI component in data retrieval. Additionally, increasing performance efficiency is pivotal. Measures like page load times, BI dashboard load times, and throughput can serve as tangible indicators during validation. If the system falls short of these benchmarks, optimization might become imperative before it’s ready for deployment. The intersection of components and their interdependence can be gauged using meta-metrics, offering a panoramic view of component interactions. For example, the “Transaction Efficiency Index” meta-metric can corroborate that the BI and E-commerce segments of the system synergistically bolster each other’s efficiency.

Establishing corresponding thresholds or benchmark scores for meta-metrics is pivotal for the practical assessment of complex IS. These benchmarks, which act as reference points, provide a context within which the computed values of meta-metrics can be evaluated to determine the system’s performance relative to predetermined standards.

One approach to determining these benchmarks is through the analysis of historical data. If an organization maintains a repository of data accrued over extended periods, this data can be analyzed to discern typical value ranges and averages for meta-metrics. Specifically, historical records can be scrutinized to determine significant percentiles, with the median often serving as a central reference point and other percentiles indicating variations.

Moreover, competitive benchmarking presents another robust methodology. In this context, an organization’s BI system meta-metrics are juxtaposed with those of industry peers or accepted industry benchmarks. By making such comparisons, the organization can ascertain its position in relation to its competitors or industry norms. It’s the pursuit of meeting or surpassing these industry standards that can guide the establishment of internal benchmarks.

Furthermore, eliciting the judgment of domain experts can provide invaluable insights. Experts, due to their deep understanding of IS and their intricacies, are equipped to offer insights into the criteria that might classify performance as satisfactory, commendable, or exceptional. This can be achieved through organized expert panels, comprehensive interviews, or using techniques like the Delphi method to arrive at a consensus regarding appropriate benchmarks.

An organization might also consider a goal-oriented strategy, whereby performance objectives are explicitly set for the IS. Once articulated, these goals can be reverse-engineered into quantifiable benchmarks for the meta-metrics. The focus here is on ensuring that the metrics align with the broader objectives of the IS and, by extension, the organization.

Statistical methodologies also offer a way to discern benchmark values. For instance, adopting the Six Sigma philosophy might involve defining thresholds that are, say, three standard deviations from the mean, encompassing the majority of the data points and deeming any deviation from this as noteworthy or anomalous.

It is also worthwhile to consider user feedback in this context. End-users, being the primary beneficiaries of the IS, are often in an optimal position to provide feedback on its performance. Their perspectives can be collated and subsequently translated into quantifiable benchmarks for pertinent meta-metrics.

Given the dynamic and evolving nature of IS, it becomes imperative to emphasize iterative refinement. As systems change and as more data is collated, there arises a necessity to continually revisit and recalibrate these benchmarks. This iterative process ensures that benchmarks remain germane and are reflective of the current realities of the system. Lastly, a composite approach often yields optimal outcomes. A synthesis of insights from historical data, expert feedback, and competitive benchmarking can be integrated to derive comprehensive and robust benchmarks. Such a holistic approach ensures that the benchmarks are both grounded in empirical evidence and strategically aligned with the organization’s vision.

The benchmarking process is another way of obtaining metric boundaries; its intricacies and variations often demand a nuanced appreciation of both the technical and contextual aspects surrounding BI systems. While technical data provides the foundation for metrics and benchmarks, the broader business context within which an IS operates can offer valuable insights into how these benchmarks should be interpreted and applied.

When employing historical data and statistical methodologies, it’s imperative to account for changing business landscapes and technological advancements. What was deemed an acceptable benchmark a few years ago may no longer hold relevance in the face of recent innovations and shifts in industry standards? This underlines the importance of ensuring that benchmarks are not just historically grounded but are also forward-looking, taking into consideration projected trends and anticipated evolutions in the IS domain.

Similarly, while competitive benchmarking offers a relative perspective on performance, it’s crucial to understand the unique challenges and opportunities that an individual organization might face. Blindly emulating industry standards without accounting for specific organizational contexts might lead to misaligned priorities and strategies.

Expert judgment, while invaluable, brings with it the need for critical evaluation. Experts, despite their depth of knowledge, come with their biases and perspectives. Therefore, a diverse panel of experts is often recommended to ensure a comprehensive and balanced view. Engaging experts from various domains— such as data science, business strategy, and user experience—can provide a multi-faceted perspective on setting benchmarks.

The emphasis on user feedback, meanwhile, underlines the growing importance of user-centricity in modern IS. As these systems increasingly cater to non-technical stakeholders, ensuring that benchmarks resonate with user expectations and experiences is paramount. It’s not just about how efficiently a system processes data but also about how effectively it communicates insights to its users.

Furthermore, the iterative nature of benchmark refinement speaks to the continuous improvement paradigm inherent in effective IS practices. As organizations grow, their data needs evolve, and BI systems must adapt in tandem. Regularly revisiting benchmarks ensures alignment with current organizational objectives and paves the way for sustained excellence.