The source of unprecedented data growth in recent years is many and varied. International Data Corporation (IDC) estimated that the Global Datasphere, a measure of all new data collected, created, and replicated in a year across the globe, will grow from 33 Zettabytes (10²¹ bytes) in 2018 to 175 by 2025 (Reinsel et al., 2018). A zettabyte is a trillion Gigabytes.

In May 2020, IDC published an update. The estimated growth of data for the year 2020 alone was 59 Zettabytes with a forecast of continued growth through 2024 with a five-year compound annual growth rate (CAGR) of 26%. The COVID-19 pandemic contributes to this figure by causing an abrupt increase in the number of work-from-home employees and rapid digitization (IDC, 2020).

A vast majority of this data is transmitted, processed, and stored at enterprise data centers. The remarkably high rate of new data creation means there need to be more storage systems in the enterprise data centers. These days, enterprise IT infrastructure consists of tens of thousands of physical and virtual servers and their associated hardware like servers, network equipment, and storage systems spread across geographies. The amount of data stored in each data center is in multiple Petabytes (10¹⁵ bytes).

Storage systems consist of dedicated servers, storage media, and related software to obtain a high-performance, high availability, and efficiently managed system. The main types of storage media used in data centers are tape drives, magnetic hard drives, and solid-state drives. Enterprise-grade hardware is meant to run continuously, twenty-four hours a day, seven days a week.

Storage systems have developed over decades, improving performance, cutting cost, and, most importantly, enabling modern computing needs by supporting the new types of workloads. The recent industry trend in IT infrastructure management emphasizes automation of the daily repetitive tasks through various commercially available software or homegrown scripts. Another trend in enterprise IT is to leverage public cloud storage for offloading some or most management responsibilities to a third party.

Enterprise data storage systems go through hardware refresh every three to five years to take advantage of newly available features and reduce risks due to aging hardware. These are multi-million-dollar decisions involving implementation and data migration plans that span months to years.

Technology assessment is the evaluation and estimation of the nature, quality, or ability of the technology. It started as a form of public policy research to examine various short and long-term consequences of technology use. Such analysis requires consideration of multiple perspectives and criteria.

Multiple Criteria Decision Making (MCDM) is one of the most widely used decision methodologies in various fields that aim to satisfy the multitude of conflicting objectives in the best possible way. We derive measurements by directly comparing objects. Thomas L. Saaty established that direct comparisons are necessary to establish measurements for intangible properties with no scales of measurement (Saaty, 2008). Methods based on pairwise comparison form a significant part of multiple criteria decision making.

Decision-makers need models that are updated, capturing all significant perspectives and criteria. IT executives in the decision-making positions often supplement their knowledge with advice from experts in the field. This study discusses the use of expert judgment in the assessment of enterprise data storage systems. Expert judgment can be defined as an expert opinion given in the context of a specific decision problem. Expert judgment is a recognized, mature research methodology suitable for assessing emerging technology where benchmarks have not been established or no objective data is available. Expert judgment quantification utilizes rating instruments in the form of a questionnaire to convert informed estimates from the experts to numeric values. This study uses a constant sum pairwise comparison to record one criterion or perspective’s importance versus another. The information that the experts provide becomes data. Conclusions are drawn by combining expert judgments as an aggregation of quantitative estimates.

In this study, we map the process of expert judgment to validate the STORE assessment model for assessing enterprise data storage systems (Shrestha & Sheikh, n.d.). We also used quantification of expert judgment in finding the criteria weights. This study speaks primarily to the decision-makers who fill one of these roles:

· Senior Executives responsible for leadership and technology purchase decisions

· Storage Managers responsible for providing the storage services

· Storage Architects responsible for the design of storage solutions

· Storage Engineers engaged in the implementation of storage solutions

· IT Operations Staff responsible for the daily operations

We performed a literature review to understand and develop a scholarly base for the three related research areas: Enterprise IT, Data Storage, and Technology Assessment as shown in Figure 1 (Shrestha & Sheikh, n.d.).

The review enabled identifying gaps in prior research, specifically, the lack of a comprehensive decision model, covering Strategic, Technological, Operational, Regulatory, and Economic (STORE) perspectives. Each of the STORE perspectives was subjectively categorized and considered a preferentially orthogonal dimension. Preferentially orthogonal perspectives are the independent dimensions. The combined dimensions are then deemed to be comprehensive for the assessment of EDSS.

Strategic perspective considers high and low level, short- and long-term goals of the organization. Technological perspective considers various criteria that relate to the capability and efficiency of the storage system. The operational perspective considers how the product will affect the day-to-day operations of the IT function. The regulatory perspective considers the legal aspects of technology implementation. The economic perspective considers the financial aspects of the solution. We derived twenty criteria by grouping the related concepts and categorized them under the five STORE perspectives.

This study expands our literature review to expert judgment quantification and constant sum pairwise comparison in finding criteria weights for multiple criteria decision making. We explore the recent use of these methods in the information technology domain.

Multiple criteria decision making refers to all methods that help designate a preferred alternative and rank alternatives based on subjective preferences, where there is more than one criterion (Ho, 2008). Some authors (Zimmermann, 1991; Chen & Hwang, 1992) categorized MCDM into two categories: 1) Multi-Attribute Decision Making (MADM) problems, where the number of alternatives is predetermined, and 2) Multi-Objective Decision Making (MODM), where they are not.

MCDM has been used in the study of cloud service selection (Rehman et al., 2011), IT infrastructure selection for smart grid (Rezagholizadehl et al., 2013), big data storage selection (Kachaoui & Belangour, 2019), IT disaster recovery site selection (Yang et al., 2015) and justification of IT investments (Borenstein & Betencourt, 2005).

Expert judgment quantification adds substantial value in analyzing complex problems when there are no universally accepted scientific laws or extensive data available. Keeney & Von Winterfeldt (1989) stressed the value of quantifying expert judgments to complement the expert’s qualitative thinking and reasoning. They also highlighted the need for explicit judgments to avoid misinterpretations and misuse.

Expert judgment has been studied in various fields, including cybersecurity (Holm et al., 2014), web development projects (Torrecilla-Salinas et al., 2019), regression models of software effort estimation (Tsunoda et al., 2012), the potential of blockchain in supply chain management (Kopyto et al., 2020) and addressing uncertainty in high technology system design (Chytka et al., 2006).

The constant sum pairwise comparison method is used in the scientific study of preferences, attitudes, and requirements engineering. It reflects the importance or priority attached by a respondent to one entity compared to another. The number of independent pairwise comparisons for n number of entities is n(n − 1)/2. In multiperspective hierarchical decision making, criteria are the lower-level entities to perspectives in the hierarchy. Constant sum pairwise comparison has a relative orientation providing more decision context than binary choices. It is unavoidably more complex than a binary or discrete choice task, resulting in inattention or higher drop-out rates (Skedgel & Regier, 2015).

Pairwise comparison has been used in various fields, including IT infrastructure refresh planning for enterprises (Daim et al., 2011), engineering design (Dym et al., 2002), document ranking algorithms in information retrieval systems (Ozbey & Dincsoy, 2020), intelligent transportation recommendation systems (Borodinov et al., 2020), and image quality assessment (Zhang et al., 2017).

Figure 2 shows the STORE assessment model with five perspectives and twenty criteria in a hierarchy (Shrestha & Sheikh, n.d.). Note that certain aspects of a criterion are covered by others in the model. For example, technical aspects of data security under regulatory perspective are considered in technology features under technological perspective.

Short definitions of the perspectives and criteria:

Strategic Perspective: Strategic perspective considers high and low level, short- and long-term goals of the organization.

Technological Perspective: Technological perspective considers various criteria that relate to the storage system’s capability and efficiency.

Operational Perspective: Operational perspective considers how the product will affect the day-to-day operations of the IT function.

Regulatory Perspective: Regulatory perspective considers the legal compliance aspects of technology implementation. Note that some regulations like GDPR cover more than one criterion in the STORE regulatory perspective.

Economic Perspective: Economic perspective considers the financial aspects of the solution.

Business Strategy: Business strategy is a set of guiding principles that aim to achieve business objectives like service enablement, revenue growth, and cost-saving.

Technology Strategy: Technology strategy explains how we should utilize technology as part of an organization’s business strategy.

Organizational Readiness: Organizational readiness refers to the availability of a skilled workforce and defined processes for technology implementation and operations.

Technology Feature: Technology features of storage systems include policy-based provisioning, orchestration, storage snapshot, and replication.

System Performance: System performance of storage systems includes throughput in IOPS and latency.

System Reliability: The system reliability of a storage system is a measure of performing consistently well. It contributes directly to the high availability of applications supported by the system.

Capacity Management: Capacity management refers to the ease with which capacity is expanded when needed. Capacity management is aided by data reduction techniques like compression and deduplication.

Technological Complexity: Technological complexity refers to a difficulty understanding the storage system and its interaction with other IT components.

Storage Implementation: Storage implementation for an application involves setting up the physical hardware and cables, configuring the device, testing, and migrating data.

Storage Administration: Storage administration tasks include provisioning storage, creating storage units, and performing cleanups.

System Monitoring: System monitoring is essential for the fine-tuning of storage systems and identifying performance bottlenecks.

System Reporting: System reporting is essential for keeping track of how well the system fulfills the needs. It also helps in making decisions related to capacity expansion and others.

Vendor Support: Vendor support is characterized by the ease of creating service requests, communicating with support engineers, and a clear escalation path.

Data Privacy: Data privacy relates to the protection of consumer data. Examples of the regulations that might be applicable are European Union General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and others.

Data Security: Data security refers to the prevention of unauthorized access. Laws like the Cybersecurity Information Sharing Act (CISA) equip organizations to secure the data from the latest cyber threats.

Data Retention: Data retention means the safe keeping of data for future access. Examples of regulations that might be applicable are the Sarbanes Oxley Act (SOX) and Health Insurance Portability and Accountability Act (HIPPA).

Data Transfer: Data transfer refers to the mobility of data. An example of data transfer regulation is cross-border data transfers under GDPR.

Capital Expense: Capital expenses (CAPEX) consist of purchasing equipment or services towards fixed assets that the company will use beyond the current year.

Operating Expense: Operating expenses (OPEX) refers to the ongoing expenses for the operation and maintenance, including the cost of power, space, and cooling in the data center.

Total Cost of Ownership: Total cost of ownership (TCO) is a holistic view of the enterprise’s expenses over time.

Figure 3 shows the process of expert judgment quantification used in this research study. It involved five steps: Define Decision Problem, Recruit Experts, Design Research Instruments, Collect Expert Judgment, and Analyze Expert Judgment.