Establishing a scientifically justifiable system of evaluation indices is crucial for selecting and evaluating military software suppliers. Based on an initial screening of the evaluation indices, the grey-rough set method was used to reduce and select the evaluation indices. As a result, a two-stage evaluation index system covering both qualification examination and supplier evaluation was ultimately established, and the meanings and applications of each evaluation index were explained. The results show that the grey-rough set method can effectively reduce and screen the evaluation indices for military software suppliers.
In recent years, with the continuous progress of national defense and military modernization, military informatization construction has been improved ( Foreman, Favaró, Saleh, & Johnson, 2015 ). The number of development projects for military software has increased ( Cho, Hwang, Shin, Kim, & In, 2021 ), and an increasing number of software vendors have participated in the development and maintenance of military software systems ( Merola, 2006 ). Selecting the most suitable software vendor from a multitude of suppliers has become the focus of attention for military units.
Regarding the issue of supplier evaluation and selection, numerous domestic and foreign scholars have conducted extensive research. In 1966, Dickson (1966) published a research article titled “An Analysis of Vendor Selection Systems and Decisions”, which established a pioneering foundation for research on supplier evaluation and selection by constructing 23 evaluation indicators encompassing past performance, technical capability, after-sales service, etc., and ranking the importance of these indicators. In the research that followed over the past fifty years, studies on supplier evaluation and selection have covered various fields, including the general manufacturing industry, the traditional construction industry, the telecommunications service industry, the modern logistics industry, and others. In their study, Song, Wang, Guo, Lu and Liu (2021) employed a combination of the mechanism equation model (SEM) and intuitionistic fuzzy analytic hierarchy process (IFAHP) to conduct a comprehensive evaluation of prefabricated modular building suppliers. Uygun, Kacamak and Kahraman (2015) utilized a combination of the Decision-Making Trial and Evaluation Laboratory (DEMATEL) and Fuzzy Analytic Network Process (Fuzzy ANP) methods to conduct a study on the selection and evaluation of outsourcing suppliers for telecommunications companies. Ghorbani and Ramezanian (2020) designed a scenario-based two-stage stochastic programming model for the evaluation and selection of carriers in humanitarian relief operations.
Regarding the selection of software vendors, some scholars ( Li et al., 2021 ) focused on evaluating and selecting management software vendors, providing a scientific basis for procurement decisions in universities. Khan, Niazi and Ahmad (2011) identified through a systematic literature review (SLR) method the factors that negatively impact the selection of offshore software development outsourcing project vendors. Rashid, Khan, Khan and Ilyas (2021) designed and developed a multi-level agile green maturity model (GAMM) to assess the maturity level of global software vendors in agile software development. Some scholars ( Huang et al., 2018 ) constructed an evaluation indicator system for BIM software vendors based on the characteristics of BIM software, providing references for scientifically selecting BIM software vendors. Other scholars ( Wang et al., 2022 ) constructed an evaluation indicator system for third-party testing vendors of military software and validated the objectivity and usability of this indicator system through examples. Currently, there is relatively little research on the selection of military software vendors. Due to significant differences between military software projects and general software service projects, such as high confidentiality requirements, long service cycles, and complex technical performance, issues arise when applying traditional software supplier evaluation and selection methods. Therefore, it is necessary to establish a targeted and practical evaluation indicator system for selecting military software vendors.
The structure of this article comprises several sections. The introduction in Chapter 1 primarily presents the research purpose and background. Chapter 2 focuses on the literature review, specifically reviewing important literature in the field of supplier selection. Chapter 3 briefly discusses the composition and structure of the article. In Chapter 4, the fundamental principles and main processes of constructing the evaluation index system for military software suppliers are introduced. Chapter 5 involves the initial selection of evaluation indicators for military software suppliers. Chapter 6 begins by introducing the grey-rough set-based indicator screening method, followed by the screening of the evaluation index system for military software suppliers, along with explanations and applications of each indicator within the index system. Chapter 7 concludes the article and highlights the universality of the results.
The construction of the evaluation index system should fully consider the uniqueness of military software. Based on the analysis of evaluation factors mentioned earlier, the following principles are formulated:
1) Combining Practicality with Operability
The evaluation index system for military software suppliers should align with the practical context of evaluating these suppliers. The quantifiable parameters of the constructed indicators should be easy to collect and calculate, enabling their practical application in the selection process of military software suppliers. The evaluation results should comprehensively and objectively reflect the suppliers’ overall capabilities, assisting military units in identifying the best suppliers during the software outsourcing process.
2) Combining Scientific Rigor with Purposefulness
Scientific rigor and purposefulness should be considered when constructing the evaluation index system. The constructed index system must adhere to scientific principles, ensuring its rationality. Simultaneously, it must also align with the purpose of supplier evaluation, facilitating subsequent supplier selection. The selection of indicators should be scientifically reasonable, accurately reflecting the characteristics of military software suppliers. The construction of indicators should exhibit distinct hierarchies and differentiation.
3) Combining Universality with Specialty
When constructing evaluation indicators for military software suppliers, it is necessary to compare different types of military software and establish evaluation indicators that encompass a wide range and have common characteristics. This ensures that the evaluation indicators apply to various types of military software outsourcing projects. Additionally, it is important to set evaluation indicators distinct from those used for general software suppliers, taking into account the unique aspects of military software, thereby ensuring both universality and representativeness.
4) Combining Qualitative and Quantitative Approaches
The evaluation index system for military software suppliers should reflect various aspects of the suppliers’ capabilities. It should include both qualitative and quantitative indicators. The selection of quantitative indicators should ensure data accessibility and operational simplicity. For factors that cannot be quantitatively described, qualitative indicators should be used to provide a comprehensive reflection of the suppliers’ overall capabilities. It is important to define the relevant meanings of the indicators and quantify them through expert ratings or other methods.
Through comprehensive analysis of the evaluation factors for military software suppliers, combined with current research on software supplier evaluation selection at home and abroad, as well as actual investigations of military units, the initial selection of evaluation indicators is conducted. The grey-rough set method is employed to optimize and reduce the indicators, ultimately determining the evaluation indicator system for military software suppliers. The construction process of the evaluation indicator system for military software suppliers is illustrated in
1) Analysis of Evaluation Factors and Initial Selection of Indicators
By collecting, summarizing, and integrating research on software supplier selection both domestically and internationally, and combining it with the research findings from relevant military units, a preliminary evaluation indicator system for military software suppliers is synthesized and organized.
2) Optimization of Indicators Based on Grey-Rough Set Theory
The selected military software suppliers are subjected to research and analysis. The “Survey Questionnaire for the Construction of Evaluation Indicator System for Military Software Suppliers” is designed, and experts, project managers, procurement personnel, and military software suppliers involved in software engineering project management are invited to rate the indicators. By employing the combined approach of grey correlation analysis and rough set theory, the evaluation indicators are reduced and representative indicators with strong representativeness for evaluating military software suppliers are determined.
3) Determination and Analysis of Evaluation Indicators
The selected indicators are explained and analyzed to establish quantitative calculation or qualitative judgment methods for each indicator. Ultimately, a comprehensive evaluation indicator system for military software suppliers is formed, enabling it to fully reflect the suppliers’ comprehensive capabilities and provide practical guidance.
Based on the analysis of evaluation factors for military software suppliers, combined with the relevant research literature on software supplier evaluation and selection both domestically and internationally, as well as actual surveys conducted in military units, the evaluation indicators for selecting military software suppliers are preliminarily selected by distinguishing between supplier qualification review and supplier evaluation and selection, following the actual steps of selecting military software suppliers. The framework for the initial selection of supplier indicators is presented in
| Stage | Level 1 Indicators | Level 2 Indicators | Stage | Level 1 Indicators | Level 2 Indicators |
|---|---|---|---|---|---|
| Supplier Qualification Review | Supplier Basic Qualification (S1) | Business Qualification (S11) | Supplier Evaluation Criteria | Product Technical Solution (B) | System Stability (B4) |
| Enterprise Investment (S12) | System Fault Tolerance (B5) | ||||
| Confidentiality Qualification (S13) | Security And Confidentiality (B6) | ||||
| Establishment Time Requirement (S14) | Scalability (B7) | ||||
| Independence (S15) | Usability (B8) | ||||
| Supplier Integrity Qualification (S2) | Intellectual Property Rights (S21) | Service Level (C) | Support Hours (C1) | ||
| Reputation (S22) | Efficiency (C2) | ||||
| Taxation And Social Security Payments (S23) | Inspection Intensity (C3) | ||||
| Supplier Technical Qualification (S3) | Capability Maturity Certification (S31) | Service Attitude (C4) | |||
| Quality Management (S32) | Training Program (C5) | ||||
| Organisational Working Environment (S33) | Response Time (C6) | ||||
| Supplier Evaluation Criteria | Supplier Strength (A) | Company Size (A1) | Product Pricing (D) | Initial Purchase Price (D1) | |
| Operation and Maintenance Price (D2) | |||||
| Market Share (A2) | Personnel Training Price (D3) | ||||
| Financial Condition (A3) | Software Upgrade Price (D4) | ||||
| Past Performance (A4) | Implementation Capability (E) | Project Management Capability (E1) | |||
| Technical Capability (A5) | Project Implementation Timeline (E2) | ||||
| Product Technical Solution (B) | Solution Reliability (B1) | Emergency Change Capability (E3) | |||
| Functional Completeness (B2) | Project Human Resource Input (E4) | ||||
| Structure Rationality (B3) | Communication and Coordination Ability (E5) | ||||
| System Stability (B4) | Personnel Skill Level (E6) |
The previous text discussed the initial selection of evaluation indicators for selecting military software suppliers. However, during the practical application, the inherent relationships and logical redundancies between these indicators, as well as their appropriateness for evaluation, may have been overlooked. These factors could affect the accuracy of the evaluation and make it difficult to directly apply the selected indicators. Therefore, it is necessary to further screen the initial set of indicators.
Common methods for simplifying indicator systems include Analytic Hierarchy Process (AHP), Principal Component Analysis (PCA), Factor Analysis (FA), and Linear Discriminant Analysis (LDA). However, traditional evaluation methods like AHP tend to be subjective, PCA can only handle linearly correlated problems, and FA and LDA require a high sample size with a large amount of data. Hence, this study adopts a combination of Grey Relation Analysis (GRA) and Rough Set Theory (RST) to simplify the evaluation indicator system.
Grey Relation Analysis (GRA) is a method that uses grey system theory to characterize the influence of multiple factors on the target factor. It has the advantages of simplicity, wide applicability, robustness, and ease of interpretation. Rough Set Theory (RST), on the other hand, is a data model used to analyze and process incomplete and uncertain data. It possesses strong feature extraction capabilities, good interpretability, and simple and reliable algorithms.
By combining these two methods, it is possible to calculate the correlation among the various indicators while identifying redundant ones. This allows for the selection and optimization of the indicator system. The specific calculation process is as follows:
STEP 1: Establish the rating matrix.
Through questionnaire surveys, experienced experts in the field of supplier selection are invited to rate the initial indicators based on four dimensions: representativeness, necessity, scientificity, and systematicity. Each dimension is scored out of 25, with a total of 100 points. The total score for each indicator across the four dimensions represents the expert’s rating. The evaluation indicator system for military software suppliers is treated as a multi-attribute decision information system.
S = { U , A , V , f }
U is the set of experts, A is the set of indicator attributes, A = C ∪ D is the set of attributes, C is the subset of conditional attributes (expert attributes), D is the subset of decision attributes (indicator attributes), V is the set of attribute values, f : U × A → V and is an information function, i.e., the attribute value of each object (the result of the scoring of the indicator by each expert).
Due to the differences in scoring among various indicators, it is necessary to standardize the raw data of expert ratings. The range method, known for its simplicity, applicability, and preservation of the original data distribution, is widely used for data standardization in various scenarios. Therefore, this article employs the range method for data standardization. As the experts’ ratings reflect the importance of each indicator and all indicators are considered beneficial, standardization should be conducted using Equation (1).
w i ( j ) = v i ( j ) − min i ( v i ( j ) ) max i ( v i ( j ) ) − min i ( v i ( j ) ) (1)
v i ( j ) represents the combined score of the expert on the indicator, and the normalized scoring matrix is W = [ w i j ] , where i = 1 , 2 , ⋯ , m ; j = 1 , 2 , ⋯ , n .
STEP 2: Establishing the correlation matrix.
Let’s set it W 0 = { w 01 , w 02 , ⋯ , w 0 n } as the reference data column. For any i ≤ j , i , j = 1 , 2 , ⋯ , m , we can obtain the correlation coefficient of the comparative data column Z i concerning the reference data column Z 0 in terms of the indicator k using Equation (2).
θ i ( k ) = min i min j | w i ( j ) − w 0 ( j ) | + ρ max i max j | w i ( j ) − w 0 ( j ) | | w 0 ( k ) − w i ( k ) | + ρ max i max j | w 0 ( j ) − w j ( j ) | (2)
Here, ρ ∈ [ 0 , 1 ] represents the discrimination coefficient. The correlation coefficients can be used to construct the correlation matrix Z, as shown in Equation (3).
Z = ( θ 11 θ 12 ⋯ θ 1 n θ 21 ⋯ θ 2 n ⋱ ⋮ θ n n ) (3)
STEP 3: Determining the optimal threshold using the F-statistic.
Since the classification of indicators can be influenced by the threshold λ , this study introduces the F-statistic method to determine the threshold to achieve more scientifically and objectively classified results.
Assuming is the set of evaluation objects to be classified, for any of B i , where represents the evaluation object score for the first indicator (where i = 1 , 2 , ⋯ , m ; k = 1 , 2 , ⋯ , n ).
Assuming that the number of categories under the threshold λ is r, the calculation of the average value of the scores of the objects in the first category in the indicator is shown in Equation (4) and represents the number of objects included in the first category j.
b ¯ i k = 1 o j ∑ i = 1 o j b i k , k = 1 , 2 , ⋯ , n (4)
The average of the scores of all evaluation subjects on the indicators is calculated as shown in Equation (5).
b ¯ k = 1 m ∑ i = 1 m b i k , k = 1 , 2 , ⋯ , n (5)
And then the F-statistic can be calculated:
F = ∑ j = 1 r o j ∑ k = 1 n ( y ¯ i k − y ¯ k ) 2 / ( r − 1 ) ∑ j = 1 r ∑ i = 1 o j ∑ k = 1 n ( y ¯ i k − y ¯ k ) 2 / ( m − r ) (6)
Equation (6), m represents the total number of evaluations to be classified, and obeys the distribution, in which the denominator represents the distance of the samples within the group and the numerator represents the distance of the samples between the groups, so the larger the value, the better the classification effect. According to the relevant knowledge of significance test in mathematical statistics, if F > F α ( r − 1 , n − r ) , where α = 0.05 , indicating that the difference between the groups is more obvious, the classification is relatively reasonable if at the same time, there is more than one value to meet the inequality F > F α ( r − 1 , n − r ) , it is necessary to further calculate the value of ( F − F α ) / F α , and select the F value with larger calculation results.
STEP 4: Indicator reduction based on rough set theory
As mentioned earlier, S = { U , A , V , f } let be an information system. When a subset of attributes P、Q ( ∀ P ⊆ Q ) is taken from this system, the indiscernibility relation P is denoted as I n d ( P ) . It divides the universe of discourse U into k equivalence classes, which can be represented as:
U / I n d ( P ) = { W 1 , W 2 , ⋯ , W k } (7)
In an information system S = { U , A , V , f } , assume that H ∈ A , H is an equivalence relation and h ∈ H , if I n d ( H ) = I n d ( H − { h } ) , h is said to be redundant in H, and vice versa, h is said to be necessary for H, and if all h are necessary for H, H is said to be independent. If two equivalence relations on a domain satisfy the conditions that M ∈ N , are independent I n d ( N ) = I n d ( M ) , then the domain U is said to be approximately reduced over the set N of attributes as a Property set M.
STEP 5: Comprehensive evaluation analysis of indicators.
This study adopts the method of calculating weights using rough set theory. The weights of the indicators are calculated based on the experts’ ratings. The specific calculation process is as follows:
In the information system S = { U , A , V , f } , A = C ∪ D is a set of attributes, in the measurement of the importance of each attribute to introduce the concept of information quantity, if K ⊆ A , and U / I n d ( K ) = { x 1 , x 2 , ⋯ , x n } , then the information quantity of K can be derived from Equation (8).
I ( K ) = ∑ i = 1 n | x i | | U | [ 1 − | x i | | U | ] = 1 − 1 | U | 2 ∑ i = 1 n | x i | 2 (8)
C = { c 1 , c 2 , ⋯ , c n } is a subset of the conditional attributes and c i is a sub-attribute, then the importance relative S i g c ( c i ) to can be found as shown in Equation (9).
S i g c ( c i ) = I ( C ) − I ( C − { c i } ) (9)
Then the weights of the sub-attributes ω i are calculated as:
ω i = S i g c ( c i ) ∑ i = 1 n S i g c ( c i ) (10)
It is further possible to calculate a composite assessment value for the indicator:
S j = ∑ i = 1 n ω i s i j (11)
Equation (11), S j represents the j comprehensive evaluation result of the first indicator and s i j represents the attribute value of the first indicator under the first attribute.
STEP 1: Establishing a scoring matrix
Due to the numerous criteria involved in the supplier qualification review stage, the 11 secondary indicators in the table are denoted for ease of subsequent calculations R 1 ~ R 11 . Through a questionnaire survey, 7 experienced experts J 1 ~ J 7 specializing in supplier selection research were invited to rate the relevant indicators across 4 dimensions. The scoring values provided by each expert for each indicator were recorded (with each dimension ranging from 0 to 25, totaling 100 points).
Based on the formula, the data has been standardized. The standardized data is presented in
STEP 2: Establishing the Association Matrix
By applying formula (2), the grey correlation matrix is computed for all
| expert | Evaluation indicators | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R110 | R11 | |
| J1 | 95 | 87 | 90 | 72 | 88 | 81 | 89 | 91 | 84 | 84 | 82 |
| J2 | 92 | 84 | 92 | 75 | 90 | 69 | 84 | 89 | 85 | 91 | 92 |
| J3 | 84 | 87 | 94 | 86 | 88 | 86 | 90 | 82 | 89 | 86 | 87 |
| J4 | 90 | 90 | 92 | 65 | 89 | 78 | 82 | 83 | 90 | 84 | 88 |
| J5 | 92 | 93 | 85 | 82 | 82 | 68 | 91 | 86 | 85 | 86 | 86 |
| J6 | 90 | 88 | 96 | 69 | 95 | 77 | 88 | 93 | 88 | 88 | 88 |
| J7 | 92 | 93 | 94 | 79 | 94 | 79 | 88 | 89 | 90 | 85 | 91 |
| expert | Evaluation indicators | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | R11 | |
| J1 | 1.00 | 0.33 | 0.45 | 0.33 | 0.46 | 0.72 | 0.78 | 0.82 | 0.00 | 0.00 | 0.00 |
| J2 | 0.73 | 0.00 | 0.64 | 0.48 | 0.62 | 0.06 | 0.22 | 0.64 | 0.17 | 1.00 | 1.00 |
| J3 | 0.00 | 0.33 | 0.82 | 1.00 | 0.46 | 1.00 | 0.89 | 0.00 | 0.83 | 0.29 | 0.50 |
| J4 | 0.55 | 0.67 | 0.64 | 0.00 | 0.54 | 0.56 | 0.00 | 0.09 | 1.00 | 0.00 | 0.60 |
| J5 | 0.73 | 1.00 | 0.00 | 0.81 | 0.00 | 0.00 | 1.00 | 0.36 | 0.17 | 0.29 | 0.40 |
| J6 | 0.55 | 0.44 | 1.00 | 0.19 | 1.00 | 0.50 | 0.67 | 1.00 | 0.67 | 0.57 | 0.60 |
| J7 | 0.73 | 1.00 | 0.82 | 0.67 | 0.92 | 0.61 | 0.67 | 0.64 | 1.00 | 0.14 | 0.90 |
indicators. The optimal grey correlation effect is achieved when the resolution coefficient is ρ = 0.5 . Consequently, the grey correlation matrix is obtained. Based on this correlation matrix, the dynamic clustering of supplier qualification review indicators is generated, as depicted in
Z = [ 1 0 .6238 0 .6182 0 .6059 0 .6364 0 .6455 0 .6246 0 .7217 0 .5181 0 .5768 0 .6616 1 0 .6051 0 .6224 0 .6442 0 .6436 0 .6309 0 .5007 0 .6453 0 .5504 0 .6595 1 0 .6095 0 .8846 0 .7022 0 .5959 0 .6784 0 .6675 0 .5182 0 .6595 1 0 .5665 0 .6398 0 .7496 0 .6140 0 .5501 0 .5778 0 .5463 1 0 .6705 0 .5411 0 .6865 0 .6225 0 .5531 0 .7102 1 0 .7036 0 .5977 0 .6527 0 .5241 0 .5995 1 0 .6659 0 .6428 0 .5680 0 .5455 1 0 .5030 0 .6211 0 .5852 1 0 .6002 0 .7040 1 0 .7589 1 ]
STEP 3: Determining the Optimal Threshold through F-Statistic
To ensure a more scientific and objective classification of indicators, this study employs the F-statistic method to determine the threshold for evaluating the indicator system. By utilizing formulas (4), (5), and (6),
The optimal threshold is determined by selecting the maximum value ( F − F α ) / F α from the table, which corresponds to λ . From
U / J = { { R 10 , R 11 } , { R 9 } , { R 6 } , { R 4 , R 7 } , { R 3 , R 5 } , { R 2 } , { R 1 , R 8 } }
STEP 4: Indicator Reduction Based on Rough Set Theory
Using the same method, the optimal classification results can be obtained by sequentially removing the ratings of each expert.
STEP 5: Comprehensive Evaluation Analysis of Indicators
By applying formula (8), the information content of the optimal classification
| classifications | F-statistic | F α | ( F − F α ) / F α |
|---|---|---|---|
| 2 | 8.0884 | 5.12 | 0.37 |
| 3 | 12.3106 | 4.46 | 0.64 |
| 4 | 18.6273 | 4.35 | 0.77 |
| 5 | 18.6011 | 4.53 | 0.76 |
| 6 | 19.9154 | 5.05 | 0.75 |
| 7 | 27.5528 | 6.16 | 0.78 |
| 8 | 25.0542 | 8.89 | 0.65 |
| 9 | 22.3207 | 19.37 | 0.13 |
| 10 | 81.5716 | 240.54 | — |
after sequentially removing attributes can be obtained.
By substituting the above results into formulas (9) and (10), the importance and weights of each attribute can be calculated.
According to formula (11), the comprehensive evaluation results for each indicator in the supplier qualification review stage can be obtained.
From the above evaluation, it is evident that the two indicators, R4 and R6,
| Index | Exclusion of Indicators | Optimal Classification |
|---|---|---|
| 1 | U / ( J − J 1 ) | { { R 10 } , { R 9 } , { R 6 } , { R 4 , R 7 } , { R 3 , R 5 , R 11 } , { R 2 } , { R 1 , R 8 } } |
| 2 | U / ( J − J 2 ) | { { R 10 } , { R 9 , R 11 } { R 8 } , { R 4 , R 7 } , { R 3 , R 5 , R 6 } , { R 2 } , { R 1 } } |
| 3 | U / ( J − J 3 ) | { { R 9 , R 10 , R 11 } , { R 6 } , { R 4 , R 7 } , { R 3 , R 5 , R 8 } , { R 2 } , { R 1 } } |
| 4 | U / ( J − J 4 ) | { { R 9 , R 10 , R 11 } , { R 4 , R 6 , R 7 } , { R 2 } , { R 1 , R 3 , R 5 , R 8 } } |
| 5 | U / ( J − J 5 ) | { { R 10 , R 11 } , { R 9 } , { R 4 , R 6 , R 7 } , { R 3 , R 5 } , { R 2 } , { R 1 , R 8 } } |
| 6 | U / ( J − J 6 ) | { { R 10 } , { R 9 } , { R 6 } , { R 4 , R 7 } , { R 3 , R 5 , R 11 } , { R 2 } , { R 1 , R 8 } } |
| 7 | U / ( J − J 7 ) | { { R 10 , R 11 } , { R 9 } , { R 4 , R 7 } , { R 3 , R 5 , R 6 } , { R 2 } , { R 1 , R 8 } } |
| Classifications | I ( J ) | I ( J − J 1 ) | I ( J − J 2 ) | I ( J − J 3 ) | I ( J − J 4 ) | I ( J − J 5 ) | I ( J − J 6 ) | I ( J − J 7 ) |
|---|---|---|---|---|---|---|---|---|
| Information Content | 102 121 | 100 121 | 100 121 | 96 121 | 86 121 | 98 121 | 100 121 | 98 121 |
| Statistics | Attribute Importance | Weights | Statistics | Attribute Importance | Weights |
|---|---|---|---|---|---|
| J1 | 2 121 | 0.06 | J5 | 4 121 | 0.11 |
| J2 | 2 121 | 0.06 | J6 | 2 121 | 0.06 |
| J3 | 6 121 | 0.17 | J7 | 4 121 | 0.11 |
| J4 | 16 121 | 0.44 |
| indicators | Evaluation results | Comprehensive Ranking | indicators | Evaluation results | Comprehensive Ranking |
|---|---|---|---|---|---|
| R1 | 90.74 | 2 | R7 | 86.73 | 7 |
| R2 | 90.39 | 3 | R8 | 86.09 | 9 |
| R3 | 92.83 | 1 | R9 | 89.4 | 5 |
| R4 | 73.89 | 11 | R10 | 86.17 | 8 |
| R5 | 89.86 | 4 | R11 | 88.7 | 6 |
| R6 | 78.73 | 10 |
rank last, and their overall evaluation values differ significantly from the other indicators. These indicators are considered non-core. The indicator regarding the establishment time requirement lacks the necessary relevance for supplier qualification review, while the indicator on reputation includes content related to the examination of intellectual property rights R6 and should be removed. Thus, the final set of indicators is obtained.
Similarly, for the evaluation and selection stage of suppliers, a reduction and screening of indicators can be performed. The calculation results indicate that the four indicators, and rank last, and their overall evaluation values differ significantly from the other indicators. These indicators are considered non-core. The indicator A2 duplicates the content of market share and market size, while the indicator includes system fault tolerance B5, which is already covered by the indicator. The indicator is difficult to quantify due to the measurement of service attitude and should not be used as an evaluation indicator. Additionally, the indicators of personnel technical level and technical capability have similar content. Therefore, all four indicators should be removed.
Based on the previous application of the grey-rough set theory to screen and reduce the evaluation indicators for military software suppliers, the final determination of the evaluation indicator system for military software suppliers, along with their respective meanings and applications, is presented in
This paper focuses on the evaluation and selection characteristics of military software suppliers. Drawing on previous domestic and international research, the initial selection of evaluation indicators for military software suppliers is conducted. The grey-rough set method is employed to reduce and select the indicator system, resulting in the final construction of the evaluation indicator system for military software suppliers. The main conclusions are as follows:
| Stage | Primary Indicator | Secondary Indicator | Meaning and Application |
|---|---|---|---|
| Supplier Qualification Review | Supplier Basic Qualifications (C1) | Business Qualification (C11) | Examining whether the supplier has a business license, organization code certificate, and tax registration certificate (Unified Certificate). |
| Enterprise Investment (C12) | Reviewing the supplier’s provided declarations and information on shareholders or investors. | ||
| Confidentiality Qualifications (C13) | Examining whether the supplier has a second-level (or higher) confidentiality qualification certificate for weapons and equipment research and production units, whether they have passed military information security product certification, and whether the relevant certificates are valid. | ||
| Independence (C14) | Examining whether the supplier issues a declaration of independence and reviewing information on units with management relationships with the supplier. | ||
| Supplier Integrity Qualification (C2) | Reputation (C21) | Review the supplier’s records for the past three years to determine if there have been any significant violations of the law, major quality and safety incidents, or inclusion in any illegal dishonesty list. Additionally, request the supplier to provide evidence of their good reputation through successful bidding and transaction cases over the past three years. | |
| Taxation and Social Security Payments (C22) | Review the supplier’s documentation of total social security payments, personal income tax payment records, tax credit rating certificates, and audit reports for the past three years. | ||
| Supplier Technical Qualification (C3) | Capability Maturity Certification (C31) | Assess whether the software supplier possesses the capability maturity model for military software, such as GJB5000B-2021 or GJB5000A. | |
| Quality Management (C32) | Evaluate whether the supplier has obtained the GJB9001C quality management system certification and possesses valid certificates for quality management systems, information security management systems, and information technology service management systems. | ||
| Organizational Work Environment (C33) | Examine whether the supplier has the necessary equipment and a suitable working environment. | ||
| Supplier Evaluation Criteria | Supplier Strength (H1) | Company Size (H11) | Primarily assess the company’s asset size, sales revenue, employee count, and economic performance. |
| Financial Condition (H12) | Mainly evaluate the supplier’s financial management performance, including aspects such as assets, liabilities, equity, income, costs, and profits. | ||
| Past Performance (H13) | Focus on the supplier’s achieved business results over a certain period. | ||
| Technical Capability (H14) | Assess the supplier’s ability to utilize their professional knowledge, skills, and experience to develop products or provide services. | ||
| Product Technical Solution (H2) | Solution Reliability (H21) | Reliability refers to the ability of the solution to ensure stability, controllability, and predictability of quality and progress during software development. This ensures that the software can function and be maintained as required. | |
| Functional Completeness (H22) | Functional completeness refers to the ability of the software solution to meet various requirements in military application environments. For example, it should effectively coordinate, control, and monitor complex military systems, handle large amounts of data and signals specific to the military domain, and support rapid response and decision-making. | ||
| Structural Rationality (H23) | Structural rationality refers to the maintainability and ease of scalability of the software architecture and components within the solution. | ||
| System Stability (H24) | System stability refers to the ability of the software system to operate reliably and complete tasks in extremely complex and dynamically changing operational environments. | ||
| Security and Confidentiality (H25) | Security and confidentiality refer to the ability of the developed software system to protect classified information from unauthorized access, tampering, and destruction. | ||
| Scalability (H26) | Scalability refers to the ability of the military software system to quickly, flexibly, and cost-effectively expand with new functionalities and additional modules while meeting the basic requirements. | ||
| Usability (H27) | Usability refers to the extent to which the software interface design and user interaction methods align with the habits and practices of military personnel, while still meeting the functional requirements. | ||
| Service Level (H3) | Support Hours (H31) | Support hours refer to the time range during which the military software supplier provides technical support and issue resolution for military units. | |
| Efficiency (H32) | Efficiency refers to the speed at which the software supplier resolves reported faults or requests from military units. | ||
| Inspection Intensity (H33) | Inspection intensity refers to the scope, frequency, and quality of regular inspection services provided by the military software supplier to maintain the stability and security of the software system. | ||
| Training Program (H34) | A training program refers to a series of educational and training plans developed by the software supplier to enhance the understanding and usage of the software system by military personnel. | ||
| Response Time (H35) | Response time refers to the duration from the moment a military unit reports a software issue to the supplier until the supplier starts addressing and responding to the problem. | ||
| Product Pricing (H4) | Initial Purchase Price (H41) | The initial purchase price refers to the initial quotation provided by the software supplier when offering software products to military units. | |
| Operation and Maintenance Price (H42) | The operation and maintenance price refers to the cost charged by the supplier for providing ongoing support and maintenance services for the software product. | ||
| Personnel Training Price (H43) | The personnel training price refers to the cost charged by the software supplier for providing training services related to software usage, implementation, and basic maintenance to military units. | ||
| Software Upgrade Price (H44) | The software upgrade price refers to the cost charged by the software supplier for providing software upgrade services to customers. | ||
| Implementation Capability (H5) | Project Management Capability (H51) | Project management capability refers to the supplier’s ability to plan, organize, lead, and control software projects during the implementation process. | |
| Project Implementation Timeline (H52) | The project implementation timeline refers to the duration from project initiation to delivery and usage of the software project. | ||
| Emergency Change Capability (H53) | Emergency change capability refers to the supplier’s ability to respond quickly, handle, and deliver change requests in urgent situations. | ||
| Project Human Resource Input (H54) | Project human resource input refers to the number of manpower resources required in a software project and the amount of working time, effort, and labor costs expended by individuals during the project lifecycle. | ||
| Communication and Coordination Ability (H55) | Communication and coordination ability refers to the supplier’s capacity to establish and maintain efficient communication channels with military units during software development and implementation. It also includes actively coordinating various resources, personnel, and time factors. |
1) A grey-rough set model for indicator selection is constructed, which calculates the correlation between indicators and identifies redundant ones, leading to a more objective and fair reduction result.
2) An evaluation indicator system for military software suppliers is established, consisting of two stages: qualification review and evaluation selection. The meanings and applications of relevant indicators are explained, providing theoretical guidance for military units in selecting military software suppliers.
Overall, this study validates the applicability of the grey-rough set model in the process of indicator reduction. Additionally, the constructed indicator system aligns well with the procurement bidding process. By considering Supplier Strength, Product Technical Solution, Service Level, Product Pricing, and Implementation Capability, a comprehensive evaluation of military software suppliers is conducted. Moreover, while incorporating general supplier evaluation aspects, this study highlights the uniqueness of evaluating military software suppliers. The constructed indicator system demonstrates broad applicability and practicality, aiming to provide valuable insights for future research. Therefore, the study confirms the suitability of the grey-rough set model in indicator reduction and presents a comprehensive evaluation framework for military software suppliers.
The authors declare no conflicts of interest regarding the publication of this paper.
Wang, X. G., Hu, K. K., Zhang, X. R., Gou, Q. Y., Wang, W. L., Deng, M. Q., Zhou, X., Ma, T., & Zhang, Z. Y. (2023). Research on the Establishment of Evaluation Index System for Military Software Suppliers. Open Journal of Business and Management, 11, 1996-2013. https://doi.org/10.4236/ojbm.2023.115110