The automotive industry is increasingly investing in innovative solutions through embedded automotive software, which can create new business opportunities and revenue streams for companies [6] . Automotive software is characterized by specific traits, such as its heterogeneous nature, numerous communication processors, management of various vehicle variants, and specific safety and reliability requirements [7] . Additionally, automotive software often manages a vehicle domain but interacts with other domains like human-machine interface, powertrain, brakes, and telematics. Changes in automotive software requirements also affect interacting domains, emphasizing the need for seamless integration for comprehensive vehicle testing and approval. [1] highlights the main differences between automotive software and laptop/mobile software, particularly focusing on reliability, functional safety, and real-time behaviour. Automotive software must exhibit extreme reliability throughout a vehicle’s lifecycle, especially concerning functions like ABS and ESP, which must be fail-safe as they are activated when the driver is at risk. Moreover, real-time behaviour is critical, as seen in the gearbox’s software needing to execute gear change strategies within milliseconds to prevent mechanical damage to the vehicle’s powertrain or compromise its functional safety.

The software development process is a complex activity, which demands many hours of effort from different professionals with different degrees of knowledge and requires a lot of skill and interpretation in building the software. That is, there is the possibility of introducing errors in the software development process and to minimize the risks of these errors remaining in the software when delivered to the customer, a software testing process, also called V & V is performed [8] .

The V & V activities in software testing, according to [8] , can be divided into two main groups. The first group, called static testing, evaluates software using functional models or specifications without the need for the software to be operational. The second group, dynamic testing, requires functional software to evaluate it dynamically.

Type approval is a crucial process that verifies whether a software, system, or vehicle meets the minimum requirements set by legislation and is approved for market launch and operation [9] [10] . In the automotive field, there are two types of approvals as outlined by [11] : Third-party certification, where a certifying entity verifies the product against specified requirements, and self-approval, where the OEM itself approves the product and provides technical evidence to the certifying entity.

This research also incorporates the concept of ISO 26262 for functional safety in automotive systems, as discussed by [12] . ISO 26262 standardizes automotive functional safety, focusing on electrical and electronic components (E/E). The standard introduces the Automotive Safety Integrity Level (ASIL) classification system to categorize hazards related to E/E systems. ASIL levels range from A (lowest risk) to D (highest risk), with ASIL D requiring the most stringent safety measures due to its high-risk probability to users and bystanders [13] .

For the development of this work, a systematic literature review (SLR) entitled “VERIFICATION AND VALIDATION OF EMBEDDED SOFTWARE IN AN AUTOMOTIVE CONTEXT: A SYSTEMATIC LITERATURE REVIEW” was carried out and published by [14] and within the established selection criteria, two SLR were selected, one on software verification and another on the validation process for automotive engineering. A third work related to the objectives of the RSL was identified and added in the conduction of the RSL, as it was cited in several other articles that discuss and define concepts for Automotive Software Engineering (ASE).

The first SLR developed by [15] investigates the current V & V requirements and regulations, the weakness of the current process of testing and challenges and opportunities to ensure the safety of autonomous driving vehicles. In this review, databases such as SpringerLink, IEEE Xplore, ACM Digital Library and Wiley were consulted. After applying the Inclusion Criteria (IC) and Exclusion Criteria (EC), two hundred articles were obtained. It has been identified a substantial number of types of testing techniques, frameworks, and philosophies.

It was concluded that the ISO 26262 standard is not ideal for the V & V process of autonomous vehicles. This is because the requirements of an autonomous vehicle cannot be defined when test cases are initially defined because the behaviour of this type of vehicle is dependent on the environment in which it operates. This means that the vehicle’s response differs depending on the boundary conditions, which are not covered by ISO 26262. In SLR, there is a consensus that the fault injection method can encounter many faults in different environments and that a formal method of testing and mutation testing may be required for the V & V process of autonomous vehicles. In summary, [15] concluded that it is not possible to define all the requirements for the V & V process of autonomous vehicles because their response depends on the environment in which the vehicle is operating and cannot be predicted. This is one of the biggest challenges for testing this type of vehicle and for the overall development process. The ISO 26262 should be updated to accommodate guidelines for the development of autonomous vehicles.

[16] carried out a second SLR to identify the research activity intensity, most common topics, research method types, contribution of studies, challenges, and promising future in the Automotive Software Engineering (ASE) domain. They collected 679 articles from various areas of research. At SLR it was demonstrated that the literature on Automotive Software Engineering (ASE) in the 1990s increased substantially. After that decade, scholarly interest waned, although it revived from 2007 onwards. After that date, research interest in ASE increased constantly, and they also observed an evolution in the topics of the articles over the years. In the 1990s, the most common topics related to the compliance support group, project support group, and requirements engineering. After 2007, the most common topics were systems/software architecture and design, followed by systems/software testing. Therefore, the most common types of studies published were evaluation surveys, which typically involved implementing a proposed solution and presenting the results obtained through case studies or surveys. In relation to the most relevant topics, proposals and lessons learned were structured, which highlighted the importance of good management of the development and testing of the ASE. Although the authors have demonstrated that the ASE theme gains relevance in the academic environment, such studies often do not consider the practical applicability from the industrial point of view. Therefore, they concluded that topics such as V & V software should have empirical evaluations of the proposed methods under real practical conditions that can happen in the industry domain.

A third work added to this SLR was published by [7] , which investigates the characteristics of Automotive Software. They described several features of ASE and distinguished between regular software and automotive software. A characteristic of automotive software is its heterogeneous nature due to different functionalities, e.g. powertrain, infotainment, driver assistance. This heterogeneity results in a lengthy system integration process, many vehicle variants with which the software needs to be compatible, and specific reliability and safety requirements.

These studies by [15] - [17] provided a good overview and were a good reference for the development of a testing process for automotive software domain. These studies highlighted gaps in standards for self-driving cars, the necessity for more research in ASE, and distinctions between automotive and regular software. However, other methodology techniques were applied in the development of this work and will be presented in the following sections.

The utilization of software to facilitate technologies in a flexible manner is prevalent across various sectors, notably within automotive industries. Nonetheless, there has been a rise in the number of vehicle recalls, particularly associated with software issues. This trend underscores the absence of structured and efficient processes for early defect detection.

Table 1 , contains the research questions (RQs) and their corresponding objectives for developing a software testing process in the automotive domain.

The methodology employed in this work to propose alternatives for addressing the increase in recall processes and late detection of software defects in the automotive domain is depicted in Figure 1 , following the flow of the Design Science Research methodology by [17] .

Design Science Research methodology was applied to identify and demonstrate the problem initially in Section 1. This involved conducting a Systematic Literature Review (SLR) to understand the artifacts used, based on data extracted from the RSL and discussions with specialists. The study progressed to map the stakeholders, as outlined in Table 2 , to comprehend their roles and duties. Furthermore, the desires and objectives of the stakeholders were delineated, as shown in Table 3 . Through mapping process facilitated the definition of project requirements, classification of problems, commencement of artifact development, application of an evaluation artifact, and subsequent classification and explanation of results, contributing to the dissemination of knowledge on the subject.

Finally, the conclusion encompassed the generalization of results to a problem class, followed by communication of the findings, which will be elaborated in subsequent sections.

After mapping the responsibilities, attributions, and desires of stakeholders in the automotive domain, an artifact design for an automotive software testing process was developed. This design encompasses a set of activities with defined inputs and resources, serving as a reference for ensuring a software V & V process specific to the automotive context. To make the Automotive Software Testing Process (ProTSA) available, a process specification is developed. In the specification, the ProTSA modules were detailed: Login, Config Menu_ and test steps, 1 - Integration between functions_FC & FS, 2 - Integration between software, 3 - Integration between CAN signals, 4 - Parameter integrity, 5 - Maturity of ECUs_SW_functions and 6 - Fail Safe Test, in order to make ProTSA available in the online format of ProTSA for easy access and expert evaluations. Afterward, a website was implemented to enable testing and documentation in a more agile and easily accessible way, which is available at https://protsa.infocept.com.br/ .

The evaluation of ProTSA involved the creation and execution of a software test case for integrating functionalities between two ECUs, specifically the ABS and Chassis ECUs. The test case was distributed to specialists in electronic format for execution. Subsequently, the specialists were asked to provide feedback by completing a survey provided by the authors of this work. To assist in the evaluation process, an online tutorial was created and made available on the YouTube platform, covering each of the ProTSA modules. You can find the tutorial at the following weblink: https://www.youtube.com/playlist?list=PLH43-llgcycsv3bKLeygtAao3nr9h9-4X .

Subsequently, e-mails were sent to eighteen professionals linked to software V & V processes in the automotive domain and who have technical knowledge and contributions in the automotive area to evaluate the ProTSA.

In this study, a set of questions was administered using the Likert scale, a widely used method for gauging individuals’ satisfaction or opinions on various subjects such as products, processes, or games. The Likert scale, devised by psychologist Renis Likert in 1932, typically consists of scales ranging from 5 to 9 points. The odd number of points ensures a neutral option, with responses ranging from strongly disagree to strongly agree [18] .

The Likert scale used in this study employed a 5-point scale with positive statements, where scale 1 indicates “Strongly disagree,” scale 5 denotes “Strongly agree,” and scale 3 represents a neutral position— “Neither agree nor disagree.” These questions aimed to gather participants’ opinions on aspects such as resource optimization, productivity, quality, effectiveness, clarity, applicability, and inconsistencies within the ProTSA framework. The results measured on the Likert scale will be used to create graphs alongside mean and standard deviation calculations, providing a visual representation of participants’ perceptions regarding the objectives of the research, learning outcomes, and overall completion of the study phases.

The potential sources of threats identified in the results refer to the nature of the domain studied. Since the area of study was the field of Software in the automotive area, a test case and survey were applied in the domain of Brakes and Control of commercial vehicle chassis. Therefore, generalizations will only cover the domains mentioned. In addition, the ProTSA evaluation invitation was sent to eighteen experts from a multinational manufacturer of commercial vehicles and working with software V & V steps, to which a total of fourteen experts answered and four experts did not respond to the invitation.

Since this is research in a specific sector and domain, the results should be interpreted as a representation of that sector and domain mentioned. However, particularities inherent to the company’s process and different contexts may cause variations. In addition, the level of knowledge of each participant is a threat to the validity of the results. Although all participants are active in software V & V processes, some are more familiar with some of the steps proposed in ProTSA than others, and this can influence different perceptions of the process and suggestions for improvement.

Therefore, the results obtained offer a view of the use of ProTSA in the domains of Brakes and Chassis Control in an automotive environment and based on the level of knowledge of the participants who responded to the survey. In short, for future applications, users are expected to interpret the results and consider the limitations and degree of knowledge of the participants in this work.

After conducting and publishing a systematic literature review (SLR), the authors [14] focused on software testing within the automotive domain, results were obtained to understand methodologies, test types, regulatory frameworks, and challenges prevalent in the automotive industry. Subsequently, individual meetings were conducted with ten professionals from various Original Equipment Manufacturers (OEMs) with diverse experience levels in software development and testing within the automotive sector. The primary goal was to identify the key challenges specific to the OEM’s processes. The study revealed that most tests conducted were functional tests, which validate software outputs against predefined inputs, and application tests, which assess the software’s compatibility across different vehicle versions and external environments.

The mapped and detailed difficulties related to the current automotive software testing process in the OEM under study are presented in Table 4 .

In the next step, discussions were held with advisors to determine the prioritized difficulties for creating the ProTSA concept and identify which challenges would be addressed within the ProTSA project. The authors outlined six steps or modules to face some of the identified difficulties. Each module is detailed in Table 5 .

The ProTSA concept aims to address at least six difficulties highlighted by experts in automotive software testing. The first module, “Integration between functions,” focuses on resolving integration challenges by proactively identifying and prioritizing the requirements necessary for functions to interact correctly during testing and development.

The second module, “Integration between software,” is designed to map interaction needs using block diagrams, enhance visualization of test and development scenarios, track function releases across different software versions, and facilitate test alignment through regular meetings, promoting transparency during testing processes.

The third module, “Integration between CAN signals,” targets challenges related to the availability of CAN signals during functional tests, often discovered late in the project’s lifecycle, typically in prototype vehicles. This module aims to cross-reference information to determine the availability of each CAN signal, helping to address issues encountered during testing.

In the fourth module, “Parameter integrity,” the goal is to guarantee that vehicles are manufactured with parameters that do not lead to failures during production or in operational use. This module involves setting test priorities, creating spreadsheets for user-loaded parameters, suggesting new parameters for testing or release, and establishing workshop schedules for parameterization and documentation tests.

In the fifth module, “Maturity of ECUs,” the objective is to evaluate the maturity of software within Electronic Control Units (ECUs). This module fosters information exchange on test results among the automaker, suppliers, and testers. It also provides percentage-based information and bar graphs to assess the current progress and maturity level of ECUs.

Finally, module 6—Fail Safe Test aims to help identify failures in advance, map the solutions found and applied, and serve as a reference for prioritization and allocation of resources through graphs of failures by ECUs or vehicles.

To enable the online testing process to be more agile to understand and evaluate by experts, ProTSA was implemented through a website, and made available at https://protsa.infocept.com.br/ . In addition, through the administrator environment that will be presented in the following sections, the possibility of configuring the online environment for the needs of different OEMs will be demonstrated.

On the ProTSA online platform, an environment was created for user registration, as shown in Figure 2 .

Subsequently, a step was developed for the registration of a project in the ProTSA environment (as shown in Figure 3 ). For this purpose, the main information in the test project for each ECU was added. Additionally, in this step, you can also select the ECU ( Figure 4 ), function ( Figure 5 ), network type data ( Figure 6 ) and other data, such as CAN signals and parameters.

In ProTSA, there are two environments. The first, named the administrator environment ( Figure 7 ), allows access to update the list of ECUs, CAN networks, parameters, and add new vehicle portfolios. This access also enables configuring weights for questions used to classify functions in the software test project, making ProTSA more adaptable to different projects.

In the second environment of ProTSA, users responsible for conducting or coordinating tests can access six different modules after registering on the website and creating a project. Additionally, there is a diagram of use cases and connection of actors provided for standard user access, as depicted in Figure 8 .

After the creation of the user and registration of a project, the tester will be given the possibility to navigate through the six modules in a sequenced way, that is, starting with module 1-Integration between functions and ending the tests in module 6-Fail Safe.

The main objective of the integration between functions module—module 1 is to enable the structuring of functional tests for integration between the functions of at least two ECUs to avoid problems such as lack of documentary information, unavailability of CAN signals, lack of integration test cases and failure in communication between the stakeholders involved in the tests.

For the purpose, this module was subdivided into four main stages: documentation analysis, CAN signals/messages, classification of client function and service function, test diagram and meetings.

After creating, selecting the project, and accessing the function integration module, the user is asked to select the required data from the first ECU involved in the test, then the selection of the ECU function ( Figure 9 ) and at the end will be asked to upload the function documentation ( Figure 10 ). It is recommended that the document uploaded to the site is a specification of the function or at least a draft of it. This step will be repeated for the second ECU to collect documents from both ECUs to perform a static test, which will be evaluated in a working group, through meetings to evaluate the proper integration of both ECUs and functions. Therefore, this step will be responsible for discovering, in a more agile way, problems of interaction between ECUs and promoting the integration at the beginning of the project of the stakeholders necessary for the success of this test.

In the next step, information from the specific ECU and function will be requested over questionnaire ( Figure 11 ), to later cross-reference the information from each of the ECUs and generate information on compatibility and availability of the information.

At this step of the questionnaire for the functions, the user will be asked to answer three questions for each function inserted in the project and, for each question, there is a different weight that can be decided at the beginning of the project by the project participants.

After the questionnaire is answered, the website will make the calculations based on the answers and will rank points, in which the function that obtains the most points will be called the customer function. The term customer function is used for a function in which the end customer can observe or perceive its operation, such as the activation of the hazard warning light during emergency braking, also known as the Emergency Stop Signal (ESS) function, ( Figure 12 ). However, the classification for the function with the fewest points will be given, as a service function, which means that this is a function that provides data for another function to perform an end activity that is perceptible to the end user. In this case, the service function would be a panic braking detection function to later activate the hazard lights via the ESS, an example of a service function is panic braking detection by means of the rapid application of braking to the brake pedal on the vehicle ( Figure 13 ).

After filling in the project information and answering the questionnaire about the functions, the classification of client and service functions will be made available on the ProTSA online platform (website) ( Figure 14 ). This classification occurs after the website ranks the users’ responses to identify functions that lead the development or integration tests, which, in this case, is called the client function, that is, a function that is visible to the end customer, which, in the case of Figure 14 , represents the Brake cut-off function on the axle.

The output of the first result in ProTSA includes presenting common CAN messages between functions, which is relevant for developers and testers to understand the connection between functions and ensure correct integration. This information is depicted in Figure 15 , showing common messages between Axle Brake cut-off and axle lift control functions.

ProTSA was designed with communication as a crucial factor for the success of function integrations, testing, and development, as highlighted by [19] . Effective communication among stakeholders helps avoid project rework. Users are recommended to schedule meetings through the ProTSA website, where meeting participants can add notes for follow-ups and schedule additional meetings if necessary.

In the last step of the function integration module and based on the analysis of literature, the ProTSA integrates visual communication methods, such as block diagrams, for analysing software requirements and planning software tests, as recommended by [19] [20] .

In the software integration module (module 2) of ProTSA, the main objective is to plan software integration tests between at least two ECUs. This involves creating a plan focused on interaction block diagrams between the software, resources needed for the tests, release plans for each software function, and a communication plan.

The first step in software integration is to create a block diagram to visualize the interface between software components in a simplistic way.

The second step of this module, involves creating a plan for the release of ECU software by breaking it down into functions. The proposal in ProTSA subdivides dates into four periods of the year, during which functions are made available for testing. In ProTSA, the four phases of release are depicted in Figure 16 , with examples like R2301, R2302, R2303, and R2304. The letter R means release and number 23, is related to the test year, which, in the figure, is 2023, and the sequence between 01 and 04 constitutes the order of the plan for releasing the software’s functions. In the online environment, as shown in Figure 16 , mentioned above, it will also be possible to register the date of each release and the description of the function.

The next phase of this module is a step to check the availability of resources needed to run tests, such as software licenses for measuring CAN data, parameterization of ECUs, and availability of proving grounds. This step aims to prevent wastage of time and budget by ensuring that all necessary resources are available before test execution.

After users register the information from the ECUs and store the block diagrams in ProTSA online, it is suggested that the user use the block diagram previously uploaded to ProTSA online to align stakeholder needs. Additionally, as a result, it is recommended to use the software release plan to carefully plan the sequence of software integration tests. Finally, as in the previous module, the importance of communication for the positive outcome of software V & V was addressed. Following the strategy of the importance of alignments, in this module, ProTSA also recommends scheduling meetings to discuss the information collected and establish plans for software integration tests.

For the CAN signal integration module—module 3, the primary goal is to proactively identify the absence of CAN signals during software testing in a prototype vehicle. This module is structured into steps for gathering availability information and conducting comparisons of signal availability.

At this step, ProTSA suggests verifying the requirement for CAN signals for each function beforehand through documentary verification, aligning with static testing principles as described by [8] and [21] . This test will be conducted based on the analysis of CAN signal information, which does not necessitate dynamically operational software. Consequently, this test serves to proactively identify potential software failures and mitigate development costs in a software project.

Figure 17 and Figure 18 illustrate the collection of information concerning the availability of the CAN signal for ECU, ABS, and Chassis, respectively. This information is then verified through a static test.

In the first result of ProTSA’s CAN signal integration module, users can check which CAN signals of the evaluated ECUs are not compatible for correct integration. This is done by requesting the website to check the presence of CAN signals in the two ECUs and their compatibility based on their status availability. Compatibility results are indicated with a “=” sign, while non-compatibility results are indicated with a ≠ sign, requiring discussions to achieve compatibility for correct integration. Figure 19 illustrates an example of the output from ProTSA’s CAN signal integration module, demonstrating compatibility and non-compatibility results for CAN signals between evaluated ECUs.

The next module 4—parameter integrity is structured with steps aimed at ensuring the manufacture of vehicle variants with parameter sets for each ECU and, in some cases, with a single software, but with different parameter sets, making it possible for a single type of car to be sold in distinct categories. In this way, a polo vehicle can have distinct categories and target audiences, the polo track version of the Volkswagen vehicle is equipped with an ABS and ESP safety system; while the same vehicle in the Highline category is equipped with the same ABS, ESP, and automatic braking systems [22] . In several automakers, the same type of vehicle is sold in different classes by offering different functionalities, made available by a set of parameters of a software. In this line of reasoning, a single type of vehicle can be sold in various categories and values by configuring a set of parameters in the same software. Therefore, this module addresses the importance of ensuring that the various sets of parameters are tested to avoid failures in the manufacture of the product and in the operation by the customer.

In this module, the first step that was conceptualized was the definition of priority of prototype vehicles for parameterization tests. At this step, prototype vehicle variants are selected for testing, since assessing all variants of an automaker’s vehicles becomes unfeasible and, according to [8] , a set of software tests tends to infinity. Figure 20 lists a group of questions defined with software testing experts in the automotive industry and are being used in ProTSA to rank the top vehicles for functional software testing. At this stage, the portfolio of vehicles used to exemplify was from the truck manufacturer Mercedes-Benz do Brasil, available on its website [23] .

After answering the questions, the online environment lists the vehicles from the highest score to the lowest according to users for each answer. Each question has different weights defined in meetings with test specialists from a vehicle assembly company and can also be changed in the administrator environment. For this development, question 1 is equivalent to 4 points, question 3 is equivalent to 3 points, question 3 is equivalent to 5 points because it is a question of approvals, that is, legislation requirements, while question 4 is equivalent to 1 point and question 5 is equivalent to 2 points. If a single vehicle is selected for all questions, its maximum score will be 15 points. The example in Figure 21 shows the classification for the portfolio in the ProTSA online environment.

In the next step, for parameterization tests, ProTSA recommends using parameterization data that worked properly in previous projects to start defining the proper parameterization set for the current project.

With this information, ProTSA recommends that users hold a workshop to align which of the contents should be released for series-produced vehicles. After the workshop, it is expected that the results of the tests will be sufficient to perform the parameterization of the software for serial production, however, if failures in the parameterization are detected, ProTSA recommends that the sequence of steps described for module 4—parameter integrity be repeated.

The ECU maturity module in ProTSA is designed to verify information among stakeholders engaged in software testing activities within the automotive domain. ProTSA introduces a structured information exchange process where the requester can approve or reject the responses received. Approved responses signify an advancement in ECU maturity, which is visually represented through a progress percentage bar.

Therefore, this module is organized into six sequential steps for information exchange during testing: Software Information, Unit or Concept Tests, HILL Tests at the supplier or assembler, Application Tests, Durability Tests, and Homologation Tests. Figure 22 illustrates the structured organization of the six steps within the online environment as part of ProTSA.

After accessing the ProTSA step for ECU maturity, users will select their project and initiate the maturity process by requesting software information from the supplier.

Following this, the requester will have the choice to either approve or reject the information provided by the supplier. In the event of a rejection, the user must request clarification from the supplier regarding the reasons for rejection and any specific requirements for subsequent approval.

Following the rejection of information, the supplier is required to provide updated data for re-evaluation by the applicant. However, upon approval of the information by the applicant, it signifies that the ECUs possess the essential information to commence the testing phases. Therefore, approval at this stage, as determined by expert input, corresponds to 10% maturity out of the total 100%.

Figure 23 displays the percentage bar, incrementing by 10% following the requester’s approval.

The subsequent steps follow a similar logic of requesting information for each phase of the tests within the ProTSA module. Approval of information in any of these phases results in an increase in the percentage bar, indicating an increase in the ECU’s maturity. The objective of this stage in ProTSA is to reach 100% maturity, as illustrated in Figure 24 . This signifies that all activities necessary to finalize the V & V process of the ECU/software/system have been accomplished, and stakeholders concur that it is sufficiently mature for production release. Consequently, this phase of ProTSA will also conclude.

The last module—Fail Safe Test was incorporated into ProTSA at the suggestion of the experts interviewed, because during years of experience with V & V, fault monitoring is used on a regular basis, discussing each one of them, to achieve success during the development phase and V & V for any ECU/software. These experts stressed the significance of fault monitoring and discussion of each one of them throughout the development and V & V phases to ensure success for any ECU/software.

The Fail Safe Test module within ProTSA was developed with the objective of proactively identifying failures during the development phase, thereby preventing these failures from propagating to serial software or vehicles for end customers. It is structured to map faults by ECU or vehicle configuration, generate fault code reports, display solution progress graphs, and organize regular workshops to prioritize resources based on the number of failures per ECU or vehicle system.

The initial step of the Fail Safe Test module within ProTSA requires users to access the Fail Safe Test menu, choose the ECU for assessment ( Figure 25 ), and upload the list of fault codes gathered from prototype vehicles ( Figure 26 ).

Subsequently, the fault codes of the previously selected ABS ECU will be displayed on the pre-selected vehicles, along with the option to download this information for future alignments and stakeholder meetings ( Figure 27 ).

The next step in the ProTSA Fail Safe Test module enables users to generate graphs illustrating the failure status of ECUs, categorizing faults as not relevant, under repair, or resolved. The status attributed to each fault must align with the individual responsible for this aspect in ProTSA. To mark a fault as resolved, evidence such as measurements with the proposed solution, specification documents, or user feedback is necessary.

The graph in Figure 28 represents the fault status of the ABS ECU in selected prototype vehicles over a specific period.

In the final stage of the Fail Safe Test module, it is advised to conduct weekly workshops to monitor the progress of fault resolution by ECUs/software and determine resource priorities. Users can schedule these meetings on the ProTSA platform by utilizing the “schedule a meeting” button and sending invitations along with generated reports to participants, allowing them to prepare in advance for the workshop and take necessary actions.

According to Figure 29 , the survey participants’ experience level in software development or testing is distributed as follows: 35.7% have 0 - 3 years of experience, 28.6% have 3 - 5 years of experience, 14.3% have 5 - 10 years of experience, and 21.4% have more than 10 years of experience in the field.

The respondents’ roles are distributed as follows: 64.3% are engineers, 14.3% are technicians, another 14.3% are analysts, and 7.1% are interns, as illustrated in Figure 30 .

Regarding which segments of automotive software testing the participants have already worked, considering that they may have worked in more than one of the segments listed, forty-six votes were obtained. The segment in which the participants most worked was V & V with 24%, in the next place, the application and integration tests stand out with 21.7% each. In the next position, prototype tests with 17.4%, while documentation tests appear with an 8.7% performance and homologation with 6.5% ( Figure 31 ).

The survey also examined the number of participants who have previously utilized a standardized process for software validation. Figure 32 reveals that 2 participants (14.3%) have employed a standardized process before, whereas 12 participants (85.7%) have never utilized a standardized process for software validation.

Related the two participants who responded about their experience with standardized processes mentioned using the Fail-Safe Test, which is also integrated into ProTSA. The second process involved a test script for new functions, in alignment with the parent company, focusing on software validation for Diesel engine ECUs.

The participants unanimously expressed a positive predisposition towards using a standardized V & V (Verification and Validation) process in their daily work for tackling testing stages. This inclination was reflected in all fourteen respondents, as illustrated in Figure 33 .

In the next question, the participants were asked about critical activities or actions that could influence the outcome of a V & V process based on their experience. This inquiry sought to identify key aspects raised by participants that ProTSA could subsequently assist in streamlining their daily work.

Among the alternatives listed, the one that obtained the most votes was the integration activity between ECUs at distinct stages of development, with 10 votes, which represents 23.8% of the total of 42 presented. The second most favored option, function integration activity, was selected by 21.4% of respondents. In the third position, there was a tie between three options: parameters integrity activity, lack of information for test preparation, and prioritization of tests and software development, each receiving 11.9% of the votes.

In the fourth position, the integration activity between CAN messages received 9.6% of the votes, followed by software integration with 7.1%. Additionally, one participant reported a lack of information from test results as impacting the V & V process, representing 2.4% of the responses, illustrating in Figure 34 .

A test case was presented to the experts in ProTSA, involving a V & V process between the ABS and Chassis ECUs. This case aimed to integrate two functions: the axle brake cut-off function in the ABS ECU and the axle lifting control in the Chassis ECU. The validation and integration were planned for a 6 × 2 commercial vehicle model currently in production. The participants were given a tutorial outlining the steps to execute this process.

When evaluating ProTSA through a survey, questions were rated using the Likert scale with a 5-point range. A rating of 1 indicated “Strongly disagree” with the statement, while a rating of 5 indicated “Strongly agree.” To quantitatively analyse the collected data, mean and standard deviation calculations were conducted.

The Likert scale question aimed to gauge experts’ opinions on whether ProTSA could enhance resource optimization and boost productivity during V & V phases. The survey yielded a mean score of 4.64 with a standard deviation of 0.47 ( Figure 35 ). The average score above 4, coupled with responses mostly falling between 4 and 5, indicates a positive impact attributed to ProTSA in terms of resource optimization and productivity gains. This aligns with the intended objectives of ProTSA, which emphasizes structured testing processes to enhance productivity, efficiency, resource optimization, and overall quality, as noted by Vieira [24] .

The second question sought participants’ opinions on whether introducing ProTSA with defined steps would positively impact the quality requirements in V & V processes for automotive software. The survey results, with a mean of 4.78 and a standard deviation of 0.41, indicate a generally positive outlook. Most responses falling between 4 and 5 suggest that most evaluators believe ProTSA’s introduction would enhance the quality of V & V processes ( Figure 36 ). This aligns with the notion in Rocha [25] that software quality is enhanced through proactive measures like early detection and prevention of faults, which ProTSA’s Fail-Safe Test module is designed to facilitate.

In the third question, the objective was to receive feedback from the experts regarding the increase in efficacy if ProTSA is introduced in V & V processes. The mean of 4.42 and standard deviation of 0.49 suggest, in a positive way, gains in terms of efficacy. However, in Figure 37 , it is possible to detect more responses at value 4, which suggests the need for some points of improvement in ProTSA to help with the efficacy requirement. The importance of efficacy is relevant and was considered in ProTSA, because, according to Galdino [26] , there is a growing demand for quality certificates by customers and how a V & V program can tend to infinite test cases. Therefore, the selected test cases need to be effective to find more defects in less time and avoid high costs for software error corrections. Thus, this requirement was included for the evaluation of the experts.

The subsequent question sought feedback from experts concerning the clarity and ease of using ProTSA. Overall, there was positive feedback with a mean score of 4.28 and a standard deviation of 0.58. However, the slight variance indicated by the standard deviation and some respondents choosing value 1 suggest differing opinions among experts regarding this aspect, as depicted in Figure 38 . To enhance this requirement, some participants proposed conducting training sessions on ProTSA to enhance clarity and adaptability with the process.

The subsequent evaluation aims to assess the relevance of the steps incorporated in ProTSA. The mean score of 4.78 and standard deviation of 0.41 suggest a high level of relevance of the steps outlined in ProTSA for a V & V process in the automotive environment. The predominance of responses at value 5 further supports the significance of these steps, as depicted in Figure 39 .

In the next question, the study sought to verify the ease of applicability of the proposed process in the automotive environment. With a mean of 4.21 and a standard deviation of 0.67, the answers with a mean higher than 4 positively suggest the ease of applicability of the process. However, the relatively high upper standard deviation indicates that there are some participants who have different opinions regarding the mean and the easy applicability factor. According to Marques [27] , there are some success factors in the implementation of a project such as adequate budget, support from top management, provision for training, project sponsor, etc. These points raised by the study are factors that some experts may have seen as difficult factors for applicability quickly and easily. Therefore, there were 2 evaluations on scale 3, as shown in Figure 40 . Suggestions for improvements pointed out by the participants will be presented in the subsequent sections.

The penultimate question asked the experts to evaluate the presence of inconsistencies in the proposed process. A score of 1 indicated few inconsistencies, while 5 indicated many inconsistencies. The average score of 1.42, close to 1, suggests that participants generally perceived few inconsistencies in the process. However, the standard deviation of 0.62 and some responses at value 1 indicate slight variation in the experts’ opinions, as illustrated in Figure 41 .

In the final question using a Likert scale, participants were asked to rate the completeness of the ProTSA steps. The responses yielded an average score of 4.28 and a standard deviation of 0.69. This indicates that most opinions fell within the range of 4 or higher, reflecting a positive trend regarding this evaluation criterion. However, two responses rated the completeness at 3, as depicted in Figure 42 , signaling room for improvement and suggesting areas that participants highlighted for enhancement, which will be detailed in subsequent sections.

The final question of the survey asked participants to suggest areas for improvement in ProTSA. In this aspect, and in a more qualitative way, the following points were listed:

1) Adds an exclusive step for SiL

One of the participants suggested including the SiL stage in ProTSA in an exclusive module, as it is a more agile type of test and is already used as part of the approval process for security systems for the specific OEM, such as the Electronic Stability Program (ESP).

2) Training for the use of ProTSA

A suggestion was made for a hands-on training format with an instructor in person to clarify doubts that can occur during the use of the process.

3) Incorporation of ProTSA into the company’s development and testing strategy

One response received was the support from middle and senior management for the effective integration of the ProTSA process into R & D teams was suggested as an important factor for successful implementation.

As it is a new tool, there is an initial phase of some adaptation, rejection, or challenges to operate without errors and management support can be a large part of the success of the implementation.

4) Automation of the online interface

Some professionals have suggested that the steps in the current version of ProTSA could be more automated or integrate them into the company’s existing processes.

5) Maturity of the prototype vehicle

Some professionals mentioned the need to incorporate a stage and verification of the maturity of the vehicle, as many functional software tests have a direct impact on the maturity of the vehicle.

6) Approach not only to testing, but to hardware and software obsolescence risks.

One survey participant reported the need to broaden the scope of ProTSA and add hardware and software obsolescence risk mapping. One proposal would be to have an exclusive module with roadmaps for each ECU and results of discussions of obsolescence topics to be available to the entire R & D team on a single platform.

7) Application in a larger number of real cases

One user reported the need to apply the process to a larger number of test cases to identify possible additional points for improvement.

8) Positive feedback

Some users have reported that the process is specified, with clear steps, steps well sequenced from each other and robust. In addition, it was also mentioned that, once the process is explained, it is easy to understand the application of the sequence of use for ProTSA.