Our previously published paper touches upon the test design using combinatorial testing approach for Contiki operating system [1] . In this paper we intend to extend the concept and explain what can be done for the Internet of Things (IoT) operating systems which do not have standard regression test suites viz. RIOT and Tiny OS. We analyze the Advanced Combinatorial Testing for Software (ACTS) generated test suite design and explain how the traditional effective metrics, code coverage can be gathered in addition to more relevant combinatorial coverage measurements using combinatorial coverage measurement tool (CCM) .

While we want to have minimal overlap of this paper with the one published already [1] , we may re-visit few sections and elaborate more. At places, we get specific and explain implementation part of testing (such as what changes are done to the test setup for gathering the data).

National Institute for Standards and Technology (NIST) has been actively supported combinatorial testing. The research carried out by NIST has been well documented [2] [3] . NIST has made several tools which are available to public [4] . C Nie et al. conducts a survey of combinatorial testing and the same is available as part of survey report [5] . Software testing is inarguably essential part of all software development cycles and the same is discussed at high levels by several papers [6] . While few papers discuss software testing techniques at very high level [7] , we get specific and discuss combinatorial testing in this paper.

Combinatorial testing is not only adopted by researchers, and evidence shows the adoption by large organizations [8] . Few papers have different perspectives about testing at large [9] . Test suite and test oracles are integral part of testing these days [10] . NIST tools for combinatorial testing take these test suites and test oracles as input entities. NIST tools are an integral part of combinatorial testing and many users have been using them successfully in testing activities of research/industry. ACTs and CCM aid the combinatorial testing immensely. ACTs has been tested with self [11] . CCM’s success is documented [12] . While abstracting the formal specifications to generate software tests is an old concept [13] , it is being still researched. Few researchers are focusing on different fields of combinatorial testing [14] , we in this paper focus on classic approaches of combinatorial testing techniques. A general strategy for t-way software testing is documented in few papers [15] and their through put is documented in few papers [16] .

As mentioned earlier, this paper focuses on classic approaches of combinatorial testing and the same is being elaborated in the following sections.

Figure 1 depicts the typical flow of work when we want to ascertain the effectiveness of the test suite using combinatorial approach. First step is choosing the operating system for Internet of Things. Then traverse through the source code of the open source code base folders to see if the test suite exists. If it exists gather the coverage data using the code coverage tools.

If the gathered data indicates inadequate test suite, redesign the test suite using the combinatorial approach and gather the data. If the coverage data is less, it calls for re-visiting the test design. Base line the test suite once the adequate test criterion is met.

As can be seen from the diagram it can be iterative process. We document the process that was used for case study operating system in further sections. We describe the approach to be followed when the test suite already exists and when it does not. Section 3 is for the case when the base test suite already exists and Section 4 is for the case when the test suite does not exist.

Figure 2 depicts the process in the case when test suite already exists. This section is for the case when the base lined test suite already exists as in the case of Contiki operating system version 2.7. We can use the either parameter based re-design or configuration based re-design as explained in the book and manual [1] [2] or combination of both. The coverage can be gathered using CCM and traditional coverage tools such as CodeCover. We did preliminary investigation using freely available tool CodeCover for the existing test suite. The coverage was less than 20%. Appendix B gives the data gathered using the CodeCover.

Then we visited the existing test suite to know the reason for low coverage. Few areas of improvements were observed in the existing test suite.

・ No formal test design document existed.

・ It appears that the test cases were concentrated around few mote types (hardware or configurations in the context of combinatorial testing).

We went through the whole regression test suite to extract the configurations supported and input parameters being used. We came up with Table 1 to be populated in the ACTS test model. When the ACTS was populated using these set of values, the generated test design document is as shown in Appendix A.

Contiki is open source operating system widely used and accepted for Internet of Things. It has base-lined regression test suite for version 2.7. Contiki gives the user friendly operating system in the form of instant Contiki which has Ubuntu like the feel with the tool chains to make the iterative development easy. The developers can use the instant Contiki to test the patches and testers can use the same environment for ascertaining the reliability of the operating system without procuring the hardware for all the mote types.

Contiki gives the simulator which is called Cooja. The Cooja simulator talks to the Contiki using Java Native Interface (JNI). The test cases are called csc files which are understandable by Cooja. We found Eighty three test cases of this type in the regression folder. However, these test cases were concentrated around few mote types.

As already mentioned, Appendix A gives the test design generated using ACTS for Table 1 input. Let us visit the column 2 of the design. We can see that the generated test cases are spread across the mote (hardware) types. Further, the generated test design takes care of the input parameters as well for the test cases.

Now the task at hand is mapping these generated test cases to functional test cases (xml files called csc) which are understandable by Cooja and gathering the coverage data again. The coverage data should improve in principle. We are working on this.

Figure 3 depicts the case when test suite does not exist. This process is more suited for operating systems which do not have standard regression test suite viz. RIOT and TinyOS. Since the functional specification and test design are both missing in case of these operating systems, we will have to come up with the functional specification document first. This will be our understanding of the functionality that these operating systems support. Once the functionality of these operating systems is understood we will have to come up with the test design. Configuration to be supported and input parameters to be supplied for each test case will act as starting point for populating the ACTS test model. Once test design is generated, we will have to understand the test environment for these operating systems and the test design need to be mapped to functional test cases to be executed for gathering the coverage data. The CCM coverage will not be appropriate as the test cases generated using ACTS tool will always give 100% combinatorial coverage. Traditional coverage such as code coverage may be handy.

In this section we document the changes that we did in the Contiki environment for the tasks at hand. Since we get implementation specific for case study operating system, this section can be conveniently skipped by the readers who are not interested in specific details for given operating system.

1. Log in as user in the instant Contiki environment.

2. Search for the .travis. yml

3. Add the build type you are interested in:

- BUILD_TYPE = “ipv6-apps”

- BUILD_TYPE = “CT”

-BUILD_TYPE = “compile-8051-ports”

4. Under the directory../contiki-2.7/regression-tests create a folder 02-CT

5. Under contiki-2.7/regression-tests/02-CT directory create *.csc files you are interested in viz. 01-custom.csc 02-custom.csc

6. The Make file should look like include../Makefile. simulation-test

7. Create a 01-custom.csc file in the Cooja tool. Use the test script editor to create a java script which will be essential while running the test case from command line using the makefile.

8. Modify the build.xml suitably as explained in Appendix C.

9. Run the regression test suite as usual.

10. Test run will create many *.clf files.

11. Create a script for analyze, merge and generate report.

In this paper we presented the approaches that could be employed for designing the regression test suite using combinatorial approach. We explained how the bench marking of the regression test suite could be done using the traditional approaches such as code coverage in addition to coverage gathered using combinatorial coverage measurement tools.

In this paper we investigated how the combinatorial approach could be applied for the cases when the regression test suite already existed viz. Contiki case. We were working on ascertaining the gain due to Combinatorial testing.

We were planning to explore the Combinatorial testing for the cases when the regression test suite did not exist viz. RIOT and TinyOS.

Further we were planning to model the System Under Test (SUT) and use New Symbolic Model Verifier (NuSMV) in conjunction with ACTs.

AbhinandanH. Patil,NeenaGoveas,KrishnanRangarajan, (2015) Test Suite Design Methodology Using Combinatorial Approach for Internet of Things Operating Systems. Journal of Software Engineering and Applications,08,303-312. doi: 10.4236/jsea.2015.87031

<!--?xml version=“1.0”?-->

<project name="“COOJA simulator” default="run" basedir=".">

<property name="java" location="java/">

.

.

<property name=“args” value=”/>

<property name=“codecoverDir” value=“/home/user/Desktop/CodeCover/codecover-batch-1.0/lib”/ >

<property name=“sourceDir” value=“/home/user/contiki-2.7/tools/cooja/java”/>

<property name=“instrumentedSourceDir” value=“instrumented”/>

<property name=“mainClassName” value=“se.sics.cooja.GUI”/>

<taskdef name=“codecover” classname=“org.codecover.ant.CodecoverTask” classpath=“${codecoverDir}/codecover-ant.jar”/ >

<target name=“clean”>

<delete>

<fileset dir=“.” includes=“*.clf”/>

</delete>

<delete file=“codecover.xml”/>

<delete file=“report.html”/>

<delete dir=“report.html-files”/>

</target>

<instrument containerId=“c” language=“java” destination=“${instrumentedSourceDir}” charset=“utf-8” copyUninstrumented=“yes” >

<source dir=“${sourceDir}”>

<include name=“**/*.java”/>

</source>

</instrument>

<save containerId=“c” filename=“codecover.xml”/>

</codecover>

</target>

<javac srcdir=“${instrumentedSourceDir}” destdir=“${instrumentedSourceDir}” encoding=“utf-8” target=“1.7” debug=“true” classpath=“${codecoverDir}/lib/codecover-instrumentationjava. jar:/home/user/contiki-2.7/tools/cooja/lib/log4j.jar:/home/user/contiki- 2.7/tools/cooja/lib/jdom.jar:/home/user/contiki-2.7/tools/cooja/lib/jsyntaxpane.jar” includeAntRuntime=“false” >

</target>

<java classpath=“${instrumentedSourceDir}:${codecoverDir}/lib/codecoverinstrumentation- java.jar:/home/user/contiki- 2.7/tools/cooja/lib/log4j.jar:/home/user/contiki- 2.7/tools/cooja/lib/jdom.jar:/home/user/contiki-2.7/tools/cooja/lib/jsyntaxpane.jar” fork=“true” failonerror=“true” classname=“${mainClassName}” >

<jvmarg value=“-Dorg.codecover.coverage-log-file=test.clf”/>

</java>

</target>

<target name=“create-report”>

<codecover>

<load containerId=“c” filename=“codecover.xml”/>

<analyze containerId=“c” coverageLog=“*.clf” name=“Test Session”/>

<save containerId=“c” filename=“codecover.xml”/>

<report containerId=“c” destination=“report.html” template=“/home/user/Desktop/CodeCover/codecover-batch-1.0/reporttemplates/ HTML_Report_hierarchic.xml” >

<testCases>

<testSession pattern=“.*”>

<testCase pattern=“.*”/>

</testSession>

</testCases>

</report>

</codecover>

</target>

<target name=“help”>

<echo>

.

.

<target name=“copy configs” depends=“init”>

<mkdir dir=“${build}”/>

<copy todir=“/home/user/contiki-2.7/tools/cooja/instrumented”>

<fileset dir=“${config}”/>

</copy>

.

.

.

<target name=“jar_cooja” depends=“init, compile, copy configs, compile instrumented “ >

<mkdir dir=“${dist}”/>

<jar destfile=“${dist}/cooja.jar” base dir=“/home/user/contiki- 2.7/tools/cooja/instrumented” >

<manifest>

<attribute name=“Main-Class” value=“se.sics.cooja.GUI”/>

<attribute name=“Class-Path” value=“. lib/log4j.jar lib/jdom.jar lib/jsyntaxpane.jar”/ >

</manifest>

</jar>

<mkdir dir=“${dist}/lib”/>

<copy todir=“${dist}/lib”>

<fileset dir=“${lib}”/>

</copy>

</target>

</project>