Evolutionary approach for combinatorial testing of software product lines
Software Product Line (SPL) focuses on common features reusability, formulated for different software products. Complete testing on the entire SPL is known to be unfeasible. This is due to the very large number of possible products to be produced; configured using all possible sets of features i...
Saved in:
| Main Author: | |
|---|---|
| Format: | Thesis |
| Language: | English |
| Published: |
2020
|
| Subjects: | |
| Online Access: | http://psasir.upm.edu.my/id/eprint/91051/1/FSKTM%202020%2014%20IR.pdf |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| id |
my-upm-ir.91051 |
|---|---|
| record_format |
uketd_dc |
| institution |
Universiti Putra Malaysia |
| collection |
PSAS Institutional Repository |
| language |
English |
| advisor |
Md. Sultan, Abu Bakar |
| topic |
Computer software - Development Software engineering Software product line engineering |
| spellingShingle |
Computer software - Development Software engineering Software product line engineering Sahid, Mohd Zanes Evolutionary approach for combinatorial testing of software product lines |
| description |
Software Product Line (SPL) focuses on common features reusability,
formulated for different software products. Complete testing on the entire
SPL is known to be unfeasible. This is due to the very large number
of possible products to be produced; configured using all possible sets
of features in the SPL.
Combinatorial Testing (CT) has been widely used in software testing as
it is able to generate test input for a single software product that deviates
from exhaustive testing, nevertheless proven to be effective. In SPL
testing, to generate minimal test configuration that maximizes t-wise
coverage is not trivial, especially when dealing with a huge number of
features and when constraints have to be satisfied, which is the case in
most SPL systems. Two salient obstacles in SPL testing are identified;
(1) time required to cover feature configuration testing for large scale
Feature Models and higher strength of t-wise testing, and (2) insufficient
coverage of feature interaction towards higher fault detection due
to uniform strength of feature combination. In the case of unfeasible
higher strength CT-based testing for SPL, this thesis proposed a relaxed
version of Covering Array (CA), i.e. Variable-strength CA (VCA). In
highly configurable systems, the entire configuration spaces contain
several groups of sub-configurations that have different level of risks
that can cause failures. By focusing on these group of subconfigurations,
the space of testing artefacts can be reduced, while still
maintain the results of testing. Practical Combinatorial Testing is challenging, such that it is hard to
decide what level of interaction strength to be applied to which different
groups of features. There are two inherent challenges; (1) to decide
what value of interaction strength one should apply, and (2) to choose
which group or set of features the interaction strength should be applied
to. For these reasons, part of this thesis presents and evaluates a
new technique aimed at tackling all the mentioned difficulties in the
combinatorial testing of SPL system. The approaches are (1) adopting
an Estimation of Distribution Algorithm approach to aid the construction
of a covering array driven by second order feature dependence,
and (2) construct a feature configuration dependence graph to assist in
building the variable-strength covering array. These techniques are
implemented as COTED (Combinatorial Testing using Bivariate EDA).
Experiments have been done to gauge the performance of COTED from
three different facets; (1) the ability to reduce test configuration redundancy,
(2) the ability to achieve better rate of interaction coverage and
(3) the ability to detect high number of faults. Four empirical studies
conducted to compare COTED against other approaches (namely ICPL,
LOOKUP and CASA). The statistical analysis of the results obtained
from the experiments are promising. For the first facet, COTED is comparable
with LOOKUP, but a slightly better redundancy is achieved by
ICPL with significant level of 0.01. For the second facet, COTED able to
outperform ICPL and LOOKUP with significance level of 0.1. For the
third facet, the two sets of VCA test configurations (i.e. (i) 2-wise and
partial 3-wise, and (ii) 2-wise, partial 3 and partial 4-wise) generated by
COTED are able to achieve competitive fault detection rate (with significance
level of 0.05) using less number of test configurations. The result
shows that COTED increase the interaction coverage rate, and
managed to produce less number of test configurations to detect an
almost same number of faults as the other strategies, hence, suggests
that it is able to improve Combinatorial Interaction Testing (CIT) for
SPL. |
| format |
Thesis |
| qualification_level |
Doctorate |
| author |
Sahid, Mohd Zanes |
| author_facet |
Sahid, Mohd Zanes |
| author_sort |
Sahid, Mohd Zanes |
| title |
Evolutionary approach for combinatorial testing of software product lines |
| title_short |
Evolutionary approach for combinatorial testing of software product lines |
| title_full |
Evolutionary approach for combinatorial testing of software product lines |
| title_fullStr |
Evolutionary approach for combinatorial testing of software product lines |
| title_full_unstemmed |
Evolutionary approach for combinatorial testing of software product lines |
| title_sort |
evolutionary approach for combinatorial testing of software product lines |
| granting_institution |
Universiti Putra Malaysia |
| publishDate |
2020 |
| url |
http://psasir.upm.edu.my/id/eprint/91051/1/FSKTM%202020%2014%20IR.pdf |
| _version_ |
1747813670680788992 |
| spelling |
my-upm-ir.910512021-10-25T04:27:08Z Evolutionary approach for combinatorial testing of software product lines 2020-07 Sahid, Mohd Zanes Software Product Line (SPL) focuses on common features reusability, formulated for different software products. Complete testing on the entire SPL is known to be unfeasible. This is due to the very large number of possible products to be produced; configured using all possible sets of features in the SPL. Combinatorial Testing (CT) has been widely used in software testing as it is able to generate test input for a single software product that deviates from exhaustive testing, nevertheless proven to be effective. In SPL testing, to generate minimal test configuration that maximizes t-wise coverage is not trivial, especially when dealing with a huge number of features and when constraints have to be satisfied, which is the case in most SPL systems. Two salient obstacles in SPL testing are identified; (1) time required to cover feature configuration testing for large scale Feature Models and higher strength of t-wise testing, and (2) insufficient coverage of feature interaction towards higher fault detection due to uniform strength of feature combination. In the case of unfeasible higher strength CT-based testing for SPL, this thesis proposed a relaxed version of Covering Array (CA), i.e. Variable-strength CA (VCA). In highly configurable systems, the entire configuration spaces contain several groups of sub-configurations that have different level of risks that can cause failures. By focusing on these group of subconfigurations, the space of testing artefacts can be reduced, while still maintain the results of testing. Practical Combinatorial Testing is challenging, such that it is hard to decide what level of interaction strength to be applied to which different groups of features. There are two inherent challenges; (1) to decide what value of interaction strength one should apply, and (2) to choose which group or set of features the interaction strength should be applied to. For these reasons, part of this thesis presents and evaluates a new technique aimed at tackling all the mentioned difficulties in the combinatorial testing of SPL system. The approaches are (1) adopting an Estimation of Distribution Algorithm approach to aid the construction of a covering array driven by second order feature dependence, and (2) construct a feature configuration dependence graph to assist in building the variable-strength covering array. These techniques are implemented as COTED (Combinatorial Testing using Bivariate EDA). Experiments have been done to gauge the performance of COTED from three different facets; (1) the ability to reduce test configuration redundancy, (2) the ability to achieve better rate of interaction coverage and (3) the ability to detect high number of faults. Four empirical studies conducted to compare COTED against other approaches (namely ICPL, LOOKUP and CASA). The statistical analysis of the results obtained from the experiments are promising. For the first facet, COTED is comparable with LOOKUP, but a slightly better redundancy is achieved by ICPL with significant level of 0.01. For the second facet, COTED able to outperform ICPL and LOOKUP with significance level of 0.1. For the third facet, the two sets of VCA test configurations (i.e. (i) 2-wise and partial 3-wise, and (ii) 2-wise, partial 3 and partial 4-wise) generated by COTED are able to achieve competitive fault detection rate (with significance level of 0.05) using less number of test configurations. The result shows that COTED increase the interaction coverage rate, and managed to produce less number of test configurations to detect an almost same number of faults as the other strategies, hence, suggests that it is able to improve Combinatorial Interaction Testing (CIT) for SPL. Computer software - Development Software engineering Software product line engineering 2020-07 Thesis http://psasir.upm.edu.my/id/eprint/91051/ http://psasir.upm.edu.my/id/eprint/91051/1/FSKTM%202020%2014%20IR.pdf text en public doctoral Universiti Putra Malaysia Computer software - Development Software engineering Software product line engineering Md. Sultan, Abu Bakar |