Introduction: Model-Based Testing
and Glossary


Model-based testing (MBT) is seen as the next generation of automated testing which automates not only the execution of tests but also their generation. MBT utilizes formal models that model the SUT from the perspective of testing usually at a high level of abstraction. MBT can be done off-line or online. In off-line testing these models are used for generating conventional test suites that can be later executed on the SUT. In contrast, an online approach constantly executes the tests as they are generated, facilitating the testing of non-deterministic systems and supporting long test runs, for instance.

In both online and off-line mode, MBT can increase test coverage compared with conventional static and linear test scripts. Increased coverage is seen as especially important in testing concurrent systems that are hard to test using conventional methods. Moreover, test suite maintenance, which can be very resource consuming especially in GUI testing, is replaced by model maintenance, which is more straightforward, at least for a competent test modeler. When the model has been updated, the new tests are generated automatically reflecting the changes. Furthermore, some previous studies, as well as our own experiences, suggest that the majority of the defects found using MBT might be found already in the modeling phase. This entails early defect detection, since modeling can be started much before the SUT is mature enough for automatic test execution.

Academic research on MBT has been conducted for about two decades and industrial interest towards the topic is steadily increasing. In the safety critical systems domain, where legislation regulates how systems are being developed, MBT is becoming increasingly important technology, since IEC-61508-3 ed2.0, a basic standard on functional safety released in 2010, “highly recommends” its use in SIL-3 and SIL-4 level software development.

TEMA Toolset

Towards easier deployment of MBT in the domain of smartphone application GUI testing, Tampere University of Technology, Department of Software Systems has developed a prototype toolset, called TEMA, and a methodology for online test generation for smartphone GUI testing. The methodology is based on long-term research on MBT and practical case-studies with industrial partners. The features of TEMA’s two-tier modeling approach include the ability to reuse high-level models as the basis of test generation among different smartphone platforms. Moreover, we are able to test application interactions in long period test runs revealing robustness issues, which are typically out of reach of conventional test automation tools as well as off-line model-based tools that produce shorter test cases.


Glossary

Action machine

A model component that describes the functionality of the SUT at the level of action words.

Action word

A high-level action executable by the SUT, implemented with keywords.

Adaptation

A mechanism where model-based tests are adapted for execution according to the abstraction level of the SUT. The execution results are interpreted and reported back to test generation tools, hereby allowing online testing.

Coverage language

The syntax of forming coverage requirements.

Coverage requirement

A formal test objective that defines the ending criteria of a test run and acts also as a guideline to the guidance algorithm with respect to the actions to be executed in order to fulfill the ending criteria.

Data statement

A piece of Python code associated with a model action in order to manage external data.

Data table

A data structure containing the external data to use in data statements.

Guidance algorithm

An algorithm governing the test execution in online testing.

Initialization machine

A model component that defines necessary initialization procedures for the SUT.

Keyword

A low-level GUI event, used for implementing action words.

Launch machine

A model component that defines the initiation sequence for an action machine at the level of keywords, used for switching between applications.

Localization data

GUI texts in some specific language.

Localization table

A data structure that contains localization data.

LSTS

Labeled State Transition System, the formalism used in TEMA model creation.

Model-Based Testing

A methodology of automated software testing that automates test generation in addition to test execution, based on a formal model that describes the SUT functionality.

Model component

A formal model that describes some aspect of the SUT, constituting the final test model with other model components.

Offline testing

An approach of model-based testing where test cases are generated from a test model for later execution.

Online testing

An approach of model-based testing where tests are generated dynamically during execution.

Parallel composition

The process of combining several model components into the final test model.

Refinement machine

A model component that contains keyword implementations for action words.

State verification

An LSTS attribute used for generating a keyword sequence in order to verify the state of the SUT.

Task switcher

A special model component generated to manage the synchronizations between the model components in the parallel composition.

Test model

A formal model that describes the functionality of the SUT in model-based testing.

Transformation

A process where a test case that has been generated from models is transformed into a form executable by the SUT. Normally usable only in offline testing.