Introduction to the special issue on test oracles

Software testing is the de facto technique for correctness verification. Although there are different motivations and approaches to testing, fundamentally, it involves the running of test cases against the software under test. An important aspect is the ability to tell whether the execution results agree with the expected outcomes. The mechanism to determine the correctness of the results is called a test oracle. Unfortunately, such a mechanism does not always exist or may exist but cannot be feasibly used, which is known as the oracle problem. This is a major challenge to software testers, severely curtailing the application of testing approaches and methods to practical situations. In spite of the vast amount of interest in software testing, studies on test oracles drew much less attention than other aspects of software testing until quite recently. The increasing number of research activities surrounding the oracle problem reflects a growing interest in the area. This special issue documents some of the new contributions in software test oracles. We received a total of 19 submissions. Each submission has been reviewed by at least three reviewers. Six papers have been accepted for publication. In general, test oracle issues may involve identifying the expected program properties, test case generation, test sequencing and concurrency, test output verification, and fault localization. To deal with the identification of expected program properties, paper 1 titled “Automated Inference of Likely Metamorphic Relations for Model Transformations” by Troya et al. proposes a method of automatically inferring likely metamorphic relations, in the context of metamorphic testing, for the verification of model transformations. To handle test case generation and conduct run-time assertion checking, paper 2 titled “Contract-Based Testing for PHP with Praspel” by Dadeau et al. makes use of contracts, which enable the automation of these processes. To tackle the test sequencing question, paper 3 titled “Temporal Algebraic Query of Test Sequences” by Prasetya studies the question of test sequences using formalism based on algebraic relations and linear temporal logic. Paper 4 titled “Stateless Techniques for Generating Global and Local Test Oracles for Message-Passing Concurrent Programs” by Carver and Lei considers the state explosion problem of test sequences and proposes a stateless method to generate test oracles for concurrent programs. To address the test output verification problem, paper 5 titled “VISOR: A Fast Image Processing Pipeline with Scaling and Translation Invariance for Test Oracle Automation of Visual Output Systems” by Kıraç et al. focuses on the verification of visual output and introduces an automated test oracle that avoids false positives in image comparison caused by scaling and translation. To investigate software fault localization in the face of the oracle problem, paper 6 titled “Exploring the Usefulness of Unlabelled Test Cases in Software Fault Localization” by Zhang et al. develops an approach based on a core idea of test classification. We would like to thank the authors for their contributions, and the reviewers for their invaluable comments.