Using metamorphic testing to evaluate DNN coverage criteria

Generating test cases and further evaluating their "quality" are two critical topics in the area of Deep Neural Networks (DNNs). In this domain, different studies have reported that metamorphic testing (MT) serves as an effective test case generation method, where an initial set of source test cases is augmented with identified metamorphic relations (MRs) to produce the corresponding set of follow-up test cases. As a result, the fault detection effectiveness (and, hence, the "quality") of the resulting test suite T, containing these source and follow-up test cases, will most likely be increased. Recently, we observed that some coverage criteria have been proposed to measure the quality of the test suites in the DNN domain. This observation leads to the following interesting and worth exploring research question (RQ): Do these DNN coverage criteria properly reflect the quality improvement after a test suite has been augmented with MRs? We conducted a preliminary empirical study to answer RQ.