An empirical study of fault detection effectiveness of metamorphic testing for object detectors

Object detectors (ODs) have been applied in various fields to identify target objects in pictures and videos. A commonly used approach for validating ODs is to compare their outputs with manually labeled data, in which labeling is tedious and time-consuming. To alleviate this problem, Metamorphic Testing (MT) based methods can be used to generate test cases automatically for testing ODs. Our survey has identified two different types of MT-based methods: input driven and output driven. We also observed that their relative effectiveness in fault detection has not yet been studied and compared. Inspired by this observation, we performed an empirical study with different types of ODs to address this research gap. In our study, we proposed an evaluation framework involving three different fault detection effectiveness metrics. An important finding of our study is that the input driven MT-based methods have better fault detection performance than their output driven counterparts.