File(s) not publicly available

METTLE: A METamorphic Testing approach to assessing and validating unsupervised machine LEarning systems

journal contribution
posted on 07.12.2020, 00:00 authored by X Xie, Z Zhang, TY Chen, Y Liu, Pak PoonPak Poon, B Xu
Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users’ requirements and specific application scenarios/contexts are indisputably two important tasks. Such assessment and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, we develop a METamorphic Testing approach to assessing and validating unsupervised machine LEarning systems, abbreviated as METTLE. Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users’ perspectives. To support METTLE, we have further formulated 11 generic metamorphic relations (MRs), covering users’ generally expected characteristics that should be possessed by machine learning systems. We have performed an experiment and a user evaluation study to evaluate the viability and effectiveness of METTLE. Our experiment and user evaluation study have shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering’s perspective, rather than a designer’s or implementor’s perspective, who normally adopts a theoretical approach.

Funding

Other

History

Volume

69

Issue

4

Start Page

1293

End Page

1322

eISSN

1558-1721

ISSN

0018-9529

Publisher

Institute of Electrical and Electronics Engineers

Additional Rights

CC BY 4.0

Peer Reviewed

Yes

Open Access

Yes

Acceptance Date

04/02/2020

External Author Affiliations

Nanyang Technological University, Singapore; Nanjing University, Wuhan University, China; Swinburne University of Technology;

Era Eligible

Yes

Journal

IEEE Transactions on Reliability