Skip to Main Content
 

Global Search Box

 
 
 
 

Files

File List


45565.pdf (6.42 MB)

ETD Abstract Container

Abstract Header

Examining Metamorphic Testing with Requirements Knowledge in Practical Settings

Peng, Zedong
http://orcid.org/0000-0002-5071-1586
http://rave.ohiolink.edu/etdc/view?acc_num=ucin1692299078054155

Abstract Details

2023, PhD, University of Cincinnati, Engineering and Applied Science: Computer Science and Engineering.
Given a test input, not knowing the expected output of the software under test (SUT) is called the oracle problem. An emerging method of alleviating the oracle problem is metamorphic testing (MT). Rather than focusing on the correctness of output from a single execution of the SUT, MT exploits metamorphic relations (MRs) as derived oracles for checking the functional correctness of the code. Although researchers have argued that MT can be a simple and effective technique to help software developers, little is known about the actual cost of constructing MRs in real-world software and the relationship between MT and the already well-adopted method in software development. This research examines a series of practices to evaluate the effectiveness of MT during software development. Our investigation is conducted within the context of a real-world scientific software, the Storm Water Management Model (SWMM), developed and maintained by the U.S. Environmental Protection Agency (EPA). To ensure SWMM's accuracy in modeling stormwater runoff and executing hydraulic and water quality simulations, the development team continually evolves the software. Among the challenges they face, software testing stands out as one of the most technically complex tasks. Our research initially investigates the current testing practices and software quality assurance (QA) workflows in scientific software development, taking the SWMM as a case study. The value of our work resides in the qualitative characterizations and quantitative assessments of the tests that scientific software developers have independently written and released within the SWMM context. Our findings indicate that oracles indeed play a role in scientific software testing. Furthermore, by employing an empirical approach, we identified four critical requirements for the improved integration of MT into scientific software development. Constructing MRs is critical because without them, MT cannot be performed. Meanwhile, constructing MRs is typically a costly task. Therefore, we present an approach to classifying input and output (I/O) variables from scientific software's user manual, then mining these variables' associations from the user forum to reduce the cost of constructing MRs. On the other hand, regression testing has long been used by scientific software developers to tackle the oracle problem amid code changes. Although MT has the potential to complement regression testing by checking functional correctness more directly, the cost of constructing MRs and their added advantage in an evolving and highly specialized scientific software system are not yet understood. To further explore this, we conducted an exploratory case study focusing on MR construction by scientific software developers, enabling us to critically evaluate the value addition MT brings to software development. This dissertation offers significant contributions in three distinct areas: Firstly, it fosters a better understanding of the software QA workflow and existing test practices, which in turn aids in more effective support for MT adoption. Secondly, it provides automated support for enhancing the diversity of MRs and reducing the cost associated with MT. Lastly, it scrutinizes the feasibility of MT and its additional advantages compared to regression testing, demonstrated through a real-world system.
Nan Niu, Ph.D. (Committee Chair)
Wen-Ben Jone, Ph.D. (Committee Member)
Tingting Yu, Ph.D. (Committee Member)
Boyang Wang, Ph.D. (Committee Member)
Michelle Simon, Ph.D. (Committee Member)
152 p.
Computer Science
scientific software; metamorphic testing; quality assurance; regression testing; water management model; software engineering

Recommended Citations

Citations

  • Peng, Z. (2023). Examining Metamorphic Testing with Requirements Knowledge in Practical Settings [Doctoral dissertation, University of Cincinnati]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=ucin1692299078054155

    APA Style (7th edition)

  • Peng, Zedong. Examining Metamorphic Testing with Requirements Knowledge in Practical Settings. 2023. University of Cincinnati, Doctoral dissertation. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=ucin1692299078054155.

    MLA Style (8th edition)

  • Peng, Zedong. "Examining Metamorphic Testing with Requirements Knowledge in Practical Settings." Doctoral dissertation, University of Cincinnati, 2023. http://rave.ohiolink.edu/etdc/view?acc_num=ucin1692299078054155

    Chicago Manual of Style (17th edition)

ucin1692299078054155
87
© 2023, some rights reserved.Creative Commons License Examining Metamorphic Testing with Requirements Knowledge in Practical Settings by Zedong Peng is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. Based on a work at etd.ohiolink.edu.
This open access ETD is published by University of Cincinnati and OhioLINK.