In general, modern programs are large and complex and it is essential that they should be highly reliable in applications. In order to develop highly reliable software, Java programming language developer provides a rich set of exceptions and exception handling mechanisms. Exception handling mechanisms are intended to help developers build robust programs. Given a program with exception handling constructs, for an effective testing, we are to detect whether all possible exceptions are raised and caught or not. However, complex exception handling constructs make it tedious to trace which exceptions are handled and where and which exceptions are passed on. In this paper, we address this problem and propose a mutation analysis approach to develop reliable object-oriented programs. We have applied a number of mutation operators to create a large set of mutant programs with different type of faults. We then generate test cases and test data to uncover exception related faults. The test suite so obtained is applied to the mutant programs measuring the mutation score and hence verifying whether mutant programs are effective or not. We have tested our approach with a number of case studies to substantiate the efficacy of the proposed mutation analysis technique. 1. Introduction Of late, computer applications have permeated heavily into every sphere of the daily life of an average person. Many of these applications are large, complex, and safety critical, requiring the software to be extremely reliable. An exception occurrence during a program execution is an abnormal computation state [1] and thus is a threat to the reliability of software. It is advocated that software should be tested adequately to trace if there is any exception in intended behavior [2]. To realize this software developers prefer exception handling mechanisms to be embedded inside code so that software can arrest and raise alarm whenever there is any exception and then cause of exception can be eliminated. Many reasons may be attributed to causes of exceptions such as erroneous input, hardware faults, logical errors in the code, semantic violation, linking errors, and limitation of system resources. It is reported that more than 50% of the operational failures that may occur in a system are due to faults in exception handling and recovery [3]. To cope with the system failures due to exceptions, most modern programming languages such as Ada, C++ and Java incorporate explicit mechanisms for exception handling. Syntactically, an exception handling mechanism consists of a means to explicitly
References
[1]
J. B. Goodenough, “Exception handling: issues and proposed notations,” Communications of the ACM, vol. 18, no. 12, pp. 683–696, 1975.
[2]
J. D. Musa, Software Reliability Engineering, McGraw-Hill, London, UK, 1998.
[3]
W. N. Toy, “Fault tolerant design of local ESS processor,” in The Theory and Practice of Reliable System Design, 1981.
[4]
S. Sinha and M. J. Harrold, “Analysis and testing of programs with exception handling constructs,” IEEE Transactions on Software Engineering, vol. 26, no. 9, pp. 849–871, 2000.
[5]
B. G. Ryder, D. Smith, U. Kremer, M. Gordon, and N. Shah, “A static study of java exceptions using JSEP,” Tech. Rep. DCS-TR-403, Rutgers University, November 1999.
[6]
N. Tracey, J. Clark, K. Mander, and J. McDermid, “Automated test-data generation for exception conditions,” Software: Practice and Experience, vol. 30, pp. 61–79, 2000.
[7]
A. Bertolino, “Software testing research: achievements, challenges, dreams,” in Proceedings of the Future of Software Engineering (FoSE '07), pp. 85–103, May 2007.
[8]
M. J. Gallagher and V. L. Narasimhan, “Adtest: a test data generation suite for ada software systems,” IEEE Transactions on Software Engineering, vol. 23, no. 8, pp. 473–484, 1997.
[9]
S. Sinha and M. J. Harrold, “Criteria for testing exception-handling constructs in Java programs,” in Proceedings of the IEEE International Conference on Software Maintenance (ICSM'99), pp. 265–275, September 1999.
[10]
M. P. Robillard and G. C. Murphy, “Analyzing exception flow in java TM programs,” in Proceedings of the Joint 7 th European Software Engineering Conference and the 7 th ACM SIGSOFT Internation al Symposium on the Foundations of Software Engineering (LNCS '99), vol. 1687, pp. 322–337, Toulouse, France, Septembe 1999.
[11]
J.-D. Choi, D. Grove, M. Hind, and V. Sarkar, “Efficient and precise modeling of exceptions for the analysis of Java programs,” in Proceedings of the ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE '99), pp. 21–31, 1999.
[12]
T. A. Budd and D. Angluin, “Two notions of correctness and their relation to testing,” Acta Informatica, vol. 18, no. 1, pp. 31–45, 1982.
[13]
S.-W. Kim, J. A. Clark, and J. A. McDermid, “Investigating the effectiveness of object-oriented testing strategies using the mutation method,” Software Testing Verification and Reliability, vol. 11, no. 4, pp. 207–225, 2001.
[14]
J. H. Andrews, L. C. Briand, and Y. Labiche, “Is mutation an appropriate tool for testing experiments?” in Proceedings of the 27th International Conference on Software Engineering (ICSE '05), pp. 402–411, IEEE, May 2005.
[15]
C. Ji, Z. Chen, B. Xu, and Z. Wang, “A new mutation analysis method for testing java exception handling,” in Proceedings of the 33rd Annual IEEE International Computer Software and Applications Conference (COMPSAC '09), vol. 2, pp. 556–561, IEEE Computer Society Press, July 2009.
[16]
http://junit.org/.
[17]
http://www.jumble.sourceforge.net/.
[18]
http://www.java.com/.
[19]
D. Samanta, Object-Oriented Programming with C++ and Java, Prentice Hall of India, New Delhi, India, 2003.
[20]
J. Gosling, B. Joy, and G. Steele, The Java Language Specification, Addison-Wesley, Reading, Mass, USA, 1996.
[21]
P. A. Buhr, “Advanced exception handling mechanisms,” IEEE Transactions on Software Engineering, vol. 26, no. 9, pp. 820–836, 2000.
[22]
G. J. Myers, The Art of Software Testing, John Wiley & Sons, Hoboken, NJ, USA, 2nd edition, 2004.
[23]
R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: help for the practicing programmer,” IEEE Computer, vol. 11, no. 4, pp. 34–41, 1978.
[24]
M. E. Delamaro, J. C. Maldonado, and A. P. Mathur, “Interface mutation: an approach for integration testing,” IEEE Transactions on Software Engineering, vol. 27, no. 3, pp. 228–247, 2001.
[25]
J. B. Rainsberger and S. Stirling, JUnit Recipes: Practical Methods for Programmer Testing, Manning, Greenwich, UK, 2005.
[26]
S. A. Irvine, T. Pavlinic, L. Trigg, J. G. Cleary, S. Inglis, and M. Utting, “Jumble java byte code to measure the effectiveness of unit tests,” in Proceedings of the IEEE Proceedings of Testing: Academic and Industrial Conference—Practice and Research Techniques, pp. 169–175, September 2007.
[27]
Y.-S. Ma, J. Offutt, and Y.-R. Kwon, “MuJava: a mutation system for java,” in Proceedings of the 28th International Conference on Software Engineering (ICSE '06), pp. 827–830, ACM, May 2006.
[28]
http://cs.gmu.edu/~offutt/mujava/.
[29]
K. Dobolyi and W. Weimer, “Changing java's semantics for handling null pointer exceptions,” in Proceedings of the 19th International Symposium on Software Reliability Engineering (ISSRE '08), pp. 47–56, November 2008.
[30]
L. Briand and Y. Labiche, “A UML-based approach to system testing,” Tech. Rep. TR SCE-01-01, Version 4, Carleton University, 2002.
[31]
A. Abdurazik, J. Offutt, and A. Baldini, “A comparative evaluation of tests generated from different UML diagrams: diagrams and data,” Tech. Rep. ISE-TR-05-04, George Mason University, Fairfax, Va, USA, 2005.
[32]
M. Sarma, Automatic test specification generation for state-based system testing [Ph.D. thesis], Indian Institute of Technology, Kharagpur, India, 2008.
[33]
M. Sarma, “System state model generation from UML 2.0,” Tech. Rep. CSE-TR-04-07, Indian Institute of Technology, Kharagpur, India, April 2007.
[34]
L. C. Briand, J. Cui, and Y. Labiche, “Towards automated support for deriving test data from UML statecharts,” in Proceedings of the Unified Modeling Language Conference (UML '03), vol. 2863 of Lecture Notes in Computer Science, pp. 249–264, Springer, San Francisco, Calif, USA, October 2003.
[35]
S. C. Ntafos, “A comparison of some structural testing strategies,” IEEE Transactions on Software Engineering, vol. 14, no. 6, pp. 868–874, 1988.
[36]
J. C. Huang, “An approach to program testing,” ACM Computing Surveys, vol. 7, no. 3, pp. 114–128, 1975.
[37]
P. G. Frankl and E. J. Weyuker, “An applicable family of data flow testing criteria,” IEEE Transactions on Software Engineering, vol. 14, no. 10, pp. 1483–1498, 1988.
[38]
M. J. Harrold and M. L. Soffa, “Interprocedural data flow testing,” in Proceedings of the ACM SIGSOFT of 3rd Symposium of Software Testing, Analysis and Verification (SIGSOFT '89), pp. 158–167, December 1989.
[39]
J. W. Laski and B. Korel, “A data flow oriented program testing strategy,” IEEE Transactions on Software Engineering, vol. 9, no. 3, pp. 347–354, 1983.
[40]
S. Rapps and E. J. Weyuker, “Selecting software test data using data flow information,” IEEE Transactions on Software Engineering, vol. 11, no. 4, pp. 367–375, 1985.
[41]
P. G. Frankl and S. N. Weiss, “Experimental comparison of the effectiveness of branch testing and data flow testing,” IEEE Transactions on Software Engineering, vol. 19, no. 8, pp. 774–787, 1993.
[42]
W. E. Howden, “Methodology for the generation of program test data,” IEEE Transactions on Computers, vol. 24, no. 5, pp. 554–560, 1975.
[43]
T. J. McCabe, “A complexity measure,” IEEE Transactions on Software Engineering, vol. 2, no. 4, pp. 308–320, 1976.
[44]
P. G. Frankl and E. J. Weyuker, “Provable improvements on branch testing,” IEEE Transactions on Software Engineering, vol. 19, no. 10, pp. 962–975, 1993.
[45]
B. Beizer, Software Testing Techniques, Thomson Computer Press, New York, NY, USA, 2nd edition, 1990.
[46]
R. Boyer, B. Elspas, and K. Levitt, “SELECT—a formal system for testing and debugging programs by symbolic execution,” in Proceedings of International Conference on Reliable Software, pp. 234–245, 1975.
[47]
L. A. Clarke, “A system to generate test data and symbolically execute programs,” IEEE Transactions on Software Engineering, vol. 2, no. 3, pp. 215–222, 1976.
[48]
J. C. King, “Symbolic execution and program testing,” Communications of the Association for Computing Machinery, vol. 19, no. 7, pp. 385–394, 1976.
[49]
C. V. Ramamoorthy, S.-B. F. Ho, and W. T. Chen, “On the automated generation of program test data,” IEEE Transactions on Software Engineering, vol. 2, no. 4, pp. 293–300, 1976.
[50]
R. A. DeMillo and A. J. Offutt, “Experimental results from an automatic test case generator,” ACM Transactions on Software Engineering and Methodology, vol. 2, no. 2, pp. 109–127, 1993.
[51]
B. Korel, “Automated software test data generation,” IEEE Transactions on Software Engineering, vol. 16, no. 8, pp. 870–879, 1990.
[52]
B. F. Jones, H.-H. Sthamer, and D. E. Eyres, “Automatic structural testing using genetic algorithms,” Software Engineering Journal, vol. 11, no. 5, pp. 299–306, 1996.
[53]
R. A. DeMillo and A. J. Offutt, “Constraint-based automatic test data generation,” IEEE Transactions on Software Engineering, vol. 17, no. 9, pp. 900–910, 1991.
[54]
A. J. Offutt, Z. Jin, and J. Pan, “The dynamic domain reduction procedure for test data generation,” Software—Practice and Experience, vol. 29, no. 2, pp. 167–193, 1999.
[55]
R. V. Binder, Testing Object Oriented Systems: Models, Patterns and Tools, The Object Technology, Addison-Wesley, 1999.
[56]
J. Z. Gao, H.-S. J. Tsao, and Y. Wu, Testing and Quality Assurance for Component-Based Software, Artech House, Norwood, Mass, USA, 2003.
[57]
K. Adamopoulos, M. Harman, and R. M. Hierons, “How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution,” in Proceedings of the Genetic and Evolutionary Computation Conference (LNCS '04), vol. 3103, pp. 1338–1349, Seattle, Wash, USA, June 2004.
[58]
A. J. Offutt and J. Pan, “Automatically detecting equivalent mutants and infeasible paths,” Software Testing Verification and Reliability, vol. 7, no. 3, pp. 165–192, 1997.
