This paper presents a selective regression testing technique and an associated tool for object-oriented software. The technique is based on the concept of Control Call Graphs, which are a reduced form of traditional Control Flow Graphs. It uses static analysis of the source code of the program. The developed tool (1) identifies the Control Call Paths potentially impacted by changes, (2) selects, from an existing test suite, the appropriate test cases, and (3) generates new JUnit test cases for control call paths that are not covered by existing tests (new ones, or those whose structure has been modified after changes). In this way, the approach supports an incremental update of the test suite. The selected JUnit test cases, including the new ones, are automatically executed. Three concrete case studies are reported to provide evidence of the feasibility of the approach and its benefits in terms of reduction of regression testing effort. 1. Introduction Software systems need to continually evolve for various reasons: adding new features to satisfy user requirements, changing business needs, introducing novel technologies, correcting faults, improving quality, and so forth. It is, therefore, important to ensure that modifications do not adversely affect the software. Software maintenance is an essential activity [1]. The costs of software maintenance keep, however, increasing [2]. Among the various maintenance activities, regression testing represents a crucial one. Regression testing is actually an important activity to ensure software quality, particularly when software is actively maintained and updated. It can also be used in the testing release phase of software development. Regression testing is the process that consists of determining if a modified software system still verifies its specification and whether new errors were introduced inadvertently [3–5]. For obvious reasons, the retest-all approach that consists in rerunning every test case in the initial test suite produced during initial development is inefficient, costly, and unacceptable in the maintenance phase [6]. Moreover, it does not consider obsolete and new test cases. In addition, it is often impractical due to the development cost and delivery schedule constraints [7]. An alternative approach, known as a selective retest strategy, assumes that not all parts of a software system are affected by changes [8]. Regression test selection in this case consists in selecting and running, from an initial test suite, a reduced subset of test cases in order to verify the behavior of modified
References
[1]
K. Bennett and V. Rajlich, “Software maintenance and evolution: a roadmap,” in Proceedings of the Conference on the Future of Software Engineering, pp. 73–87, ACM, New York, NY, USA, 2000.
[2]
I. Sommerville, Software Engineering, Addison Wesley, 9th edition, 2010.
[3]
D. C. Kung, J. Gao, P. Hsia, J. Lin, and Y. Toyoshima, “Class firewall, test order, and regression testing of object-oriented programs,” Journal of Object-Oriented Programming, vol. 8, no. 2, pp. 51–65, 1995.
[4]
G. Rothermel and M. J. Harrold, “A safe, efficient regression test selection technique,” ACM Transactions on Software Engineering and Methodology, vol. 6, no. 2, pp. 173–210, 1997.
[5]
M. J. Harrold, J. A. Jones, T. Li et al., “Regression test selection for Java software,” in Proceedings of the Conference on Object-Oriented Programming, Systems, Languages and Applications (OOPSLA '01), pp. 312–326, October 2001.
[6]
G. Rothermel and M. J. Harrold, “Analyzing regression test selection techniques,” IEEE Transactions on Software Engineering, vol. 22, no. 8, pp. 529–551, 1996.
[7]
Y. Wu, M. H. Chen, and H. M. Kao, “Regression testing on object-oriented programs,” in Proceedings of the 10th International Symposium on Software Reliability Engineering (ISSRE '99), pp. 270–279, November 1999.
[8]
O. Pilskalns, G. Uyan, and A. Andrews, “Regression testing UML designs,” in Proceedings of the 22nd IEEE International Conference on Software Maintenance (ICSM '06), pp. 254–263, September 2006.
[9]
T. L. Graves, M. J. Harrold, J. M. Kim, A. Porter, and G. Rothermel, “An empirical study of regression test selection techniques,” ACM Transactions on Software Engineering and Methodology, vol. 10, no. 2, pp. 184–208, 2001.
[10]
E. Engstr？m, M. Skoglund, and P. Runeson, “Empirical evaluations of regression test selection techniques: a systematic review,” in Proceedings of the 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM '08), pp. 22–31, October 2008.
[11]
H. K. N. Leung and L. White, “A study of integration testing and software regression at the integration level,” in Proceedings of the Conference on Software Maintenance, pp. 290–301, San Diego, Calif, USA, November 1990.
[12]
J. Laski and W. Szermer, “Identification of program modifications and its applications in software maintenance,” in Proceedings of the Conference on Software Maintenance, 1992.
[13]
E. Engstr？m, P. Runeson, and M. Skoglund, “A systematic review on regression test selection techniques,” Information and Software Technology, vol. 52, no. 1, pp. 14–30, 2010.
[14]
L. J. White and H. K. N. Leung, “A firewall concept for both control-flow and data-flow in regression integration testing,” in Proceedings of the Conference of Software Maintenance, pp. 262–271, 1992.
[15]
G. Rothermel and M. J. Harrold, “A comparison of regression test selection techniques,” Tech. Rep. 114, Clemson University, Clemson, SC, USA, April 1993.
[16]
G. Rothermel and M. J. Harrold, “Framework for evaluating regression test selection techniques,” in Proceedings of the 16th International Conference on Software Engineering, pp. 201–210, May 1994.
[17]
P. Hsia, X. Li, D. C. Kung et al., “A technique for the selective revalidation of OO software,” Journal of Software Maintenance and Evolution, vol. 9, no. 4, pp. 217–233, 1997.
[18]
L. White and K. Abdullah, “A firewall approach for the regression testing of object oriented software,” Software Quality Week, 1997.
[19]
M. Skoglund and P. Runeson, “A case study of the class firewall regression test selection technique on a large scale distributed software system,” in Proceedings of the International Symposium on Empirical Software Engineering (ISESE '05), pp. 74–83, November 2005.
[20]
H. K. Leung and L. White, “A cost model to compare regression test strategies,” in Proceedings of the International Conference on Software Maintenance, pp. 201–208, 1991.
[21]
L. C. Briand, Y. Labiche, and G. Soccar, “Automating impact analysis and regression test selection based on UML designs,” in Proceedings of IEEE International Conference on Software Maintenance, pp. 252–261, October 2002.
[22]
Y. Chen, R. L. Probert, and H. Ural, “Model-based regression test suite generation using dependence analysis,” in Proceedings of the 3rd International Workshop Advances in Model Based Testing (AMOST '07), pp. 54–62, September 2007.
[23]
Y. Li and N. J. Wahl, “An overview of regression testing,” ACM Software Engineering Notes, vol. 24, no. 1, pp. 69–73, 1999.
[24]
A. Von Mayrhauser and N. Zhang, “Automated regression testing using DBT and sleuth,” Journal of Software Maintenance and Evolution, vol. 11, no. 2, pp. 93–116, 1999.
[25]
P. L. Vincent, L. Badri, and M. Badri, “Regression testing of object-oriented software: a technique based on use cases and associated tool,” in Proceedings of the International Conference on Advanced Software Engineering & Its Applications (ASEA '12), T.-H. Kim, C. Ramos, H. Kim, A. Kiumi, S. Mohammed, and D. ？l？zak, Eds., Lecture Notes in Computer Science, Springer, Jeju, Korea, December 2012.
[26]
M. Fahad and A. Nadeem, “A survey of UML based regression testing,” in IFIP International Federation for Information Processing, Z. Shi, E. Mercier-Laurent, and D. Leake, Eds., vol. 288, pp. 200–210, Springer, Boston, Mass, USA, 2008.
[27]
K. Abdullah, The firewall concept for regression testing and impact analysis of object oriented systems [Ph.D. thesis], Case Western Reserve University, 1998.
[28]
G. Rothermel, M. J. Harrold, and J. Dedhia, “Regression test selection for C++ software,” Journal of Software Testing, Verification and Reliability, vol. 10, no. 2, pp. 77–109, 2000.
[29]
L. White, K. Jaber, and B. Robinson, “Utilization of extended firewall for object-oriented regression testing,” in Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM '05), pp. 695–698, September 2005.
[30]
N. Mansour, H. Takkoush, and A. Nehme, “UML-based regression testing for OO software,” Journal of Software Maintenance and Evolution, vol. 23, no. 1, pp. 51–68, 2011.
[31]
Y. Chen, R. L. Probert, and D. P. Sims, “Specification-based regression test selection with risk analysis,” in IBM Center Advanced Studies Conference, 2002.
[32]
J. Law and G. Rothermel, “Whole program path-based dynamic impact analysis,” in Proceedings of the 25th International Conference on Software Engineering, pp. 308–318, May 2003.
[33]
L. Badri, M. Badri, and D. St-Yves, “Supporting predictive change impact analysis: a control call graph based technique,” in Proceedings of the 12th Asia-Pacific Software Engineering Conference (APSEC '05), pp. 167–175, December 2005.
[34]
A. Orso, T. Apiwattanapong, and M. J. Harrold, “Leveraging field data for impact analysis and regression testing,” in Proceedings of the 9th European Software Engineering Conference Held Jointly with 11th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE '11), pp. 128–137, Helsinki, Finland, September 2003.
[35]
W. E. Wong, J. R. Horgan, S. London, and H. Agrawal, “Study of effective regression testing in practice,” in Proceedings of the 8th International Symposium on Software Reliability Engineering (ISSRE '97), pp. 264–274, November 1997.
[36]
P. Ammann, P. E. Black, and W. Ding, “Model checkers in software testing,” NIST-IR 6777, National Institute of Standards and Technology, 2002.
[37]
D. Wu, M. A. Hennell, D. Hedley, and I. J. Riddell, “A practical method for software quality control via program mutation,” in Proceedings of the 2nd Workshop on Software Testing, Verification, and Analysis, pp. 159–170, 1988.
[38]
G. Larman, UML et les Design Pattern, Campus Press, 2002.
[39]
Y. F. Chen, D. S. Rosenblum, and K. P. Vo, “TestTube: a system for selective regression testing,” in Proceedings of the 16th International Conference on Software Engineering, pp. 211–220, Sorrento, Italy, May 1994.
[40]
B. Korel and A. M. Al-Yami, “Automated regression test generation,” in Proceedings of ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 98), pp. 143–152, Clearwater Beach, Fla, USA, 1998.
[41]
T. Ball, “On the limit of control flow analysis for regression test selection,” in Proceedings of ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 98), pp. 134–142, Clearwater Beach, Fla, USA, 1998.
[42]
A. K. Onoma, W. T. Tsai, M. H. Poonawala, and H. Suganuma, “Regression testing in an industrial environment,” Communications of the ACM, vol. 41, no. 5, pp. 81–86, 1998.
[43]
L. White, H. Almezen, and S. Sastry, “Firewall regression testing of GUI sequences and their interactions,” in Proceedings of the International Conference on Software Maintenance, pp. 398–409, September 2003.
[44]
T. Koju, S. Takada, and N. Doi, “Regression test selection based on intermediate code for virtual machines,” in Proceedings of the International Conference on Software Maintenance, pp. 420–429, September 2003.
