Software development teams that use agile methodologies are increasingly adopting the test-driven development practice (TDD). TDD allows to produce software by iterative and incremental work cycle, and with a strict control over the process, favouring an early detection of bugs. However, when applied to large and complex systems, TDD benefits are not so obvious; manually locating and fixing bugs introduced during the iterative development steps is a nontrivial task. In such systems, the propagation chains following the bugs activation can be unacceptably long and intricate, and the size of the code to be analyzed is often too large. In this paper, a bug localization technique specifically tailored to TDD is presented. The technique is embedded in the TDD cycle, and it aims to improve developers' ability to locate bugs as soon as possible. It is implemented in a tool and experimentally evaluated on newly developed Java programs. 1. Introduction Test driven development (TDD) is a technique to incrementally develop software, that was sporadically used for decades, and that re-emerged in the last years as development paradigm in the context of the so-called agile methodologies. The increasing adoption of extreme programming (XP) in various industrial projects as well as the identification of TDD as a key strategy in agile software development have captured in recent years an increasing attention also by academic research. As stated by Janzen and Saiedian [1], the TDD is an example of how the academic research sometimes follows the most spread and accepted software practice, rather than leading it. The XP methodology is conceived to reduce the time-to-market of software systems, and the use of TDD practice in XP allows developing more robust programs. Several studies showed this trend, claiming the attention of various researchers, which started to study the ability of TDD to detect software faults earlier in the development process [2–5]. In TDD, the developer writes unit tests from a set of user requirements/functionalities, before writing the code itself. Then s/he implements the code needed to pass the tests, until he succeeds. When a bug is detected, it is promptly fixed. Once the tests are passed, the developer performs the refactoring of the code to acceptable standards; then s/he proceeds to define a new set of test cases for other functionalities and implements a new piece of code to pass them. In this cycle, even when the tests are successful, the code for new functionalities can easily compromise the previously implemented ones, introducing what
References
[1]
D. Janzen and H. Saiedian, “Test-driven development: concepts, taxonomy, and future direction,” Computer, vol. 38, no. 9, pp. 43–50, 2005.
[2]
R. Kaufmann and D. Janzen, “Implications of test-driven development: a pilot study,” in Proceedings of the 18th ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA '03), pp. 298–299, Anaheim, Calif, USA, 2003.
[3]
E. M. Maximilien and L. Williams, “Assessing test-driven development at IBM,” in Proceedings of the 25th IEEE International Conference on Software Engineering (ICSE '03), pp. 564–569, IEEE CS Press, May 2003.
[4]
A. Gupta and P. Jalote, “An experimental evaluation of the effectiveness and efficiency of the test driven development,” in Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM '07), pp. 285–294, September 2007.
[5]
R. C. Martin, “Professionalism and test-driven development,” IEEE Software, vol. 24, no. 3, pp. 32–36, 2007.
[6]
B. George and L. Williams, “A structured experiment of test-driven development,” Information and Software Technology, vol. 46, no. 5, pp. 337–342, 2004.
[7]
L. Williams, E. M. Maximilien, and M. Vouk, “Test-driven development as a defect-reduction practice,” in Proceedings of the 14th IEEE International Symposium on Software Reliability Engineering (ISSRE '03), p. 34, IEEE Computer Society, 2003.
[8]
B. George and L. Williams, “An initial investigation of test driven development in industry,” in Proceedings of the ACM Symposium on Applied Computing (SAC '03), pp. 1135–1139, ACM Press, 2003.
[9]
T. Bhat and N. Nagappan, “Evaluating the efficacy of test-driven development: industrial case studies,” in Proceedings of the 5th ACM/IEEE International Symposium on Empirical Software Engineering (ISESE '06), pp. 356–363, ACM Press, 2006.
[10]
A. Geras, M. Smith, and J. Miller, “A prototype empirical evaluation of test driven development,” in Proceedings of the 10th International Symposium on Software Metrics (METRICS '04), pp. 405–416, IEEE Computer Society, 2004.
[11]
M. M. Müller and O. Hagner, “Experiment about test-first programming,” in Proceedings of the International Conference of Software Engineering (ICSE '02), vol. 149, no. 5, pp. 131–136, 2002.
[12]
M. Pancur, M. Ciglaric, M. Trampus, and T. Vidmar, “Towards empirical evaluation of test-driven development in a university environment,” in Proceedings of the International Conference on Computer as a Tool (EUROCON '03), vol. 2, pp. 83–86, 2003.
[13]
M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin, “Dynamically discovering likely program invariants to support program evolution,” IEEE Transactions on Software Engineering, vol. 27, no. 2, pp. 99–123, 2001.
[14]
M. D. Ernst, J. H. Perkins, P. J. Guo et al., “The Daikon system for dynamic detection of likely invariants,” Science of Computer Programming, vol. 69, no. 1–3, pp. 35–45, 2007.
[15]
C. Liu, X. Yan, L. Fei, J. Han, and S. P. Midkiff, “SOBER: statistical model-based bug localization,” in Proceedings of the 10th European Software Engineering Conference (ESEC/FSE '05), pp. 286–295, 2005.
[16]
W. Dickinson, D. Leon, and A. Podgurski, “Finding failures by cluster analysis of execution profiles,” in Proceedings of the 23rd International Conference on Software Engineering (ICSE '01), pp. 339–348, 2001.
[17]
B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan, “Scalable statistical bug isolation,” in Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '05), pp. 15–26, 2005.
[18]
T. M. Chilimbi, B. Liblit, K. K. Mehra, A. V. Nori, and K. Vaswani, “HOLMES: effective statistical debugging via efficient path profiling,” in Proceedings of the 31st IEEE International Conference on Software Engineering (ICSE '09), pp. 34–44, IEEE Computer Society, 2009.
[19]
L. Jiang and Z. Su, “Context-aware statistical debugging: from bug predictors to faulty control flow paths,” in Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE '07), R. E. K. Stirewalt, A. Egyed, and B. Fischer, Eds., pp. 184–193, ACM Press, 2007.
[20]
B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan, “Scalable statistical bug isolation,” in Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '05), V. Sarkar and M. W. Hall, Eds., pp. 15–26, ACM Press, 2005.
[21]
J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula automatic fault-localization technique,” in Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE '05), D. F. Redmiles, T. Ellman, and A. Zisman, Eds., pp. 273–282, ACM Press, 2005.
[22]
M. Hiller, J. Christmansson, and M. Rimèn, “An experimental comparison of fault and error injection,” in Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE '98), pp. 369–378, 1998.
[23]
R. Chillarege, I. S. Bhandari, J. K. Chaar et al., “Orthogonal defect classification—a concept for in-process measurements,” IEEE Transactions on Software Engineering, vol. 18, no. 11, pp. 943–956, 1992.
[24]
J. Dur？es and H. Madeira, “Emulation of software faults: a field data study and a practical approach,” IEEE Transactions on Software Engineering, vol. 32, no. 11, pp. 849–867, 2006.
[25]
E. Clarke, O. Grumberg, and D. Peled, Model Checking, MIT Press, 1999.
[26]
T. Ball, M. Naik, and S. K. Rajamani, “From symptom to cause: localizing errors in counterexample traces,” in Proceedings of the 30th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL '03), pp. 97–105, 2003.
[27]
M. Weiser, “Program slicing,” IEEE Transactions on Software Engineering, vol. 10, no. 4, pp. 352–357, 1984.
[28]
M. Weiser, “Programmers use slicing when debugging,” Communications of the ACM, vol. 25, no. 7, pp. 446–452, 1982.
[29]
B. Korel and J. Laski, “Dynamic program slicing,” Information Processing Letters, vol. 29, no. 3, pp. 155–163, 1988.
[30]
S. Horwitz, B. Liblit, and M. Polishchuk, “Better debugging via output tracing and callstack-sensitive slicing,” IEEE Transactions on Software Engineering, vol. 36, no. 1, pp. 7–19, 2010.
[31]
D. Binkley, “Semantics guided regression test cost reduction,” IEEE Transactions on Software Engineering, vol. 23, no. 8, pp. 498–516, 1997.
[32]
J. Krinke, “Context-sensitivity matters, but context does not,” in Proceedings of the IEEE International Workshop on Source Code Analysis and Manipulation (SCAM '04), pp. 29–35, 2004.
[33]
L. Mariani and M. Pezzè, “Behavior capture and test: automated analysis of component integration,” in Proceedings of the 10th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS '05), pp. 292–301, 2005.
[34]
V. Dallmeier, C. Lindig, A. Wasylkowski, and A. Zeller, “Mining object behavior with ADABU,” in Proceedings of the International Conference on Software Engineering (ICSE '06), pp. 17–24, 2006.
[35]
B. Schmerl, D. Garlan, and H. Yan, “Dinamically discovering architectures with DiscoTect,” in Proceedings of the European Conference on Software Engineering (ESEC/FSE '05), pp. 103–106, 2005.
[36]
S. Hangal and M. S. Lam, “Tracking down software bugs using automatic anomaly detection,” in Proceedings of the 24th International Conference on Software Engineering (ICSE '02), pp. 291–301, 2002.
[37]
D. Lorenzoli, L. Mariani, and M. Pezzè, “Automatic generation of software behavioral models,” in Proceedings of the 30th International Conference on Software Engineering (ICSE '08), pp. 501–510, ACM Press, New York, NY, USA, 2008.
[38]
A. Podgurski, D. Leon, P. Francis et al., “Automated support for classifying software failure reports,” in Proceedings of the 25th International Conference on Software Engineering (ICSE '03), pp. 465–475, 2003.
[39]
M. B. Dwyer, A. Kinneer, and S. G. Elbaum, “Adaptive online program analysis,” in Proceedings of the 29th IEEE International Conference on Software Engineering (ICSE '07), pp. 220–229, IEEE Computer Society, 2007.
[40]
R. Santelices, J. A. Jones, Y. Yu, and M. J. Harrold, “Lightweight fault-localization using multiple coverage types,” in Proceedings of the 31st IEEE International Conference on Software Engineering (ICSE '09), pp. 56–66, IEEE Computer Society, 2009.
[41]
A. Zeller and R. Hildebrandt, “Simplifying and isolating failure-inducing input,” IEEE Transactions on Software Engineering, vol. 28, no. 2, pp. 183–200, 2002.
[42]
H. Cleve and A. Zeller, “Locating causes of program failures,” in Proceedings of the 27th International Conference on Software Engineering (ICSE '05), pp. 342–351, ACM Press, New York, NY, USA, 2005.
[43]
A. Zeller, “Yesterday, my program worked. Today, it does not. Why?” in Proceedings of the 7th European Software Engineering Conference Held Jointly with the 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE '07), pp. 253–267, Springer, London, UK, 2007.
[44]
J. Tucek, S. Lu, C. Huang, S. Xanthos, and Y. Zhou, “Triage: diagnosing production run failures at the user's site,” in Proceedings of the 21st ACM SIGOPS Symposium on Operating Systems Principles (SOSP '07), pp. 131–144, ACM Press, 2007.
[45]
J. Clause and A. Orso, “A technique for enabling and supporting debugging of field failures,” in Proceedings of the 29th International Conference on Software Engineering (ICSE '07), pp. 261–270, IEEE Computer Society, 2007.
[46]
C. Yilmaz, A. M. Paradkar, and C. Williams, “Time will tell: fault localization using time spectra,” in Proceedings of the 30th International Conference on Software Engineering (ICSE '08), W. Schafer, M. B. Dwyer, and V. Gruhn, Eds., pp. 81–90, ACM Press, 2008.
[47]
A. J. Ko and B. A. Myers, “Debugging reinvented: asking and answering why and why not questions about program behavior,” in Proceedings of the 30th International Conference on Software Engineering (ICSE '08), pp. 301–310, ACM Press, New York, NY, USA, 2008.
[48]
G. Xu and A. Rountev, “Precise memory leak detection for java software using container profiling,” in Proceedings of the 30th International Conference on Software Engineering (ICSE '08), pp. 151–160, ACM Press, New York, NY, USA, 2008.
[49]
J. Clause and A. Orso, “LEAKPOINT: pinpointing the causes of memory leaks,” in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (ICSE '10), vol. 1, pp. 515–552, ACM Press, New York, NY, USA, 2010.
[50]
S. Park, R. W. Vuduc, and M. J. Harrold, “Falcon: fault localization in concurrent programs,” in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (ICSE '10), vol. 1, pp. 245–254, ACM Press, New York, NY, USA, 2010.
