An artificial intelligence system, designed for operations in a real-world environment faces a nearly infinite set of possible performance scenarios. Designers and developers, thus, face the challenge of validating proper performance across both foreseen and unforeseen conditions, particularly when the artificial intelligence is controlling a robot that will be operating in close proximity, or may represent a danger, to humans. While the manual creation of test cases allows limited testing (perhaps ensuring that a set of foreseeable conditions trigger an appropriate response), this may be insufficient to fully characterize and validate safe system performance. An approach to validating the performance of an artificial intelligence system using a simple artificial intelligence test case producer (AITCP) is presented. The AITCP allows the creation and simulation of prospective operating scenarios at a rate far exceeding that possible by human testers. Four scenarios for testing an autonomous navigation control system are presented: single actor in two-dimensional space, multiple actors in two-dimensional space, single actor in three-dimensional space, and multiple actors in three-dimensional space. The utility of using the AITCP is compared to that of human testers in each of these scenarios.
References
[1]
Halawani, S. Safety Issues of computer Failure. Technical Report. Available online: http://amubaraki.kau.edu.sa/Files/830/Researches/55979_26288.doc (accessed on December 7, 2012).
[2]
AdiSrikanth; Kulkarni, N.J.; Naveen, K.V.; Singh, P.; Srivastava., P.R. Test Case Optimization Using Artificial Bee Colony Algorithm. In Proceedings of ACC 2011, Part III, CCIS 192; Springer Verlag: New York, NY, 2011; pp. 570–579.
[3]
Mondada, F.; Floreano, D. Evolution of neural control structures: some experiments on mobile robots. Robotics Auton. Syst. 1995, 16, 183–195, doi:10.1016/0921-8890(96)81008-6.
[4]
Felgenbaum, E.A. The Art of Artificial Intelligence, Stanford University Technical Report, STAN-CS-77-621. Stanford University, Cambridge, MA, 1977.
[5]
Chandrasekaran, B. On Evaluating AI Systems for Medical Diagnosis. The AI Magazine 1983, 48, 34–37.
[6]
Cholewinski, P.; Marek, V.W.; Mikitiuk, A.; Truszczynski, M. Computing with Default Logic. Artif. Intell. 1999, 112, 105–146, doi:10.1016/S0004-3702(99)00053-3.
[7]
Brooks, R.A. Artificial Life and Real Robots; MIT Artificial Intelligence Laboratory: Cambridge, MA, USA. (in press).
[8]
Boooks, R.A. Elephants Don’t Play Chess. In Designing Autonomous Agents: Theory and Practice from Biology to Engineering and Back; Maes, P., Ed.; MIT Press: Cambridge, MA, USA, 1990.
[9]
Brooks, R.A. Intelligence Without Reason. In Proceedings of the International Joint Conferences on Artificial Intelligence; Morgan Kaufmann Publishers: San Francisco, CA, USA, 1991; pp. 569–595.
[10]
Brooks, R.A. New Approaches to Robotics. Science 1991, 253, 1227–1232.
[11]
Billings, D.; Davidson, A.; Schaeffer, J.; Szafron, D. The Challenge of Poker. Artif. Intell. 2002, 134, 201–240, doi:10.1016/S0004-3702(01)00130-8.
[12]
Dai, P.; Mausam; Weld, D.S. Artificial Intelligence for Artificial Artificial Intelligence. In Proceedings of the 25th AAAI Conference on Artificial Intelligence; AAAI Press: Palo Alto, CA, USA, 2011; pp. 1153–1159.
[13]
Pitchforth, J.; Mengersen, K. A Proposed Validation Framework for Expert Elicited Bayesian Networks. Expert Syst. Appl. 2012, 40, 162–167, doi:10.1016/j.eswa.2012.07.026.
[14]
Wotawa, F.; Nica, S.; Nica, M. Debugging and test case generation using constraints and mutations. In Proceedings of the 9th Workshop on Intelligent Solutions in Embedded Systems (WISES); IEEE: New York, NY, USA, 2011; pp. 95–100.
[15]
Suri, B.; Singhal, S. Analyzing Test Case Selection & Prioritization using ACO. ACM SIGSOFT Softw. Eng. Notes 2011, 36, 1–5.
[16]
Harman, M. The role of Artificial Intelligence in Software Engineering. In Proceedings of the 1st International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE); IEEE: New York, NY, USA, 2012; pp. 1–6.
[17]
Pop, C.B.; Chifu, V.R.; Salomie, I.; Baico, R.B.; Dinsoreanu, M; Copil, G. A Hybrid Firelfy–Inspired Approach for Optimal Semantic Web Service Composition. Scalable Comput.: Pract. Exp. 2011, 12, 363–369.
[18]
Shah-Hosseini, H. Problem Solving by Intelligent Water Drops. In Proceedings of the IEEE Congress on Evolutionary Computation, Singapore, 25–28 September 2007; IEEE: New York, NY, USA, 2007; pp. 3226–3231.
[19]
Duan, H.; Liu, S.; Wu, J. Novel Intelligent Water Drops Optimization Approach to Single UCAV Smooth Trajectory Planning. Aerosp. Sci. Technol. 2009, 13, 442–449, doi:10.1016/j.ast.2009.07.002.
[20]
Gendreau, M.; Hertz, A.; Laporte, G. A Tabu Search Heuristic for the Vehicle Routing Problem. Manag. Sci. 1994, 40, 1276–1290, doi:10.1287/mnsc.40.10.1276.
[21]
Glover, F. Heuristic for Integer Programming Using Surrogate Constraints. Decis. Sci. 1977, 8, 156–166, doi:10.1111/j.1540-5915.1977.tb01074.x.
[22]
Glover, F. Tabu Search: A Tutorial. Interfaces 1990, 20, 74–94, doi:10.1287/inte.20.4.74.
[23]
Yang, X.; Deb, S. Cuckoo Search via Levy Flights. In Proceedings of the World Congress on Nature and Biologically Inspired Computing, India, 2009; IEEE: New York, NY, USA, 2009; pp. 210–214.
Bulatovic, R.R.; Dordevic, S.R.; Dordevic, V.S. Cuckoo Search Algorithm: A Metaheuristic Approach to Solving the Problem of Optimum Synthesis of a Six–Bar Double Dwell Linkage. Mech. Mach. Theory 2013, 61, 1–13, doi:10.1016/j.mechmachtheory.2012.10.010.
Huckle, T. Collection of Software Bugs. Institut für Informatik TU München: Munich, Germany. Available online: http://www5.in.tum.de/~huckle/bugse.html (accessed on 7 December 2012).
[28]
Jet Propulsion Laboratory. Mars Climate Orbiter. Jet Propulsion Laboratory: Pasadena, CA. Available online: http://mars.jpl.nasa.gov/msp98/orbiter/. (accessed on 28 May 2013).
[29]
Dershowitz, N. Software Horror Stories. Tel Aviv University School of Computer Science: Tel Aviv, Israel. Available online: http://www.cs.tau.ac.il/~nachumd/verify/horror.html (accessed on 7 December 2012).
[30]
Leveson, N. Medical Devices: The Therac–25. Massachusets Institute of Technology: Cambridge, MA. Available online: http://sunnyday.mit.edu/papers/therac.pdf (accessed on 7 December 2012).
[31]
Jorgensen, P. Software Testing: A Craftsman’s Approach; Auerbach Publications: Boca Raton, FL, USA, 2008; pp. 7–8.
[32]
Huber, J.; Straub, J. Human Proximity Operations System Test Case Validation. In Proceedings of the 2013 IEEE Aerospace Conference, Big Sky, MT, USA, 2–9 March 2013; IEEE: New York, NY, 2013. (in press).
