Quality Engineering in the Development of an Intelligent Agent
- Tim Halverson, Aptima, Inc., Woburn, Massachusetts, United States
- Michelle Caisse, Air Force Research Laboratory, L3Harris, WPAFB, Ohio, United States
- Mary Freiman, Aptima, Inc., Woburn, Massachusetts, United States
- Christopher Myers, Cognitive Science, Models, & Agents, Air Force Research Laboratory, WPAFB, Ohio, United States
AbstractOur laboratory is involved in the development of an intelligent agent that operates a remotely piloted aircraft with two human teammates that communicate using text chat. The task is well-defined, but there are potentially numerous and unpredictable inputs during varied 40 minute missions. To assure reliability of agent behavior, we must run a large number of missions and analyze the behavior of the agent at milliseconds resolution. To support this requirement, we have developed 1) a scripting language and control system that drives a mission with simulated teammates and environmental events, 2) scripted missions using actual chat input from a previous study, 3) output files for each mission that trace agent actions, situation state, and program events, and 4) scripts that analyze the output files based on performance heuristics and differences from known-good output. This framework allows us to verify complex agent behavior as development progresses.