What do drones, quidditch, and augmented reality, and have in common?

Now, Virginia Tech researchers are exploring applications for unmanned aircraft systems to assist in safer and more effective inspection of civil infrastructure. The use of drones for this type of assessment has the potential to make the inspection process quicker and more efficient, thus enhancing public safety in the U.S. and around the the world.

Craig Woolsey, professor in the Kevin T. Crofton Department of Aerospace and Ocean Engineering, is the lead investigator of a $1.5 million National Science Foundation grant to research new processes for humans collaborating with unmanned aircraft systems. The project, “First Person View and Augmented Reality for Airborne Embodied Intelligent Cognitive Assistants,” will develop a framework for workers collaborating with drones to inspect hard-to-access spaces, such as bridges, tunnels, dams, towers, pipelines, and electrical transmission lines.

Lack of maintenance and the deterioration of aging structures can lead to catastrophic events, as in the cases of the recent bridge collapses in Genoa, Italy, and Kolkata, India. Required visual inspections are time consuming, costly, impede surrounding traffic, and pose safety hazards to the worker conducting the assessment. Given the sheer number, scale, and complexity of these structures, it’s easy to see how regular, routine inspections can be difficult to manage.

“The potential for this project to enhance the economic and social well-being of people around the world is immeasurable,” noted Woolsey. “First, more efficient and effective infrastructure inspection ensures public safety, but this technology also has the ability to improve both workers’ daily experience and employers’ financial metrics.”

The use of augmented reality and first-person-view in the operator interface for drones also supports job training, since the same technology can be used in a simulated environment, and the skills a worker acquires can be transferred to other applications in addition to infrastructure inspection.

Woolsey and team members conducted preliminary field experiments with aerial robotic inspection of the Coleman Memorial Bridge in Yorktown, Virginia, in late 2017. The Coleman Bridge is the nation’s largest “double swing span” bridge, with a steel deck truss type construction. The bridge requires rigorous visual inspection every other year. Numerous challenges arose, including strong wind gusts that disturbed the drone’s motion and interference from the steel bridge structure with GPS and compass measurements critical for the drone’s autopilot.

Flying the drones using first person view, meaning equipped with goggles, the pilot gains the visual perspective of the aircraft for better flight control, would make the inspection work faster and more manageable, but would also have sacrificed situation awareness by narrowing the worker’s field of view.

In order to optimize the use of unmanned aircraft systems for these types of inspections, the team at Virginia Tech will focus on four distinct research thrusts:

• Joseph Gabbard, associate professor in the Grado Department of Industrial and Systems Engineering, will spearhead the effort to develop new principles of shared situation awareness and integrate augmented reality into the first person view interfaces. His work will look at going beyond human-computer interaction, i.e., where the worker and drone are somewhat “aware” of what the other is doing, shared situation awareness, the current task, and what is next on the list.
• Pratap Tokekar, assistant professor in the Bradley Department of Electrical and Computer Engineering, will develop worker-responsive planning of inspection paths and increase visual perception to remotely inspect bridges for defects and obtain high-quality images from appropriate viewpoints. The  complex geometry of a bridge poses challenges in properly positioning the aircraft, especially in confined, poorly lit regions. Using virtual reality headsets with eye-tracking capabilities, Tokekar will predict the focus of the human’s attention and then adapt the position of the drone to capture high-quality imagery. Gaze can also be used to adapt global plans if unanticipated defects are detected during flight.
• Woolsey will lead the development of worker-tunable control performance for flight control. His research focus on maintaining stable flight in strong gusts and using high-precision control of the aircraft to capture clear, high-quality imagery that require a considerable amount of the drone’s energy. The inspection worker will be able interchange modes of flight between the intensive mission tasks and a more relaxed mode to conserve energy usage.
• Matthew Hebdon, assistant professor in the the Charles E. Via Jr. Department of Civil and Environmental Engineering, will expand defect analytics to improve the effectiveness with which workers assess the health of the infrastructure. Hebdon’s work will seek to develop a learning-based model that will enable the drone to identify degradation and defects. Using augmented reality to supplement the first-person viewpoint would improve the inspector’s performance and provide visual cues to previously flagged issues or system-learned flaws, such as corrosion, cracks, and spalling.

In conjunction with Virginia Tech’s Center for the Enhancement of Engineering Diversity (CEED), the research team plans to introduce drone Quidditch, or “Quidditch,” a team sport where human-piloted multirotor drones compete in a game made popular in J. K. Rowling’s Harry Potter franchise.

While there are no flying magicians in Quidditch, first person view-controlled quadcopters play the roles of bludger, quaffle, and golden snitch. Drones will not be physically knocked out of the air by the bludgers; instead, drones will be freeze-tagged by an opponent’s close approach. Undergraduate research assistants, along with engineering doctoral students, will simulate first-person view gameplay, conduct hardware testing and refinement, and then engage participants in CEED’s diversity-focused summer programs to participate in matches at Virginia Tech’s Drone Park.