CoE Prof Creates Curating Software for Drone Images

Drone technology used to be exciting in and of itself. We’ve moved on. Now, we want to know what it can do for us.

Dr. Antonios Kontsos, associate professor in the Department of Mechanical Engineering and Mechanics (MEM), has one answer. Through his startup company, NOESIS Analytics, Kontsos and his team at the College of Engineering are developing a business-to-business software tool that allows non-specialists to expertly analyze their drone images of buildings, crops, bridges, storm-damaged resorts, and everything in between.

Want to assess structural wear in urban infrastructure? Find out if the windows installed in the corporate headquarters are energy efficient? Determine whether the back 40 is showing signs of crop disease? A user can plug drone images into the NOESIS software and the adaptive algorithm will store, crunch, and stitch them together so they can be scrutinized at much higher resolutions than are available to the unaided eye.

“Using a smart phone does not necessarily make you a smarter person. The iPhone is a conduit for you to access Uber, or to access technology, or to find out something quickly. It’s about how you put technology into your daily life. That’s what we’re talking,” says Kontsos. “It’s how this adoption of technology enables you to do things that before were unthinkable for you.

“By 2020, there are going to be literally millions of drones flying around. All of this is going to create data. And at this point, there’s no software or freeware or website where you can go and reliably say, help me manage and curate all this data,” he explains. “We want to provide a service for people to fly their drones and then actually make quick use of the data produced.”

Kontsos, who prefers the phrase “unmanned aerial vehicle” (UAV), calls his product “agnostic”: the steps the algorithm uses to sort images are not pegged to specific imagery but can be adapted by a non-technical user to suit changing criteria. He envisions the product having applications far beyond the startup specs, expanding into uses like pothole tracking, snow coverage monitoring, crime and security, parks and tree health, or even WiFi access for disadvantaged neighborhoods.

256 Shades of Gray

The product works by exploiting machine learning techniques that allow the algorithm to search data in an intelligent way—in a machine way—to find things the human eye cannot see. For example, NOESIS can identify 256 shades of gray, differentials that people are unable to detect on their own. These differentials highlight patterns that in turn can indicate defects like cracks or corroded areas that otherwise might require close, in-person inspection to locate – a costly, labor-intensive, time-consuming process that too-often is marred by subjective assessment.

Images are shunted through a five-step process that includes uploading the data, “de-noising” and filtering, extracting information, stitching data together according to user parameters, and then enhancing or highlighting the desired data group. Alterations in shade or color in the images are so greatly enhanced during the process that users can zero in on them to determine if they indicate a need for action.

NOESIS, which means “intelligence” in Greek, is developing three initial applications for the as-yet unnamed product: bridge monitoring, energy inspection of buildings, and crop management. Two of these applications are already completed. The third is still being tested.

Kontsos estimates that a software product will be available for commercial use by the end of 2018. He is in the process of forming a C Corporation for the product.

A Classic Pivot

So how did a professor whose primary research involves material mechanical behavior characterization and modeling end up exploring drone applications? It began when a colleague asked him to think about whether the computational simulation techniques used to analyze the built environment could be used on an analysis of concrete. Kontsos’ lab was already using cameras in similar analyses. But the question prompted him to consider other ways to use them and other perspectives -- namely, aerial perspectives. And so he thought, UAVs.

“For me, it was all about imaging,” says Kontsos. “I wanted to know if the drone could be a means for me to get more images at locations where I would not otherwise have easy access. Then, I could use my computational techniques and the post-processing to extract the information on where damage is, or whether there’s corrosion, or whether the building shakes in a different way after storm damage.

“That’s exactly how the idea was born.”

A technical paper on drones and data-collection software authored by Kontsos landed on the May 2018 cover of Materials Evaluation. Drexel colleagues Ivan Bartoli, associate professor with the Department of Civil, Architectural and Environmental Engineering, and Andrew Ellenberg, a former MEM doctoral candidate who has since graduated Drexel, were co-authors on the piece.

The article lays down a case for how “rapid and remote sensing systems implemented on an unmanned aerial vehicle platform could contribute to the next generation of civil infrastructure assessment.” NOESIS Analytics provided the analysis of drone data discussed in the piece.

Drexel Ventures Steps In

In January of 2018, Kontsos and NOESIS received a $50,000 grant from Drexel Ventures to get the project started and help it become an “investable” company, says Shintaro Kaido, director of startup services for Drexel Ventures, the technology commercialization unit of Drexel. Drexel Ventures provides acceleration services as the “one-stop, front door” to the university’s innovative ecosystem.  

“Antonios is committed to building a product based on academic research to improve society,” Kaido says. “Aging infrastructure is a serious problem. NOESIS has the potential to identify structural damages in critical infrastructure, like bridges, more accurately and cost effectively than manual inspection.”

The grant was preceded by a 10-week, in-residence business development boot camp, which Kontsos completed in the spring of 2017.

Then, last spring, two of Kontsos’ BS/MS students tested the NOESIS software on Main Building here on campus as part of a senior design project. A drone collected images from the entire façade. The images were run through NOESIS, which stitched them together to provide an assessment of window condition, the presence of cracks, and the energy efficiency of the building.

Brian Tindall, who graduated in June, was one of the student developers. Tindall says Main provided an “excellent façade” for testing the software and that the program worked well in assembling images. He sees future users benefiting from the ease of organizing data.

“I imagine the average user doing a refined version of our test flights to best utilize our product. This would involve using an existing drone platform, quickly circling the building or structure under inspection, and collecting a series of overlapping photos of each face. This is a simple process that can be automated with most modern drone controllers,” says Tindall.

“After that, the user would simply load the data into our program and choose the data relevant to what they want. The user does not need to know what each algorithm is or how to apply it, just the data that he or she ultimately needs. It would be a simple extension of what they already do, but with significantly less work and time invested.”

Real-Time Objectives

NOESIS already has several adopters, one of which is the City of Philadelphia. The UAV software dovetails with Philadelphia’s Smart Cities Initiative headed up by the Office of Innovation and Technology, which seeks to develop pragmatic, smart technologies that work well in Philadelphia’s specific urban character. Kontsos is also working with the Department of Streets. The NOESIS software is being used in a pilot program to assess several of the city’s bridges.

“What happens in a research and development lab, like here at CoE, satisfies a scientific need. That’s the first thing. You have to answer the scientific questions. But to make them be of general use is not a one-to-one, straight path,” says Kontsos. “We have to respect what the industry’s doing and what society wants, and then translate the development so that it will be more understandable for a user.

“When you use a UAV, the quantity of data is unprecedented. This is not a one-off thing. I can do this routinely. I can record the information and data and do an analysis,” he concludes. “My goal with NOESIS is to show that technology can be created in a way that essentially does good once you have a framework to develop along. It’s a whole new way to think about drones.”