Sterblue Perception

Sterblue Perception contains our 3D toolbox and a collection of libraries that we use all accross our pipeline for trajectory computing and output processing. It exploits video games computer graphics technology in a smart way to offer efficient and relatively easy algorithms. It is mainly based on Three.js, a famous javascript 3D library that itself depends on WebGL.

Why Sterblue Perception#

At Sterblue, we believe that monocular cameras and off-the-shelf drones are sufficient to carry out an inspection. We develop software to perform accurate trajectories and to extract rich information from the pictures without the need of lidar, for instance. Instead, our trajectory relies on a parameterized 3D model that acts as a virtual twin of the actual asset that is being inspected. The use of this 3D model is not limited to trajectories computation, it can be used in many other ways: simple display, photorealistic renderings for AI, anomalies location, etc. The following diagram shows how this 3D model integrates in our inspection pipeline.

perception-diagram

The virtual twin paradigm#

The idea behind using a virtual twin is that 3D computer graphics allows us to enrich the inspection process. First, Sterblue Perception provides a calibration and trajectory engine based on the 3D model to manage the on-field operation. But the usage of this model is not limited to the operation in itself and extends to the outputs post processing. From one inspection picture and its metadata, we are able to obtain several virtual representations of what is seen in the picture, with colors encoding various information, e.g. segmentation of the parts of the structure, depth, surface normals...

perception-semantic-encoding

Those representations enable a lot of features that trully enhance the inspection outputs. The easiest example is the ability to tell what is seen through the camera on each piture: take the rendering that encodes each part of the structure with a given color (bottom middle on the above illustration) and list the different colors that are on the picture. But you can do much more than that, like telling more precisely what zone of each structure can be seen on the picture. Sterblue Perception contains several algorithms in a dedicated library that help post processing the inspection outputs.

How to use Sterblue Perception#

Bellow is a non exhaustive list of use cases that Sterblue Perception can address:

  • Create a 3D structure with given parameters out of the structure library

  • Add a custom 3D structure to the structure library using the structure toolbox

  • Calibrate a 3D structure with real data using the calibration algorithms

  • Add a custom calibration process for an existing or custom 3D structure

  • Create an inspection trajectory from a calibrated 3D structure using the inspection library

  • Add a custom inspection strategy to the inspection library

  • Compute the coverage on an inspection

  • Locate an anomaly in GPS coordinates and relatively to a structure from independant detections on several pictures

  • Represent inspection pictures at the right location in a 3D scene

  • Build a 3D model out of pictures using the photogrammetry pipeline

  • Build an orthopicture aggregating pictures using the photogrammetry pipeline

  • Add a custom algorithm to the algorithms library