Chapter 02

Adaptive Robot Perception

Back to The Tutorial Back to overview

Welcome to the second chapter of our hands-on course!

This time, you'll dive into the basics of perception. You will learn to utilize sensors to detect objects within our task-adaptive robot perception framework, specifically focusing on a milk carton.

Goal: By the end of the session, you will successfully detect a milk carton on real-world sensor data.

Prerequisites

  • Finished Chapter 1 and have a basic understanding of computer vision.

Theoretical Background

Mobile manipulation robots encounter different perception requirements during the execution of a task. Detecting an object, estimating its pose, tracking a human, recognizing an activity, or identifying free space require different perception methods and processing schemes. A general robot perception system therefore cannot rely on a single fixed vision pipeline. It must select and combine suitable perception methods according to the current task, execute dependent methods in the required order, and support both one-shot and continuous perception. Task-specific perception also avoids executing computationally expensive methods when their results are not required for the current robot action.

RoboKudo is an open-source perception framework for mobile manipulation systems that generates and executes task-specific perception processes by combining multiple vision methods. The framework is based on Unstructured Information Management on Behavior Trees (UIMoBT), which combines the data-processing principles of Unstructured Information Management (UIM) with the execution semantics of Behavior Trees (BTs). RoboKudo implements this concept using Perception Pipeline Trees (PPTs), an extension of Behavior Trees for representing robot perception processes.

A PPT specifies which perception methods are executed and how their execution is coordinated. Individual perception methods are represented by Annotators. An Annotator can implement, for example, object detection, pose estimation, point-cloud processing, attribute classification, or human activity recognition. During processing, Annotators produce Annotations that add information to the current sensor observation. Annotations can represent object hypotheses, class labels, poses, colors, shapes, human keypoints, or other information inferred from the sensor data.

Annotators exchange information through a shared data structure called the Common Analysis Structure (CAS). The CAS contains the sensor data together with the Annotations generated during perception. Annotators retrieve the information required for their computation from the CAS and store newly inferred information back into it. They therefore do not require direct connections to other Annotators. A subtree with an associated CAS is called a Pipeline. Multiple Pipelines can exist within one PPT, which allows different sensor streams or perception subprocesses to use separate CAS instances while remaining part of the same overall perception process.
The execution of a PPT follows Behavior Tree semantics. Sequence nodes execute dependent processing steps in order. Parallel nodes organize computations that can be performed independently. Fallback nodes represent alternative processing strategies and allow another branch to be executed when a preceding branch fails. Nodes return execution states such as Success, Failure, or Running. These return values determine how execution continues through the tree. Reusable subtrees and looping control structures allow the same process model to represent single-shot perception, continuous tracking, parallel processing, and reactive perception processes.

RoboKudo is designed as part of the perception-action loop of a robot system. A high-level robot control program requests information through a query-answering interface, for example, "find a milk box in the fridge." The perception task is translated into a task-specific PPT containing the Annotators and control-flow structures required to obtain the requested information. The active perception process can therefore change as the robot progresses through its task. A robot may first require object detection and pose estimation for grasping, then continuous human tracking during interaction, and later free-space estimation for object placement. The same running RoboKudo instance can execute these different perception processes according to the current task.

The required structure of a PPT depends on the inputs and outputs of its Annotators. If one Annotator requires information produced by another, the corresponding nodes can be executed sequentially. Independent computations can be placed in parallel branches. Alternative methods that produce equivalent information can be represented using fallback structures. This makes the dependencies between perception methods explicit in the process representation and allows perception components to be reused in different task-specific PPTs.

RoboKudo connects a perception request from the robot control system with the perception methods required to answer it. Instead of defining one fixed perception pipeline for the robot, the framework organizes available perception components into task-specific processes that can be selected and modified during runtime. Ultimately, the information gained from the analysis of the sensor data (e.g. a detected milk carton), will be asserted into the Semantic Digital Twin belief state of the robot. Therefore, it provides the necessary world information to enable the robot to act.

Interactive Actions and/or Examples

For the Hands-On Exercises of this chapter, please visit the following link:

Interactive Tutorial Robot Perception

Summary

By the end of the session, you’ll have a clearer understanding of the challenges associated with perception in robotics and how task-adaptivity can be realized in a cognitive architecture.

Further Reading/Exercises

  • RoboKudo is built upon behavior trees. If you want to read more about the general concept of behavior trees, there is an excellent, comprehensive book on arxiv which can be seen here.
  • For sensor data processing, Robokudo makes heavy use of OpenCV and Open3D.
  • Challenge: Experiment with different object detection models to improve the accuracy of the object detection process.

RoboKudo Overview:

 

Authors and Contact Details

 

Back to top