Action recipe editor

The manual creation of an action recipe can be tedious task that is prone to errors like incorrect transition specifications or wrong action arguments. The graphical editor for action recipes is intended to speed up the creation and update of action recipes. It further serves as compact visualization of an action recipe and as supervision interface during task execution.

The following image gives an overview of its interface. In the top row, there are three groups of buttons for loading recipes from RoboEarth, for saving them back, and for starting the execution on the UNR Platform. The bottom left area visualizes the task specification, i.e. the single actions, their respective properties, and the transitions between them (whose type is visualized by different colors — green for ‘OK’, red for ‘ERROR’). The two groups of form elements on the right-hand side describe properties of the recipe as a whole. The base IRI defines the name space of the OWL elements that are part of the recipe; the default value can be kept here. The other forms are described below.

The action recipe editor is part of the knowrob_gui repository that you can checkout from

git clone https://github.com/knowrob/knowrob_gui.git

The knowrob_gui directory needs to be in your ROS_PACKAGE_PATH environment variable. You can then compile the program using rosmake:

rosmake knowrob_gui

Knowrob_gui depends on the knowrob stack and its dependencies. Please have a look at the installation instructions if you haven't installed knowrob yet. You can then start the editor with the following command:

rosrun knowrob_gui KnowRobGUI

There are some important properties of action recipes that should be known to understand the editor and execution system:

• Action recipes are not executable code, but rather a description how to perform the task — similar to a cooking recipe. Like a cook’s brain is needed to read a recipe and call the respective actions, we need an ‘execution engine’ that reads the recipes, calls the actions, checks if they were successful, handles errors etc.
• Each action is a specialization of a generic action class (e.g. GraspBottle is a subclass of GraspingSomething) and inherits the properties of the generic classes (e.g. that a gripper is needed to grasp something).
• In the recipe, the specialized actions are described using properties, e.g. the objectActedOn or the toLocation. These property descriptions are normally abstract and qualitative, and need to be translated into concrete action parameters at execution time (e.g. a spec like ‘Point2D and inReachOf bottle1’ into a concrete position in the environment to which the robot can drive).
• There are two kinds of hierarchy involved: In a composition hierarchy, actions can be decomposed into more fine-grained descriptions, like a GraspBottle-action composed of reaching, grasping, lifting etc. In a specialization hierarchy, action classes are derived from more general classes (GraspingABottle from GraspingSomething from ActionOnObject). In the latter, action properties are inherited from higher levels of the hierarchy.
• Task structure: Apart from action sequences, the language supports more complex structures like partially-ordered plans or state machine specifications. The latter are used for the definition of recipes in the UNR-PF context.
• A recipe therefore consists of
  • a set of actions
  • each annotated with properties, e.g. the spokenText of a SpeechSynthesis action
  • transitions between these actions that specify in which case (i.e. which outcome of the action) which subsequent action shall be chosen

• drag with the right mouse button pressed: shift the canvas around
• drag with the left mouse button pressed (after click on action): move the action around
• click on an action or transition: activate that element
• double-click on an action or transition: open the properties window for that element
• DELETE key when action or transition is active: delete that element; when deleting an action, incoming and outgoing transitions are also deleted

Before you can start to add actions, you need to create the recipe itself. The main information that is needed here are a unique identifier, usually written in CapitalizedCamelCaseFormat, and an arbitrary natural-language label. The latter is used to search for recipes in the RoboEarth DB, so it should describe briefly which task the recipe is supposed to accomplish. After entering this information, click on "create’. A box with the natural-language label will appear in the top left corner, indicating the recipe you are currently editing.

Now you can start to add actions to the recipe. A click on the “Add new action to recipe” button on the right opens the following editor window. You can use TAB/SHIFT-TAB to jump forward/backward through the form fields.

The following information is required:<br />

• The identifier is the name of the action class you are about to create. It should be quasi-unique and describe what the action is supposed to do. It will be extended with the recipe’s base IRI as namespace.<br />
• The class is an existing action class in the ontology from which the new action is derived. The easiest way to choose one is to click on “select” or to hit “ENTER” while being in the form field. A class selector wizard will open (see next section).<br />
• You can add an arbitrary number of action parameters by first selecting the property from the list, then entering its value and hitting ENTER (or clicking on ‘ADD’). The property-value pair will be added to the list of action properties.<br />
• If you would like to use the output of an action as input for another action, you can refer to that output by specifying the value of an input property (e.g. spokenText) in the form _priorActionoutputProperty, e.g. _answer-queryresultText. This specification will be translated to a search among previous actions in the current task execution that were of type ‘answer-query’. If any are found, the value of their property ‘resultText’ will be read and used as input for the current action.<br />
• To remove an action parameter, select it in the list at the bottom and click on “remove”.<br />
• When you are finished, click on “add to task” to close the window. The new action appears on the canvas and can be shifted around.

The class selector wizard contains the hierarchy of action classes defined in the KnowRob ontology. You can navigate through the classes with the mouse (shift the view around with the right mouse button pressed if the window becomes too small). If you know approximately how the desired class is called, it is more convenient to search for it using the text field. The search is full-text in both the class IRI (its ‘official name’) and its natural-language label. If you enter text, hit TAB to search and ENTER to use the currently selected class in the action editor.

Action transitions are visualized as arrows between the action blocks. Each action has two ports, an incoming port at the top and an outgoing port at the bottom. Each port can accommodate multiple connections, though each outgoing transition needs to be of different type. The transition types are visualized by different colors:<br />

• OK – green
• CONDITION_TRUE – light blue
• CONDITION_FALSE – dark blue
• ERROR – dark red
• OUT_OF_RESOURCES – light red
• ABORT – purple
• TIMEOUT – orange
• other – gray

To add a connection between two actions, left-click on the outgoing port of the first action. A gray arrow will appear that leads from that port to your mouse pointer. By clicking anywhere on the target action, you create a connection between them. By default, the connection has the type ‘OK’. To change it, double-click on the arrow and select the desired transition type from the transition properties window below.

In order to execute this action graph, the start- and end-actions need to be set so that the system knows where the task starts and when it is finished. The respective form fields are in the upper right block. To select an action, click on the “select” button and then click on the action; it will automatically be entered into the form field.

The editor GUI is integrated with RoboEarth: Recipes can be loaded from the RoboEarth database by clicking on “Load recipe from RoboEarth”. In the search window pictured below, enter the natural-language label of the action, click on “search” and select one of the results from the list. After editing, recipes can be uploaded to RoboEarth using the “Save recipe to RoboEarth”, which works both for updated and new recipe definitions.

• The RoboEarth language: Representing and Exchanging Knowledge about Actions, Objects, and Environments (Moritz Tenorth, Alexander Clifford Perzylo, Reinhard Lafrenz, Michael Beetz), In IEEE International Conference on Robotics and Automation (ICRA), 2012.
• Exchange of Action-related Information among Autonomous Robots (Moritz Tenorth, Michael Beetz), In 12th International Conference on Intelligent Autonomous Systems, 2012.
• RoboEarth Action Recipe Execution (Daniel di Marco, Moritz Tenorth, Kai Häussermann, Oliver Zweigle, Paul Levi), In 12th International Conference on Intelligent Autonomous Systems, 2012.