Reasoning about actions

The knowrob_tutorial package contains an example task description for making pancakes. You can start the system for this tutorial using the following commands:

 roscd knowrob_basics_tutorial
 rosrun rosprolog rosprolog knowrob_basics_tutorial
 ?- owl_parse('owl/pancake-making.owl', false, false, true).

The PlanVis visualization can be used to visualize task specifications including action properties and subactions. PlanVis is part of the mod_vis package.

Start the visualization and load a plan (here: ServeADrink) and all it subactions:

 ?- planvis_create(_).
 ?- planvis_load(pancake:'MakingPancakes', _).

You can highlight an action with the following command. This action must be a child action of the currently loaded action.

 ?- planvis_highlight(pancake:'MixFlourAndMilk', _).

Clearing all highlights is done by

 ?- planvis_clear_highlight(_).

The visualization shows the actions in the task in the correct order although they are unordered in the OWL file (e.g. FlippingAPancake is the second action in the OWL file). Before visualization, the sub-actions have automatically been sorted based on the specified ordering constraints.

The class_properties(S,P,O) predicate can be used to read action properties specified on the class level, such as the subActions:

 ?- class_properties(pancake:'MakingPancakes', knowrob:subAction, Sub).
 Sub = 'http://ias.cs.tum.edu/kb/pancake-making.owl#CrackingAnEgg' ;
 Sub = 'http://ias.cs.tum.edu/kb/pancake-making.owl#FlippingAPancake' ;
 Sub = 'http://ias.cs.tum.edu/kb/pancake-making.owl#MixEggAndDough' ;
 Sub = 'http://ias.cs.tum.edu/kb/pancake-making.owl#MixFlourAndMilk' ;
 Sub = 'http://ias.cs.tum.edu/kb/pancake-making.owl#PourDoughOntoPancakeMaker'

The plan_subevents(Super, Sub) predicate reads the sub-actions of a plan and applies topological sorting based on the partial ordering constraints between the sub-actions to generate a sequence of actions that complies with these constraints:

 ?- plan_subevents(pancake:'MakingPancakes', Sub).
 Sub = ['http://ias.cs.tum.edu/kb/pancake-making.owl#CrackingAnEgg',
        'http://ias.cs.tum.edu/kb/pancake-making.owl#MixFlourAndMilk',
        'http://ias.cs.tum.edu/kb/pancake-making.owl#MixEggAndDough',
        'http://ias.cs.tum.edu/kb/pancake-making.owl#PourDoughOntoPancakeMaker',
        'http://ias.cs.tum.edu/kb/pancake-making.owl#FlippingAPancake'].

There is also a recursive version of the plan_subevents predicate that also returns sub-actions of sub-actions:

 ?- plan_subevents_recursive(pancake:'MakingPancakes', Sub).

Using the class_properties predicate, one can query for properties of an action such as the objectActedOn or the toLocation:

 ?- class_properties(pancake:'PourDoughOntoPancakeMaker', P, O).
 P = 'http://ias.cs.tum.edu/kb/knowrob.owl#objectActedOn',
 O = 'http://ias.cs.tum.edu/kb/knowrob.owl#LiquidTangibleThing' ;
 P = 'http://ias.cs.tum.edu/kb/knowrob.owl#objectActedOn',
 O = 'http://ias.cs.tum.edu/kb/knowrob.owl#Dough' ;
 P = 'http://ias.cs.tum.edu/kb/knowrob.owl#toLocation',
 O = 'http://ias.cs.tum.edu/kb/knowrob.owl#PancakeMaker'

By giving the desired property as argument, one can select only specific results, e.g. only the objectActedOn:

 ?- class_properties(pancake:'MixFlourAndMilk', knowrob:objectActedOn, O).
 O = 'http://ias.cs.tum.edu/kb/knowrob.owl#CowsMilk-Product' ;
 O = 'http://ias.cs.tum.edu/kb/knowrob.owl#WheatFlour' ;

The object_change library and the action_effects and process_effects modules it includes provide methods for computing the effects of actions and processes, for projecting the world state after execution of an action, and for planning actions to achieve a certain effect (possibly by intentionally starting a process as indirect action effect).

 ?- use_module(library(object_change)).

We'll use the pancake making task as example. The file pancake-making.owl contains a simple spatial configuration of objects needed for the task in addition to the MakingPancakes task specification. Initially, the egg instance egg1 is computed to be on table1 (see here for an explanation how this is computed):

 ?- rdf_triple(knowrob:'on-Physical', A, pancake:table1).
 A = 'http://ias.cs.tum.edu/kb/pancake-making.owl#egg1' .

We can now perform the projection by querying for the postActors relation of an action. Like the 'on-Physical' relation in the previous step, this relation is computed on demand using computables. There are abstract projection rules for most action classes, defined in the action_effects module. These rules read the inputs of the action, compute the predicted outputs, and assert the relations between the action instance and the related objects and locations. The results is an object transformation graph such as the one shown below in the next section.

The comments after the following predicate calls describe what is happening in the background, such as state changes, object transformations (indicated by the arrow '→') or location changes.

 ?- rdf_triple(knowrob:postActors, pancake:turnon1, Post).
 pancakemaker1 switched on 
 
 ?- rdf_triple(knowrob:postActors, pancake:cracking1, Post).
 egg1 -> EggShell0
 egg1 -> EggYolk-Food1
 
 ?- rdf_triple(knowrob:postActors, pancake:mixing1, Post).
 EggYolk-Food1 added to -> Dough2
 milk1 added to -> Dough2
 pancakemix1 added to -> Dough2
 
 ?- rdf_triple(knowrob:postActors, pancake:pour1, Post).
 Dough2 on top of pancakemaker1
 Dough2 -> Baked4
 
 ?- rdf_triple(knowrob:postActors, pancake:put1, Post).
 Baked4 on top of plate1

After this projection sequence, the world state has changed and the ingredients have been transformed into the baked pancake. In particular, the egg has been destroyed by the cracking action and is therefore not on the table any more:

 ?- rdf_triple(knowrob:'on-Physical', A, pancake:table1).
 false.

The object transformation graph visualized below links actions to their inputs and outputs using special properties that describe in detail what happens to the objects over the course of the action. For instance, the inputsCommitted property indicates that an input is merged into the output, while outputsCreated means that a new output object is created by the action (which had not existed beforehand).

The graph can therefore be used to reason about which objects have been transformed into which other ones by looking only at the inputs and outputs, neglecting for now the action in between. In the one direction, one can query for what has been transformed into Baked4, the pancake instance, to list all initial and intermediate ingredients:

 ?- rdf_triple(object_change:transformedInto, A, knowrob:'Baked4').
 A = 'http://ias.cs.tum.edu/kb/knowrob.owl#Dough2' ;
 A = 'http://ias.cs.tum.edu/kb/knowrob.owl#EggYolk-Food1' ;
 A = 'http://ias.cs.tum.edu/kb/pancake-making.owl#egg1' ;
 A = 'http://ias.cs.tum.edu/kb/pancake-making.owl#milk1' ;
 A = 'http://ias.cs.tum.edu/kb/pancake-making.owl#pancakemix1'

On the other hand, one can track into which other objects something has been transformed, for example for the egg:

 ?- rdf_triple(object_change:transformedInto, pancake:egg1, Res).
 Res = 'http://ias.cs.tum.edu/kb/knowrob.owl#Baked4' ;
 Res = 'http://ias.cs.tum.edu/kb/knowrob.owl#Dough2' ;
 Res = 'http://ias.cs.tum.edu/kb/knowrob.owl#EggShell0' ;
 Res = 'http://ias.cs.tum.edu/kb/knowrob.owl#EggYolk-Food1'

During the projection, we manually added the action turnon1 in the beginning. It is not part of the original recipe, and normally, humans omit such instructions since they consider them to be obvious. However, robots do not have the common-sense knowledge needed to infer that such an action is needed to successfully perform the task.

Using the action and process representation presented earlier, the robot can infer that something is missing and plan the appropriate actions. Due to the compatible representation of the effects of both actions and processes, the system can thereby not only plan actions that directly lead to the desired effects, but also actions that trigger a process which then achieves the desired result.

Since the knowledge base already contains the projected plan, we restart from a clean state for this example:

 rosrun rosprolog rosprolog knowrob_tutorial
 ?- use_module(library(object_change)).
 ?- owl_parse('owl/pancake-making.owl', false, false, true).
 ?- object_change:project_and_debug(pancake:'MakingPancakes', OrigActionSeq, DebuggedActionSeq).
 
 egg1 -> EggShell1
 egg1 -> EggYolk-Food2
 milk1 added to -> Dough4
 flour1 added to -> Dough4
 Dough4 added to -> Dough6
 EggYolk-Food2 added to -> Dough6
 Dough4 on top of pancakemaker1
 
 OrigActionSeq = ['CrackingAnEgg',
                  'MixFlourAndMilk', 
                  'MixEggAndDough',
                  'PourDoughOntoPancakeMaker', 
                  'FlippingAPancake'],
 
 DebuggedActionSeq = ['CrackingAnEgg',
                      'MixFlourAndMilk', 
                      'MixEggAndDough', 
                      'PourDoughOntoPancakeMaker', 
                      'TurningOnHeatingDevice', 
                      'BakingFood',
                      'FlippingAPancake'].

The procedure temporarily instantiates the action classes, projects their effects, checks for problems, and if inconsistencies are detected (e.g. a required input is computed not to be available at the respective point in time), it starts planning backwards to make that input available. As a result, it returns the original action sequence and the debugged one, additionally containing the TurningOnHeatingDevice action and the BakingFood process triggered by the contact between the dough and the heat source.

Actions can have prerequisites in terms of components or capabilities a robot needs to have in order to execute them. The Semantic Robot Description Language allows to describe robot components, robot capabilities, and dependencies of actions.

The following are just some examples, a full overview of SRDL can be found in the tutorial.

 ?- register_ros_package(mod_srdl).
 ?- srdl2:required_cap_for_action(pancake:'MakingPancakes', Cap).
 Cap = 'http://ias.cs.tum.edu/kb/srdl2-cap.owl#PickingUpAnObjectCapability' ;
 Cap = 'http://ias.cs.tum.edu/kb/srdl2-cap.owl#ArmMotionCapability' ;
 Cap = 'http://ias.cs.tum.edu/kb/srdl2-cap.owl#PickingUpAnObjectCapability' ;
 Cap = 'http://ias.cs.tum.edu/kb/srdl2-cap.owl#ArmMotionCapability'

After loading a robot definition, the action requirements can be matched against the available capabilities to determine which ones are missing:

 ?- owl_parse('path/to/mod_srdl/owl/PR2.owl', false, false, true).
 ?- missing_cap_for_action(pancake:'MakingPancakes', 'http://ias.cs.tum.edu/kb/PR2.owl#PR2Robot1', M).
 M = 'http://ias.cs.tum.edu/kb/srdl2-cap.owl#PickingUpAnObjectCapability'