Modeling tasks and actions

This page describes how actions and tasks are modeled in KnowRob. There are two kinds of descriptions: general task descriptions and models of concrete, e.g. observed, actions.

In KnowRob, Knowledge is modeled in Description Logic (DL) using the OWL syntax. A special property of DL is the distinction between the TBOX (terminological box) and theABOX (assertional box). The former describes classes, their relations and properties, while the latter contains instances of these classes.

In case of actions, these instances may be

• actually observed or performed actions (something that happened at a certain time)
• planned actions (something one intends to do)
• inferred actions (something one imagines that happens)
• asserted actions (some action someone told me has happened)

In any case, one talks about an action that could be asserted a (past or future) time and actor.

In contrast, the tbox describes general information about classes of actions. These can be quite general classes like PuttingSomethingSomewhere, or very specific ones like PuttingDinnerPlateInCenterOfPlacemat. However, all these class specifications describe types of actions that, when they are actually executed, get instantiated to the corresponding actions in the ABOX.

KnowRob provides a taxonomic structure describing several subclasses of Action:

• Action → PurposefulAction
  • Perceiving (similar to SensoryEvent, but in the action context)
  • VoluntaryBodyMovement, e.g. Reaching or ReleasingGrasp (Movements that are not directly manipulating an object)
  • ActionOnObject (any kind of object interaction)
    • ControllingSomething: tap, electrical device, …
    • HoldingAnObject (different grasps, no movement involved)
    • Movement-Translation: picking up, putting down, or moving objects, walking around, …
    • RemovingSomething: cleaning activities
    • OpeningSomething: bottle, cupboard, drawer, …
    • ClosingSomething: bottle, cupboard, drawer, …

There are many more and more detailed action classes available, and the taxonomy can easily be extended. These action classes form the vocabulary for describing the different kinds of actions the system knows.

Each of these actions can be described by its properties: For instance, one could state that every action of type Movement-TranslationEvent is a subclass of Action with the properties fromLocation, and toLocation set. This description can be used to sort any instance of an action that has these properties into the class Movement-TranslationEvent.

As an example, let's have a look at the description of the class PuttingSomethingSomewhere that corresponds to transporting an object from one position to another. Obviously, this kind of action involves to pick up the object, move to the goal position, and put the object down again. These sub-events are modeled in the following piece of code:

  <owl:Class rdf:about="#PuttingSomethingSomewhere">

      <rdfs:subClassOf rdf:resource="#Movement-TranslationEvent"/>
      <rdfs:subClassOf rdf:resource="#TransportationEvent"/>

      <rdfs:subClassOf>
          <owl:Restriction>
              <owl:onProperty rdf:resource="#subEvents"/>
              <owl:someValuesFrom rdf:resource="#PickingUpAnObject"/>
          </owl:Restriction>
      </rdfs:subClassOf>
      <rdfs:subClassOf>
          <owl:Restriction>
              <owl:onProperty rdf:resource="#subEvents"/>
              <owl:someValuesFrom rdf:resource="#CarryingWhileLocomoting"/>
          </owl:Restriction>
      </rdfs:subClassOf>

      <rdfs:subClassOf>
          <owl:Restriction>
              <owl:onProperty rdf:resource="#subEvents"/>
              <owl:someValuesFrom rdf:resource="#PuttingDownAnObject"/>
          </owl:Restriction>
      </rdfs:subClassOf>

      <rdfs:subClassOf>
          <owl:Restriction>
              <owl:onProperty rdf:resource="#orderingConstraints"/>
              <owl:hasValue rdf:resource="#SubEventOrderingPuttingSomethingSomewhere1"/>
          </owl:Restriction>
      </rdfs:subClassOf>
      <rdfs:subClassOf>
          <owl:Restriction>
              <owl:onProperty rdf:resource="#orderingConstraints"/>
              <owl:hasValue rdf:resource="#SubEventOrderingPuttingSomethingSomewhere2"/>
          </owl:Restriction>
      </rdfs:subClassOf>

  </owl:Class>
  

Note that the ordering of the three subEvent restrictions in the OWL file does not have any meaning. In order to describe their allowed orderings, we need to add some constraints. These constraints are described as follows:

 
  <PartialOrdering-Strict rdf:about="#SubEventOrderingPuttingSomethingSomewhere1">
      <occursBeforeInOrdering rdf:resource="#PickingUpAnObject"/>
      <occursAfterInOrdering rdf:resource="#CarryingWhileLocomoting"/>
  </PartialOrdering-Strict>
 
  <PartialOrdering-Strict rdf:about="#SubEventOrderingPuttingSomethingSomewhere2">
      <occursBeforeInOrdering rdf:resource="#CarryingWhileLocomoting"/>
      <occursAfterInOrdering rdf:resource="#PuttingDownAnObject"/>
  </PartialOrdering-Strict>

Unfortunately, the constraints cannot be checked automatically by a description logic reasoner, but applications need to verify them themselves.

Complex robot tasks can be decomposed into primitive actions and movements. If the sub-actions for lower-level actions are already modeled, tasks can be described conveniently on a rather abstract level, like the already mentioned PuttingSomethingSomewhere actions. The following code is an excerpt of a plan for setting a table.

The upper region describes the task SetATable as a subclass of Action with a set of sub-actions.

  <owl:Class rdf:about="#SetATable">
      <rdfs:subClassOf rdf:resource="&knowrob;Action"/>
      <rdfs:label rdf:datatype="&xsd;string">set a table</rdfs:label>
      <owl:equivalentClass>
          <owl:Class>
              <owl:intersectionOf rdf:parseType="Collection">
                  <owl:Restriction>
                      <owl:onProperty rdf:resource="&knowrob;subEvents"/>
                      <owl:someValuesFrom rdf:resource="#PuttingSomethingSomewhere1"/>
                  </owl:Restriction>
                  <owl:Restriction>
                      <owl:onProperty rdf:resource="&knowrob;subEvents"/>
                      <owl:someValuesFrom rdf:resource="#PuttingSomethingSomewhere2"/>
                  </owl:Restriction>

              [...]

              </owl:intersectionOf>
          </owl:Class>
      </owl:equivalentClass>
  </owl:Class>

  <owl:Class rdf:about="#PuttingSomethingSomewhere1">
      <owl:equivalentClass>
          <owl:Class>
              <owl:intersectionOf rdf:parseType="Collection">
                  <owl:Restriction>
                      <owl:onProperty rdf:resource="&knowrob;objectActedOn"/>
                      <owl:someValuesFrom rdf:resource="&knowrob;PlaceMat"/>
                  </owl:Restriction>
                  <owl:Class rdf:about="&knowrob;PuttingSomethingSomewhere"/>
                  <owl:Restriction>
                      <owl:onProperty rdf:resource="&knowrob;toLocation"/>
                      <owl:someValuesFrom rdf:resource="#Place1"/>
                  </owl:Restriction>
              </owl:intersectionOf>
          </owl:Class>
      </owl:equivalentClass>
  </owl:Class>

  <owl:Class rdf:about="#Place1">
      <owl:equivalentClass>
          <owl:Class>
              <owl:intersectionOf rdf:parseType="Collection">
                  <owl:Restriction>
                      <owl:onProperty rdf:resource="&knowrob;inFrontOf-Generally"/>
                      <owl:someValuesFrom rdf:resource="&knowrob;Chair-PieceOfFurniture"/>
                  </owl:Restriction>
                  <owl:Class rdf:about="&knowrob;Place"/>
              </owl:intersectionOf>
          </owl:Class>
      </owl:equivalentClass>
  </owl:Class>

[...]

The lower part shows how task-specific subclasses describe actions with certain parameters. While setting a table, the robot may be told to put a place mat in front of the chair. This command is translated into a class specification: a subclass of PuttingSomethingSomewhere actions with a PlaceMat as objectActedOn and a toLocation Place1, which by itself is described as some Place which is inFrontOf-Generally of some Chair-PieceOfFurniture.

Though the modeling of such tasks in the TBOX may seem a bit clumsy at first glance, it is necessary to describe the general 'action recipe'. There may not be a lot of actions where a place mat is positioned in front of a chair, but nevertheless, this is a class of actions that comprises a set of action instances.

General task descriptions are useful for storing general knowledge about activities and for planning actions. Whenever the robot is reasoning on actually performed actions, either by itself or by a human, it needs to describe action instances.

These instances can, for example, be generated by an action recognition system that interacts with the knowledge base and populates the set of action instances based on observations of humans. Based on these observations, the system can set parameters like the startTime, the endTime, the objectActedOn, or the bodyPartUsed.

For more information, see tenorth09dataset or beetz10ameva.

So why all this effort?

• Action recognition: If you have observed a sequence of actions, the objects and the locations they are put to, you can automatically (i.e. without writing any code for this, just using a DL reasoner) determine if these actions are a valid instance of a given task specification. The qualitative location descriptions are generated on-the-fly using computables, and if everything matches the description, you will get a positive result.

• Action verification: If the action does not 100% fit, you can check all sub-events and see which one differs.

• Action execution: The abstract task model can be used to generate code for the robot's planning system (see the paper on importing web instructions.