Tutorial on "Robots and the Internet"

This tutorial was given at the 2013 LUCIA Winter School on “AI and Robotics” in Örebro, Sweden. This page describes the practical examples complementing the presentation; the slides can be found here.

This page has not been updated to the new, catkin-based version of KnowRob. Some packages used by the instructions below, such as opencyc, are not available in this version.

To run all of the examples, you would need the (restricted) knowrob-dev setup, but most of them also work with the (open) knowrob-all variant. Please follow the linked installation instructions, set up your workspace, and have a look at the Getting Started guide to test your installation.

This demonstration is based on the paper “Understanding and Executing Instructions for Everyday Manipulation Tasks from the World Wide Web”, presented at ICRA 2010.

In a first step, we can perform the import using a GUI that allows to step through the different stages and to inspect the generated data structures. You need to launch the OpenCyc knowledge base in a separate terminal first and wait about one minute until it's completely loaded.

$ roslaunch opencyc opencyc.launch
$ roscd comp_ehow
$ rosrun comp_ehow ehow_gui

Alternatively, we can call the importer from within KnowRob to load the generated task descriptions into the knowledge base and ask queries about them. The following loads the same instructions into KnowRob, sends them to the action visualization, and asks for the sub-actions of the plan as well as for the properties of one of them.

rosrun rosprolog rosprolog flowchart_vis
?- rdf_triple(knowrob:forCommand, A, 'make pancakes').
?- planvis_create(_).
?- planvis_load(ehow:'MakePancakes', _).
?- plan_subevents(ehow:'MakePancakes', Sub).
?- class_properties(ehow:'FluidFlow-Translation3', P, O).

This part requires the restricted knowrob-dev repository, but is listed here for completeness. It is based on the paper “Decomposing CAD Models of Objects of Daily Use and Reasoning about their Functional Parts” that has been presented at IROS 2013. The first query asks for the handle of a spatula and returns the instance identifier of this part. The GUI that is opened visualizes the different processing steps, and the fields on the right-hand side allow to inspect the elements that have been identified.

$ rosrun rosprolog rosprolog flowchart_vis
?- rdf_triple(knowrob:properPhysicalParts, knowrob:spatula1, Part), 
   annotation_handle(Part, knowrob:'Handle'), 
   mesh_annotator_highlight_part(knowrob:spatula1, Part).

The second example is to find and highlight a bottle cap, which is defined as the topmost cone annotation using a Prolog rule. This rule is evaluated on the components of the bottle identified in the CAD model.

?- bottle_cap(knowrob:'mix1', Cap), 
   mesh_annotator_highlight_part(knowrob:mix1, Cap).

Online shops can provide valuable information about objects and their properties. The package comp_germandeli contains an OWL ontology that has been generated from the GermanDeli online shop, selling German products in the US. It can be loaded into KnowRob and expands the 'objects' branch of the ontology with about 7,000 additional classes.

$ rosrun rosprolog rosprolog comp_germandeli

?- owl_subclass_of(A, germandeli:'Bakery').
A = germandeli:'Bakery' ;
A = germandeli:'Apple_Strudel' ;
A = germandeli:'German_Apfelstrudel_10_slices_individually_wrapped' ;
A = germandeli:'Breads' ;
A = germandeli:'Assorted_Breads' ;
A = germandeli:'Pruente_Landbrot_500g' ;
A = germandeli:'Pruente_Sonnenkernbrot_Sunflowerseed_Rye_Bread_500g' ;
A = germandeli:'Ruhrtaler_Vollkornbrot_Rye_Bread_500g' ;
A = germandeli:'Ruhrtaler_Volllkornbrot_Rye_Bread_250g' ;
[...]

?- owl_subclass_of(A, germandeli:'Bakery'), owl_subclass_of(A, knowrob:'HeatSensitive').
A = germandeli:'Pumpkin_Seed_Bread_475g' ;
A = germandeli:'Bahlsen_Comtess_Choco_Chips_Cake_400g' ;
A = germandeli:'Bahlsen_Comtess_Malaga_400g' ;
A = germandeli:'Bahlsen_Comtess_Schoko_Kokos_Genuss_400g' ;
A = germandeli:'Comtess_Zitronen_Kuchen_Lemon_Cake_400g' ;
A = germandeli:'Comtess_a_la_Banana_Split_Cake_400g' ;
A = germandeli:'Baumkuchen_Vollmilch_Tree_Cake_w_Milk_Chocolate' .
[...]

?- class_properties(germandeli:'Vitalis_Fruechte_Muesli_Fruit_Cereal_375g', P, O).
P = germandeli:productCode,
O = literal(type(xsd:string,'4000521642006')) ;
P = germandeli:brand,
O = literal(type(xsd:string,'Dr. Oetker')) ;
P = germandeli:countryOfOrigin,
O = literal(type(xsd:string,'Germany')) ;
P = germandeli:productID,
O = literal(type(xsd:string,oetkermuesli)) ;
P = knowrob:weight,
O = literal(type(xsd:string,'13.3oz (375g)')) ;
P = knowrob:purchasePrice,
O = literal(type(xsd:string,'6.99')) ;

RoboEarth is a web-based database for exchanging information between robots via the Internet. The content of the database can be searched using a web interface as well as using a REST-style API. The following examples show some tools that can interact with RoboEarth; the first two are graphical editors that can up-and download information to and from RoboEarth, the latter is a query interface that directly loads the information into KnowRob.

The Semantic Map Editor can read and write information from/to RoboEarth. To get an impression, search for room '3002' and download the 'lucia.testmap'.

Also the Action Recipe Editor is able to exchange task descriptions via RoboEarth. You can download an example definition by searching for 'environment exploration' and downloading 'lucia.exploration'.

There is a KnowRob client for RoboEarth that can download and upload information directly from the knowledge base. In addition to KnowRob, you need a checkout of the current RoboEarth stack that you can install using

svn co https://ipvs.informatik.uni-stuttgart.de/roboearth/repos/public/tags/latest roboearth
rosmake re_comm

The following are some example queries for downloading an action recipe and an environment map into KnowRob.

$ rosrun rosprolog rosprolog re_comm 
?- visualisation_canvas(_), planvis_create(_).

% Download an action recipe and add it to the visualization
?- re_download_action_recipe('serve a drink', pr2:'PR2Robot1', Recipe), 
   re_generate_cpl_plan(Recipe, CplPlan),
   planvis_load(Recipe,_).
   
% Download an environment map and visualize it
?- re_request_map_for([['kr:roomNumber',3002], ['kr:floorNumber','3'], 
                       ['kr:streetNumber','45'], ['rdfs:label','Karlstrasse']], M),
   owl_individual_of(Map, knowrob:'SemanticEnvironmentMap'), 
   add_object_with_children(Map, _).

The web-based visualization canvas allows to interact with KnowRob via the Web by sending queries and visualizing their results. The setup is a bit complicated since Web servers are not installable purely via ROS, and opening a Prolog console to the world can be a security leak. However, for controlled environments, you can find the installation instructions and example queries here.