"OpenStreetMap and Pharo" by Thierry Goubier on at ThierryGoubier/5y08m9uu71o8i7a35ijwr766p under #Smalltalk, #Pharo, #OpenStreetMap, #NeoJSon, #Roassal   3 thanks

OpenStreetMap and Pharo

1. OpenStreetMap and Pharo, part 1

We all have this issue of recovering significant map data to be able to know where to put sensors on (and obstacles, and etc...).

In a place like Jakarta, which is already well mapped, a solution is to use OpenStreetMap data (not tiles, the true "shape" data). How to to that? Just try the Overpass query language with the IDE here:

http://overpass-turbo.eu/

The language reference is there:

http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide

Ok. What can be done with overpass turbo? Enter Kampung Melayu as a search term, and select: Kampung Melayu, Daerah Khusus Ibukota Jakarta, Jawa, Indonesia.

It will center the map on Kampung Melayu, Jakarta. Zoom out a bit and you will see the Ciliwung river and a lot of small houses along Gang Pulo 1, Gang Pulo2, Gang Pulo 3 ...

Now, on the left pane, we can start selecting data: Let's try to select Gang Pulo 3: it should be a way with name "Gang Pulo 3". Try with this query:

This is an example Overpass query.
Try it out by pressing the Run button above!
You can find more examples with the Load tool.
-->
<osm-script output="json" timeout="25">
  <!-- gather results -->
  <union>
    <query type="way">
      <has-kv k="name" v="Gang Pulo 3"/>
      <bbox-query {{bbox}}/>
    </query>
  </union>
  <!-- print results -->
  <print mode="body"/>
  <recurse type="down"/>
  <print mode="skeleton" order="quadtile"/>
</osm-script>

Run it: You should see a blue line along Gang Pulo 3, and, in the data tab, a long json file containing a way, named Gang Pulo 3, and many nodes with their position.

Now, how to get the Ciliwung river? We need to understand how OpenStreetMap categorize things, and for that we can use OpenStreetMap.org itself.

Open http://openstreetmap.org, search for Ciliwung, select Ciliwung River. You see on the left that it has an attribute waterway=river (and that it is a way).

Go back to overpass-turbo, change the request so that we look for the waterway river with <has-kv k="waterway" v="river"/>

We see the river drawn in blue... http://overpass-turbo.eu/s/58h

Congratulations!

Part 2 coming tomorrow...

2. OpenStreetMap and Pharo, part 2

It's now time for the part 2 of our OpenStreetMap experience. So we know how to access a street and a river on our chunk of the Jakarta map, and we get the result as some json.

So, we know all the data is there. But how do we get all the different tags and names and stuff? Querying blindly in that doesn't give good results.

So we'll do a different kind of blind search, one more usefull: retrieve everything in one area and note the type of things we find. First, we will zoom a lot in the image, to reduce the number of things visible (but still keep interesting stuff, like the river, a street, and a few buildings). And we will write that query:

  <!-- gather results -->
  <union>
    <!-- query part for: “type=building” -->
    <query type="node">
      <bbox-query {{bbox}}/>
    </query>
    <query type="way">
      <bbox-query {{bbox}}/>
    </query>
    <query type="relation">
      <bbox-query {{bbox}}/>
    </query>
  </union>
  <!-- print results -->
  <print mode="body"/>
  <recurse type="down"/>
  <print mode="skeleton" order="quadtile"/>
</osm-script>

(http://overpass-turbo.eu/s/58q)

Everything is now displayed in blue and we have a lot of data. Let's look into the data and select interesting stuff.

I see many nodes: don't know what they are for. Higher level elements are more interesting: I see a way with tags:

    "waterway": "riverbank"
  } 
Oh, this is the Ciliwung riverbank, then!

Another interesting way, a bit down:

    "access:roof": "yes",
    "addr:full": "Jalan Jatinegara Barat",
    "amenity": "school",
    "building": "yes",
    "building:levels": "2",
    "building:roof": "tile",
    "building:structure": "reinforced_masonry",
    "building:walls": "brick",
    "capacity:persons": "Lebih dari 500",
    "name": "SD, SMP, Santa Maria Fatima",
    "type:id": "SD SMP"
  }

Ok: a two level buildings, a school.

Another one: a way again:

    "admin_level": "8",
    "area": "yes",
    "boundary": "administrative",
    "flood_prone": "yes",
    "is_in:district": "Jakarta Selatan",
    "is_in:hamlet": "BUKIT DURI",
    "is_in:province": "DKI Jakarta",
    "is_in:subdistrict": "TEBET",
    "name": "RW 10"
  }

Cool: this is a RW administrative boundary (how is that RT/RW stuff translation in english? A sub-sub-sub-district?). And look, it even has a tag "flood_prone:yes" !

(and we can probably keep on looking through that data with interest, but the last one gives me something I really want to query for!)

Let's unzoom to see Jakarta as a whole. Choose the wizard and enter the following query:

type:way and admin_level=8 and area=yes and boundary=administrative and
flood_prone=yes

And execute the query. Yes, we have all the flood prones RW of Jakarta !

http://overpass-turbo.eu/s/58r

This is really cool. The people filling the data for Jakarta on OpenStreetMap have really done a great job. I didn't expect to be able to do such a powerfull query.

This is all for now; we know have fairly powerfull query abilities and json data out of OpenStreeMap. Stay tuned for part 3

3. OpenStreetMap and Pharo, Part 3

In the part 3 of this series, we will try to see if we can, via Pharo, access directly the OpenStreetMap data we want. First, with a bit of work from http://openstreetmap.org, we will select an area in Kampung Melayu.

I select more or less the same area as before (the Ciliwung river loop West of the Kampung Melayu name), get the center coordinates and select an area around that with the following coordinates

(They can be entered in OpenStreetMap to see the points)

Now, looking at the openstreetmap wiki, I saw that I can tap the following http address for queries:

http://overpass-api.de/api/interpreter

Back on overpass-turbo.eu, I select that same area (more or less) and I use the wizard to generate a query for the riverbanks (type:way and waterway=riverbank). Then I export, choose request and ask overpassQL, and I get the following (coordinates may not be exactly the same, I rewrote with the ones I chose above):

[out:json]
[timeout:25]
;
(
  way
    ["waterway"="riverbank"]
    (-6.22170,106.85923,-6.21970,106.86323);
);
out body;
> ;
out skel qt;

Now I fire up Pharo, version 3 and I will try to query directly OpenStreetMap. Where is the documentation for Zinc already? Oh, that's it: I need a ZnClient.

In Pharo, what I do is then the following:

| response |
response := ZnClient new
    url: 'http://overpass-api.de/api/interpreter';
    queryAt: 'data' put: '[out:json]
[timeout:25]
;
(
  way
    ["waterway"="riverbank"]
    (-6.22170,106.85923,-6.21970,106.86323);
);
out body;
> ;
out skel qt;'
    get;
    response

Select this and inspect, and you will get, after a bit of time (the time needed for the request), a ZnResponse in an inspector. Choosing entity in there will give you... the osm data:

a ZnStringEntity(application/json 9579B {
  "version": 0.6,
  "generator": "Overpass API",
  "osm3s": {
    "timestamp_osm_base": "2014-09-23T20:42:02Z",
    "copyright": "The data included in this document is from
www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [
...

Note: the entity is long and won't be displayed entirely in the inspector.

Ok, now, what do we need: a JSON reader in Pharo. Go to the configuration browser, choose NeoJSON and select install stable. With NeoJSON, we will use the NeoJSONReader as a content reader for the ZnClient and change our request so:

| response |
response := ZnClient new
    url: 'http://overpass-api.de/api/interpreter';
    queryAt: 'data' put: '[out:json]
[timeout:25]
;
(
  way
    ["waterway"="riverbank"]
    (-6.22170,106.85923,-6.21970,106.86323);
);
out body;
> ;
out skel qt;'
    contentReader: [ :entity | NeoJSONReader fromString: entity contents ];
    get.

And inspect the result. Tada! We now have a dictionary with all our elements nicely tagged. Inside elements, we will find our way(s), nodes, and everything...

Next step, displaying... This will be for tomorrow. Good night!

4. OpenStreetMap and Pharo, Part 4

Ok, from part 3, we had the data. Now, it's time to display it, with Roassal: the favorite tool for drawing and displaying data.

To load Roassal, do the following:

Gofer it
   url: 'https://smalltalkhub.com/mc/Pharo/MetaRepoForPharo30/main';
   configurationOf: 'Roassal2';
   loadDevelopment

Ok. Now we're set, we have all the tools we need. But what are the things we do need now? We need to project the coordinates we have received in the OpenStreetMap data, rebuilt the object contained (the way(s)), and draw them.

To project the data, we will use the same projection as OpenStreetMap: the Mercator projection, with two blocks:

latToX := [:l | (l + 180) / 360 ]
lonToY := [:l | 0.5 - (((Float pi / 4) + (l degreesToRadians / 2)) tan
ln / (2.0 * Float pi)) ]

The problem is that this is a projection with coordinates between 0 and 1, and it doesn't work well when scaling in Roassal, so I spare you my experiments and tell you that we need to scale the results:

mP := 256 * (2 ** 15).
lonToX := [ :l | mP * ((l + 180 )/ 360) ].
latToY := [ :l | mP * (0.5 - (((Float pi / 4) + (l degreesToRadians /
2)) tan ln / (2.0 * Float pi))). ].

So we have our formulas. What we want to do now is to project the node coordinates. We will also create a Dictionary storing, for each node id, the projected coordinates.

nodes := (
    ((response at: 'elements') select: [ :e | (e at: 'type') = 'node'])
        collect: [:e |
            (e at: 'id') -> ((lonToX value: (e at: 'lon')) @ (latToY value: (e
at: 'lat')))]
    ) asDictionary.

And we will, for each object of type way, create a RTPolygon object with the node points, and add it to the Roassal view.

v := RTView new.
v @ RTDraggableView.
((response at: 'elements') select: [:e | (e at: 'type') = 'way'])
    do: [ :way | | s e |
    s := RTPolygon new.
     s color: ((((way at: 'tags') at: 'building' ifAbsent: []) = 'yes')
ifTrue: [Color gray] ifFalse: [Color blue]).
    s vertices: ((way at: 'nodes') collect: [:n | nodes at: n]).
    e := s elementOn: (way at: 'tags').
    e @ RTPopup.
    v add: e ].
v open.

Special here: if the way is a building, then the inside color is gray, else the inside is blue (I will show you why). To finish that, we will ask the view to scale and center itself:

v canvas camera focusOnCenterScaled

Ok. Now I'll put it all together, with a slightly more complex query which retrieve the buildings and the river banks in our start area. This is a long script, enjoy!

| response lonToX latToY nodes v mP |
response := ZnClient new
    url: 'http://overpass-api.de/api/interpreter';
    queryAt: 'data' put: '[out:json]
[timeout:25]
;
(
   way["waterway"="riverbank"](-6.22170,106.85923,-6.21970,106.86323);
   way["building"="yes"](-6.22170,106.85923,-6.21970,106.86323);
);
out body;
 >;
out skel qt;';
    contentReader: [ :entity | NeoJSONReader fromString: entity contents ];
    get.
mP := 256 * (2 ** 15).
lonToX := [ :l | mP * ((l + 180 )/ 360) ].
latToY := [ :l | mP * (0.5 - (((Float pi / 4) + (l degreesToRadians /
2)) tan ln / (2.0 * Float pi))). ].
nodes := (((response at: 'elements') select: [ :e | (e at: 'type') =
'node']) collect: [:e | (e at: 'id') -> ((lonToX value: (e at: 'lon')) @
(latToY value: (e at: 'lat')))]) asDictionary.
v := RTView new.
v @ RTDraggableView.
((response at: 'elements') select: [:e | (e at: 'type') = 'way'])
    do: [ :way | | s e |
    s := RTPolygon new.
     s color: ((((way at: 'tags') at: 'building' ifAbsent: []) = 'yes')
ifTrue: [Color gray] ifFalse: [Color blue]).
    s vertices: ((way at: 'nodes') collect: [:n | nodes at: n]).
    e := s elementOn: (way at: 'tags').
    e @ RTPopup.
    v add: e ].
v open.
v canvas camera focusOnCenterScaled

With a screenshot, of course...

blog comments powered by Disqus