OpenStreetMap

1. The most simple way is to count just the ways with the highway tag that are at version 1

But …

This fails when we approach the fact that when you split a way, the longer segment will remain with the WAY id, and the smaller way will get a new ID, meaning that it will be version 1. Alt text

2. We can use the nodes of the ways

Using the nodes as a identifier, because even if the way is new, all of the nodes are still from the old way, meaning that the timestamp will be different, the user that created the node will be a different user then the user of the way.

3. But how can we find out this in OverPass ?

Until now, i have this overpass turbo (Go To Halifax, Canada) script that tries to do some filtering, meaning that it will give just the ways that are version 1, and will filter ways that have more then 3 tags. (Because when we add roads, we add a maximum of 3 tags, name, highway type and oneway) If i split a way that have 6 tags, even if it will be version 1, it will be excluded because it have more then 3 tags.

But there should be a better method. And here we still get some ways that are splitted but have less then 3 tags. And in residential areas you get a lot of false positive.

Any ideas how to refine or other technique that i could use to achive the task of getting just the new roads ?

And don’t tell me use the full planet history, because there is no such think as a full planet history file that you can use, compile, run. Nope, it`s a legend.

Discussion

Comment from ImreSamu on 13 June 2017 at 13:40

How can we identify, … just the new roads made by a user ? And don’t tell me use the full planet history,

Why not?

example for a small history: “romania.osh.pbf, a file that contains the full OSM history for this region for processing with e.g. osmium.” ( this method should work with planet history! )

only 2 command line:

wget http://download.geofabrik.de/europe/romania.osh.pbf osmium cat romania.osh.pbf -f opl | grep baditaflorin | grep “ v1 “ | grep “highway=” > baditaflorin_v1_highway.opl

And I have found 11039 data cat baditaflorin_v1_highway.opl | wc -l 11039

tail baditaflorin_v1_highway.opl w424982374 v1 dV c40024550 t2016-06-14T19:15:06Z i398300 ubaditaflorin Thighway=track Nn4243347098,n4243347109,n4243347117,n4243347120,n4243347125,n4243347132,n4243347137,n4243347143,n4243347153,n4243347176,n4243347197,n4243347205,n4243347210,n4243347217,n4243347216,n4243347212,n4243347204,n4243347192,n4243347180 w424982375 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn4243347071,n4243347066,n4243347059,n4243347045,n4243347032,n4243347017,n4243347006,n4243347004,n4243347002,n4243346996,n4243346574,n4243346560,n4243346546,n4243346545,n4243346544,n4243346543,n4243346542,n4243346541,n4243346540,n4243346539,n4243346538,n4243346537,n4243346536,n4243346535,n4243346534,n4243346533,n4243346532,n4243346531,n4243346530,n4243346529,n4243346528 w424982376 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn255357385,n4243347278,n4243347295,n4243347302,n4243347307 w424982377 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn4243347153,n4243347161,n4243347170,n4243347187,n4243347202 w424982378 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn4243347063,n4243347068,n4243347074,n4243347088,n4243347089,n4243347092,n4243347100,n4243347110,n4243347127,n4243347138,n4243347152,n4243347171,n4243347196,n4243347206,n4243347211,n4243347231,n4243347232,n4243347236,n4243347243,n4243347250,n4243347256,n4243347261,n4243347262,n4243347263,n4243347265,n4243347267,n4243347266,n4243347268,n4243347270,n4243347271,n4243347272,n4243347287,n4243347296,n4243347297,n4243347300,n4243347304,n4243347306,n4243347313,n4243347315,n4243347317,n4243347320,n4243347324,n4243347338,n4243347349,n4243347353,n4243347369,n4243347391,n4243347405,n4243347408,n4243347411,n4243347419,n4243346814,n4243346823,n4243346832,n4243346840,n4243346846,n4243346844,n4243346837,n4243346821 w424982379 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn4243347393,n4243347401,n4243347396,n4243347388,n4243347387,n4243347386 w424982380 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn4243347087,n4243347086,n4243347069,n4243347044,n4243347041,n4243347037,n4243347030 w424982381 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn4243347019,n4243347003,n4243346994,n4243346586,n4243346573,n4243346571,n4243346568,n4243346558 w424982382 v1 dV c40024550 t2016-06-14T19:15:07Z i398300 ubaditaflorin Thighway=track Nn4243347072,n4243347067,n4243347031,n4243347021,n4243347008,n4243346989,n4243346578,n4243346569 w424982866 v1 dV c40024550 t2016-06-14T19:19:56Z i398300 ubaditaflorin Thighway=track Nn255357375,n4243353116,n4243353115,n4243353114,n4243353113,n4243353112,n4243353111,n4243353110,n4243353109,n4243353108,n4243353107,n4243353106,n4243353105,n4243353104,n4243353103,n4243353102,n4243353101,n4243353100

Comment from ImreSamu on 13 June 2017 at 13:41

again:

wget http://download.geofabrik.de/europe/romania.osh.pbf

osmium cat romania.osh.pbf -f opl grep baditaflorin grep “ v1 “ grep “highway=” > baditaflorin_v1_highway.opl

Comment from baditaflorin on 13 June 2017 at 14:48

WIth Osmium i did not tried, i tried with osm-history-renderer so that i can load the data in postgres, but it failed every single time

Comment from ImreSamu on 13 June 2017 at 16:41

for osmium tool ( opl ) see Usage Examples :

Comment from mmd on 13 June 2017 at 17:47

Hello friends of Overpass (btw: no camel casing in the name),

why can’t you check, if your ways with version 1 have been created in same changeset id as one of its nodes? If the way has been created as part of a split, its changeset id should be different from any of its nodes’ changeset id.

That should could probably be done quite easily via changeset() in an if: filter.

Also, I don’t quite get how the osmium filter on “v1” helps to solve the issue looking at the output above. Right now, the version filter used in the Overpass query does exactly the same already.

Comment from Adityo on 14 June 2017 at 07:15

Hi baditaflorin,

Great diary you just made. I have some question related to overpass-turbo, is it possible to find object with specific changeset, such as hotosm-taskingmanager or any else?

Comment from mmd on 14 June 2017 at 07:26

@Adityo: you can only check for a changeset number. It’s not possible to query by changeset metadata, such as some “#hotosm” tags. I don’t think this will change anytime soon. See https://github.com/drolbr/Overpass-API/issues/189

Comment from Adityo on 14 June 2017 at 08:09

@mmd

I see. Thank you for give the link about this issue!

Comment from ImreSamu on 14 June 2017 at 12:29

Hi @mmd,

Also, I don’t quite get how the osmium filter on “v1” helps to solve the issue looking at the output above. Right now, the version filter used in the Overpass query does exactly the same already.

The osmium cat is processing “osh”( history ) file, and my problem with normal overpass “user” is that very hard to query the first v1:”user” of the object.

So for me the “osmium cat -opl” preprocessing is much easy.

I have created a draft query, but maybe this is only a 15% of the final solution. ( I don’t know the original context of the problem )

This method is good enough for me - most of the times, but not perfect for calculating the original length of the new roads made by given user. ( this is the other ~ 85% of the work … )

  • osmium cat filtering OpenStreetMap history data ( osh /canada/nova-scotia.osh.pbf )
  • generated overpass query ( query the current state of the ways )
    • this is the output of previous script: overpass_query.txt ( Copy - Paste to overpass - and run in “ /canada/nova-scotia” )

This is only a draft solution ( Proof of concept ) and can be improved in the “osmium filter” and the “overpass” side.

As I see - on the Overpass Side the generated query can be improved with:

Comment from mmd on 14 June 2017 at 16:07

@ImreSamu :

and my problem with normal overpass “user” is that very hard to query the first v1:”user” of the object.

If the object currently still has object version 1, the query looks as follows:

way({{bbox}})(user:florinbadita_telenav)(if:version()==1); out geom meta;

(pending on the release of version() == 1, which is available for pre-release testing via http://overpass-turbo.eu/s/pLj)

Otherwise, if the object already has versions > 1, it’s indeed difficult to uncover the original creator. I described this issue here, but unfortunately, there’s no solution for it yet. Today, it would require so set the [date:...] to some value in the past and check the version for that particular point in time, which is not really convenient, especially if you look at a larger number of objects.

So, by checking the .osh.pbf file, you get direct access to all version 1 objects. That same information would be in Overpass API (back to September 2012). However, there’s no easy way to query for those object versions (for many different objects) as of today.

Comment from Adityo on 25 July 2017 at 03:16

Hi, it’s me again..

I have another question.

In the first line of your overpass queries, you put this line:

{{data:overpass,server=http://dev.overpass-api.de/api_new_feat/}}

Are those queries important to put on the first line? If yes, does it mean we use new develop api for overpass to run the queries?

I want to try like what you did in the example (Canada), but I always have some error especially if i put this query:

(if:version()==1)

So, i wonder whether i need to put your first line query in order to run fine.

Sorry for my bad english and thank you!

Comment from Adityo on 25 July 2017 at 04:32

Sorry, just figure out comment from mmd that is still available just in development server.

Case closed :D

Comment from mmd on 25 July 2017 at 05:25

@Adityo: why did you create a new thread for this very same topic on GIS SE. Please keep it all at one location.

Comment from Adityo on 25 July 2017 at 06:24

@mmd sorry, i create that post before i read your post above in detail and also my internet connection have some problem before.

I pretty sure i already stop the loading on the page and close my tab but i think it still registered as new post.

So I already deleted the post at GIS SE to make sure it’s not duplicate.

Log in to leave a comment