OpenStreetMap
I have been testing precision localization for a few years now using RTK, PPP calculations, and RTKLIB.
Since a few weeks, I have my own GNSS base, which allows me to generate gpx traces with an accuracy of a few centimeters, or a few tens of centimeters.
Here is an example of about 15 overlapped gpx traces:
I think I’m not the only one interested in high-precision localization, and the arrival of smartphones with dual-frequency gnss receivers will probably accelerate the movement, which is a great thing for OpenStreetMap. Of course, I send these traces to the Osm servers, and they are available for everyone.
The problem is that theses gpx are mixed with the others, look at this:
At this time, I don’t know how contributors can know that they are of better quality, I don’t know how to “enhance” them.
How could we do that?
-
The GPX format does not store information on point accuracy, but allows you to add private namespaces. However, everyone must use the same one. I don’t think it’s a good idea.
-
On the developers’ discussion list, there was mention of a version 2 of the gpx format, but the last message was almost a year ago.
-
The files generated with Rtklib are .pos files that include all the necessary information. Could the database handle this format?
One last point: - by default, the coordinates of the gpx files must be in the WGS84 system. The problem is that Osm data is not necessarily in this system, but rather in the datum used locally. In France, it is the RGF93, which is derived from the European ETRS89 datum. I think that all European data are in ETRS89 and not in WGS84. There is currently a gap of about 70cm between ETRS89 and WGS84, and this is gradually increasing. So, the GPX traces I send to the Osm servers are not in WGS84, but in ETRS89. I do not respect the GPX standard. I think it should be possible to specify in the gpx files the used datum.
What do you think about this?
Discussion
Comment from smaprs on 23 May 2019 at 21:58
Wow, great achievement, that really deserves a plugin to be implemented!
Comment from ndrw6 on 23 May 2019 at 23:11
Wait, every piece of ddocumentation I have seen so far claims we use WGS84 worldwide and transform from it. Which makes sense, as having multiple coordinate systems in the same database would be asking for trouble.
In practice 60cm error is not an issue - we are rarely more accurate than that, but we should definitely agree on what coordinate system we intend to use.
Comment from StephaneP on 24 May 2019 at 05:01
@nrdw6 Yes It was a surprise for me too, but now I think it’s normal: On the earth, nothing is really static. We are all living on a plate which is slowly moving. So the national geographic agency usually set a local reference system that move with the plate. If they didn’t do that, every point’s coordinates would change continuously. It’s not practicable.
It was not a big deal with standard gnss receivers and low to mid res aerial imagery, as we could not see this offset. But with better imagery, more accurate gnss receiver, we’re starting to detect it.
Usually, country reference system is based on ITRS, just like the WGS84, but locked to a specific epoch. RGF93 in France, is locked to ITRS from 1989. Since then, the eurasian plate is moving to nord-east at about 2 to 2.5 cm per year.
In the united states, the local datum is NAD83, but I don’t know if the osm data are in WGS84 or NAD83.
I think that at some point in the future, local datum will be updated, and we will have to move entire plate in the osm database. :-)
Comment from Andy Allan on 24 May 2019 at 07:32
All OSM data is in WGS84. There are no local datums in OSM.
Comment from StephaneP on 24 May 2019 at 08:14
@Andy The aerial imagery we’re using in France use the RGF93 datum. When I use an official gnss base to compute RTK position, this base transmits its coordinate within the RGF93 datum.
If accuray isn’t better than 1 meter, it doesn’t matter, but we use the RGF93 datum without knowing it.
I’m not sure, but I think it’s the same thing in many other countries because using a “moving” datum would be a nightmare.
I took a random aerial imagery in U.S. and the system is … NAD83, the local datum: https://njgin.state.nj.us/NJ_NJGINExplorer/ShowMetadata.jsp?docId=%7B188471FF-2803-4145-A5AD-605DE86D3B4D%7D
Comment from SimonPoole on 24 May 2019 at 13:07
@StephaneP you simply need to convert to WGS84 / re-project tiles and the like, or your GPS tracks (which btw should be in WGS84 too), before using them for tracing adding them to OSM.
Yes, that -does- have the effect that older OSM data will be slightly off compared to a remeasurement of the same object later, but the only country where that would be a significant concern, over the time spans we are currently looking at, is Australia.
Comment from kucai on 25 May 2019 at 10:30
my suggestion would be to put the corrected imagery into the offset imagery database in JOSM. That’s pretty much the only way people can know that the imagery has been aligned to the best accuracy available at this moment.
Comment from StephaneP on 25 May 2019 at 12:27
We should think twice.
Let’s jump in 2030 or 2040.
-2cm/pixel Aerial imagery is a standard value and new capture campaign occurs every 1 or 2 years
Do you think that we will see the small offset between campaign epoch? I don’t. Each layers will be aligned to the local reference system as it is today.
Do you think that in the Osm world we should add this small offset ? And we should correct this offset every year ?
The answer isn’t so easy.
-Every smartphone integrated gnss receiver offers at least a half meter accuracy I don’t know it these receivers will display the coordinate in WGS84 or in the local datum. I just know that today, professionnal in GIS don’t use WGS84.
@SimonPoole I’m not responsible for these aerial tms/wms, I can’t do it.
My original post was more about finding a way to store the accuracy in gpx files, but all the comments are on the datum problem. I think I will create another diary to split the subjects.
Comment from SimonPoole on 25 May 2019 at 13:00
@StephaneP IMHO currently the differences tend to be so small (with the exception of AUS as mentioned where you probably could detect the difference between a very well traced building from 2007 or so and now), that it is not a real issue. However as we know -when- a node was added, in principle we could apply a cumulative correction at point in time when the difference has become too great (we likely wouldn’t want to do that for other reasons though).
“Professional” surveyors typically don’t have the problem as they are not trying to produce a global, easy to consume geo-dataset, so it is completely OK for them to use whatever the national datum is.
Comment from SimonPoole on 25 May 2019 at 13:12
And what I wanted to mention you can tag the tracks on upload, which likely won’t help very much, Further you could set attributes for the track points that make the situation clearer too https://www.topografix.com/GPX/1/1/#type_wptType but again if that is really useful is a different question.
Comment from StephaneP on 25 May 2019 at 13:12
@SimonPoole Do you know how the australian contributors manage this? Is their aerial imagery locked to a local datum or if it moves with the tectonic plate?
Comment from Dzertanoj on 25 May 2019 at 18:45
Performing an appropriate shift to compensate for the tectonic movement is technically possible when a dataset retains just one little bit of metadata: acquisition date.
Comment from smaprs on 25 May 2019 at 20:54
The correction of images is made by the providers of the URLs. Perhaps Bing and DG may have that doccumented. OSM don’t change that.
I’ve read that, for the continental plates, Australian plate is the fastest moving,~7mm/year, wich means that in 15 years it will be displaced by 1m. I imagine that for self-driving cars 1m may be a great difference, from driving in the correct lane to driving in the contrary. OSM already has 14 years. So much of the vectors have already some displacement. Software may compensate this for some time without changing the vectors, but one day it will probably need a major correction. But the greatest danger ever for trafffic still is the existence of that left-hand driving.
Comment from StephaneP on 28 May 2019 at 12:37
@smaprs It’s not 7mm/year, but about 70mm.
I tried to find which datum is usually used on australian aerial imagery, but I didn’t find a real answer. It’s GDA94, GDA2020, or WGS84. On the talk-au mailing list, I found this old message (2009) from an aerial imagery provider, Nearmap: https://lists.openstreetmap.org/pipermail/talk-au/2009-December/004656.html The today Nearmap website says this about accuracy: https://docs.nearmap.com/display/ND/Accuracy
I think it’s easier to create data on a plate-fixed datum, and move these data if you want to display them with another datum.
If you want to do the opposite, you have to store the original date and compute the delta for every single node in the database.
@SimonPoole Which attribute seems ok for you to store the point accuray?
Comment from smaprs on 28 May 2019 at 13:02
You’re right! I’ve swalloed that decimal
https://en.wikipedia.org/wiki/Australian_Plate
https://en.wikipedia.org/wiki/Australian_Plate#/media/File:AustralianPlate.png
10m in 15 years! :-o
Comment from StephaneP on 28 May 2019 at 13:09
Yes, that’s fast, and they are moving to a new plate-fixed datum: GDA2020 https://www.researchgate.net/publication/319454295_GDA2020_AUSGeoid2020_and_ATRF_An_Introduction
Comment from SimonPoole on 28 May 2019 at 13:28
Well we already have the date for every coordinate in the database, including all historic versions.
I would invent something for the tag and use
https://www.topografix.com/GPX/1/1/#type_fixType
dgps to indicated that it is a differential gps generated position.
Comment from StephaneP on 28 May 2019 at 14:11
We know when the node was added in the database, but it doesn’t say what is the original datum epoch. And If the contributor add a node from an aerial imagery, we have to know what is the wms/tms datum….and it’s not an easy task (I spent a lot of hours trying to find the real datum of various Australian imagery).
I know that some of you are thinking that I’m splitting the hair, and it is likely right for today. But in the future we will have to manage these problems.
about gpx: I’d like to be able to add an attribute, as easily as in OpenStreetMap, but the gpx would be invalid and the attribute would be simply ignored. I’ve sent a message on the gpx dev mailing list.
Comment from StephaneP on 28 May 2019 at 14:33
I finally found a “mark” in Autralia, a gnss station. Go to http://www.ga.gov.au/ngrs/ Then search for a NGDB station, select GDA94 datum and type stromlo glonass in the “name” field. Then you can copy/paste these GDA94 coordinates in Josm (S35°18’ 58.19897” E149°0’ 36.54767”) to add this node. Just add one of the available imagery layer and look at the result:
We should check other survey_points to confirm, but It seems that the imagery use the GDA94 datum, not the WGS84, or we would see a 1.7m offset.
Comment from StephaneP on 30 May 2019 at 15:22
As this gnss station is in the IGS network, we can download the observation data, and then use an online service to compute a PPP solution. The AUSPOS service gives back the coordinate with several datums. Here are the results from 2018-03-31 with GDA94, IRTF14(2018) and GDA2020 datums.
I’m more and more confident that this aerial imagery isn’t using an actual WGS84 datum…like the french aerial imagery, and potentialy many others.
Comment from jidanni on 20 August 2019 at 10:49
Alas, precise coordinate entry of points is not encouraged for regular users.
Comment from TheSwavu on 20 August 2019 at 11:07
@jidanni Here’s a nickel go and buy yourself a real editor.
;-)
Comment from jidanni on 20 August 2019 at 11:12
@TheSwavu https://help.openstreetmap.org/questions/28637/how-to-repurpose-mouse-buttons-in-josm
Comment from TheSwavu on 20 August 2019 at 11:17
@jidanni I didn’t say it was pretty, just that it get the job done. Strangely I was just discussing this on the tagging list: https://lists.openstreetmap.org/pipermail/tagging/2019-August/047582.html
Comment from jidanni on 20 August 2019 at 11:32
@TheSwavu, Yes, and I admire your ability to handle switching from left to right, for me it messed me up for a whole month.
Comment from Alexey Vazhnov on 9 March 2021 at 19:18
@StephaneP, did you find a solution how to filter high-accuracy tracks?
I found that I can filter tracks manually in JOSM, from a list with names and descriptions. So I can upload tracks with description suffix like “ZED-F9P with correction: …” and I will filter them. But it is better to have possibility to set, for example, VDOP/HDOP in GPX tracks, for every track point.
Comment from StephaneP on 15 March 2021 at 20:28
@Alexey
Nothing new since this diary, but there are some tickets in Josm to recognize single/float/fix inside nmea and gpx files. That’s a good start. https://josm.openstreetmap.de/ticket/20600 https://josm.openstreetmap.de/ticket/20601
Comment from StephaneP on 15 March 2021 at 20:31
I forgot to say that now, you can open RTKLib .pos files inside Josm. It helps me a lot : https://josm.openstreetmap.de/ticket/17829