OpenStreetMap
In India, MapMyIndia is one of the biggest aggregators of commercial map data, including running their own fleet of mapping cars, so it was quite interesting to find OSM data in their map for areas i’m familiar with and have mapped on the ground.
Its not all OSM though, will need to dig more into how they mix the data.
Planning a trip to a remote area and realized it would be really cool to meet other mappers in the region where i’m traveling. Wish there was an easy way to broadcast a message to the top/recent contributors in a location. Right now the easiest way seems to be looking at the the list of top users when editing in iD and sending them message.
Any other tools that could help? Very few users are on any community communication channels here in India.
Some late night weekend experiments resulted in this Airport map style.
Runway and taxiway lights
Terminal buildings
Airport boundaries
My laptop seems to heat up a bit trying to render it, but its great to see the detailed results of all the intricate work of the mappers (hooray runway and taxiway areas!). If you update your airport on OSM, you should see the result on the map in a few minutes. My previous experiment with adding runway lights to maps looked like this
PS: Feel free to build on top of the style, you can download this style.json and upload it into Mapbox Studio for a live map.
There are over 40,000 features with old style wikipedia:lang=* tags that need to be migrated to the more popular wikipedia=lang:* format. Should this just be a mechanical edit? Or should we have a worldwide campaign of community members checking each and every one?
UPDATE
Looked into the features with wikipedia:ru and the majority seem to be place nodes in Ukraine. Around 28k of them already have a wikipedia (mostly Ukranian) tag, so maybe migrating them may really not have any added benefit.
So the question now is, if its better now to discard the wikipedia and wikipedia:lang tags and just have a wikidata tag? What information would we lose?
Special shoutout to Undearius and LogicalViolinist for their efforts in improving Wikidata coverage in Africa and for bringing a smile to the face of the Sahara.
Some trivia, the Sahara is not all endless sand dunes, one can chance upon some oases and also some potential mappers. Something to lookout for on your next trip there.
CC-by-sa TASSINE Ilyas
Sahara Race CC-by-sa Jon Doe
You’ve probably already seen the video of the Fukuoka sinkhole swallowing up large parts of an intersection. This obviously necessitates a map update, as it could be a little bit of an inconvenience to drive on this stretch.
In Japan updates are quick, both on the map and in the real world. From the detailed traffic restoration plan the road might actually be back to normal before any other map gets updated ;)
Wikidata, the crowdsourced database of structured knowledge by the Wikimedia movement has grown to over 24 million entries and by now has structured information for every major settlement on earth. These are extremely useful properties like multlingual labels, statistics like populations and GDP, and other related information like politics, history and media about the place (See London, New York City, Timbuktu).
Geolocated articles on Wikidata, with those added in the last year highlighted in pink. Source: Wikidata Map
One of the great strengths of OSM is to leverage the data to create create multlingual maps that make the map accessible to a lot more readers than just the local population. Since the beginning of the project, the community have been adding various name:code tags for this purpose, and has resulted in map features with a ever growing list of multilingual names eg. the node for London has 171 properties, of which 155(90%) are name tags in various languages.
A more scalable approach would be to leverage the Wikidata entry for London, which has the translated name in 248 languages, and growing automatically with every Wikipedia page of the city that is created in a new language.
This would also enable the translation of a map to languages where that language on OSM would be considered non local and not worthy on adding to the map, eg. Ukranian labels for cities and towns in UK.
The first step to start leveraging the power of Wikidata from OSM is adding a simple wikidata property to the feature on OSM with the associated QID of the corresponding concept on Wikidata eg. wikidata=Q84 for London. Check out this video by user:polyglot on doing this via the JOSM Wikipedia plugin or via the iD editor.
Just like OSM, Wikidata items of places have tags describing the feature and coordinates that make it possible to automatically match a feature on OSM to the corresponding feature on Wikidata. Unfortunately the geographical accuracy of Wikidata entries cannot be trusted, as many of the coordinates are derieved from Wikipedia pages which in turn are usually derived from Google Maps. Moreover entries of lesser known places may not be tagged correctly on Wikidata and might result in ambiguous matches to an OSM feature. For this reason manual confirmation of a match is necessary.
At the Mapbox data team, we have been experimenting with adding Wikidata tags to cities and towns on OSM based on an exact name and location match. The possible matches were loaded onto a spreadsheet with the match distance and Wikidata description of the corresponding item. After a manual review, its easy to confirm the match with a very high degree of confidence based on the name, distance and description of the match. With this approach we have found that just an exact name and location match can give a 99% success rate for places.
Over 5,300 cities and towns have been updated with corresponding wikidata tags in the last two weeks http://overpass-turbo.eu/s/jGy
There are two cases when the name matching happens: - Unique matches: One OSM feature matches to one Wikidata feature - Duplicate matches: One OSM feature matches to multiple Wikidata features with the same name
Unique matches
In most cases, the location of the matched feature on Wikidata is less than a few Kms, and by confirming from the description that the feature is also a city or town, its possible to confirm this was the correct match. It is important to be careful about the feature description as in some cases Wikidata may have ambiguous entries that represents multiple concepts like both a city and a province with the same name as one object.
For unique matches with a large match distance >10kms, it is likely the match was to another place with the same name and is an incorrect match. In a few rare cases, the Wikidata location was found to be incorrect and was actually a correct match.
Duplicate matches
When an OSM feature matches to multiple Wikidata entries with the same name, it is considered a duplicate match. In most cases a distance filter of around 10km enables a unique match, and a further look at the description can confirm the match is correct.
In a few rare cases multiple OSM features with the same name and location match to a single Wikidata feature. These are places with duplicate nodes on OSM itself and need to be merged.
Large scale map features like countries, cities, towns and water bodies are great candidates to start matching with Wikidata as they are fairly well defined on both projects and can be matched without ambiguity. Doing this will allow us to better understand the value that Wikidata can add to OSM, and help pave the wave for more interesting map services that can be built on open data.
There’s been some amazing work from EdwardBetts on matching all of Wikidata to OSM. You can see the results and this can be a good push to the efforts of contributors like User:Pigsonthewing on bringing the two biggest crowdsourced open data projects in humanity can get closer together.
Might depend on what exactly one considers a point of interest, but looking at some common keys:
we get around 20 million features that could be potential landmarks or points of interest.
Just counting points that have names, we get around 12.9 million features.
Addresses
70 million individual addresses and 49 million postcode locations
Maybe there is a better way of finding out, but here’s a new conversation starter at your next mapping party!
Found myself at the Royal Observatory this morning (on the map), and discovered a bit of trivia since I expected to be located at longitude: 0°0 as my geography class would have made me believe. Well, its not:
Prime meridian at Greenwich. CC-by-sa ChrisO
The prime meridian established at the Royal Observatory, Greenwich in 1851 is actually at 0°0′5.3″W, thats 102m west of the the 0°0 prime meridian used in the WGS 84 coordinate system. Why? Because the modern coordinate system was corrected to have its origin at the Earth’s centre of mass.
Read more at https://en.wikipedia.org/wiki/Prime_meridian#International_prime_meridian
GPS coordinate at the Greenwich meridian. CC-by-sa Jckcip
There is an open dataset of turn restrictions in Toronto which gives a rare opportunity to completely map every restriction in the city.
Inspecting the dataset in QGIS
The dataset consists of the topology of road centerline of every junction in the city, and at every junction, each possible turn direction is an individual line with a property that indicates the type of restriction on it.
Most of the restrictions are u-turns and no turns into one ways. The only restrictions which look valuable to add into OSM seem to be the no turns (By law) which number about 520.
All turns at a junction with their restriction status
It seemed like an interesting exercise to compare this with the restrictions already mapped on OSM and check if it can be used for validation or add missing ones. To do this, try this simple data tool to browse the Toronto restriction dataset and verify it against the map.
Turn restrictions in Toronto (Red: no left, Orange: no right, Green: no straight) on the data-mapper tool
Clicking on the map allows you to open the location in JOSM, and also set a review marker. You could either review it as:
The existing OSM restrictions show up as purple lines once you zoom in. Interestingly, I could notice plenty of restrictions on the map that is not present in the Toronto dataset, so looks like the map may actually be more updated in many areas than the data the city has.
Feel free to explore and share your observations. Toronto could well become the first city on OSM with comprehensive turn restriction data that is verified against an official data source.
While tracing an area of continuous buildings in India, realized there was no easy way to create such a pattern of features in JOSM.
How does one split a group of ways using an intersecting line. The More Tools>Split Object command comes closes but works only with a single way and line that it encloses and is quite laborious.
Terracer plugin does not work well since the area is not a perfect rectangle, so it does not get the axis right. Any other editor/plugin capable of drawing this quickly?
A new user accidently added addr:city and addr:province tags to a large selection of objects, including rivers and individual nodes.
The user mentioned that the tags were intended only for buildings, so thought I would document the cleanup for future efforts like this.
It is possible to extract all objects with addr:city added by the particular user using this Overpass query.
Resulting features
This data can be exported into JOSM via remote control for further inspection and cleanup.
Once all the data is loaded, we need to select only those objects or features that have been incorrectly tagged with the addr:city and addr:province. Since we know that the tags are valid on building ways, we will use a filter to select such tags on anything that is not a building way. We can use the find dialog (ctrl+F or cmd+F) to look for exactly this.
Find all objects tagged with addr:city and not a building
This selected 3,989 objects from which the tag can be deleted. Just select and delete the tags.
As a bonus got to also cleanup duplicate nodes using the validator.
Upload with a descriptive changeset comment and all done!
There is some interesting research on the validity of polygons/area features mapped on OSM by Jochen Topf the creator of the osmium suite of OSM processing tools. There seems to be two major problems:
In his usual style, there exists an in-depth explanation of the problem. Bonus is a chapter on OSM polygons and how they are rendered for GIS nerds.
Of the over 220 million (multi)polygons in OSM more than 100,000 contain mapping errors of one kind or another and about 250,000 are tagged old style with tags on the outer ways instead of on the relation making multipolygon tagging and processing much more complicated and much more expensive than it needs to be. Read more
I dug into the dump of invalid polygons and inspected it in QGIS to get an idea of the breakup of issues to see if they can be prioritized to fix:
There are 2500 invalid areas with over 50,000 nodes, so these are some very very large issues.
82% of issues are role mismatches. For eg. an outer area inside an outer area, or vice versa. The next biggest issues are duplicate members or self intersection. Other issues are less than 2%.
Here is a map of all the issues. Just click on any one and you should be able to open it in JOSM for a closer look: http://osmlab.github.io/fixing-polygons-in-osm/map/
I tried to dig a deeper into each type of issue to understand the problem and get a sense of how it could be fixed. Feel free to explore and post your findings as well.
Might require manual cleanup based on context
Requires manual cleanup - trivial. Requires shifting the problematic node by a small distance in the right direction to clear the error.
- http://osmlab.github.io/fixing-polygons-in-osm/map/#17.64/13.05598/80.21087 
Requires manual cleanup - trivial. Requires ungluing the touching node and handling like a duplicated node
- http://osmlab.github.io/fixing-polygons-in-osm/map/#19.25/51.51343/-0.11559
Requires manual cleanup - Not trivial. Needs careful modification of relation geometry and roles if multiple segments are involved. Trivial if on the same segment just like way intersection.
Could be auto cleaned - trivial. If the role is inner with the same tags as the relation, it could be considered a hole and the tag discarded.
Edits: This is not always true and it could be an error on the mappers part that would require a careful decision.
Could be auto cleaned - trivial. Two nodes on the way in the exact same area. JOSM validator flags and fixes automatically.
Requires manual cleanup - Not trivial. There seems to be many special cases here.
Could be auto cleaned - trivial. Drop the duplicated member from the relation
- http://osmlab.github.io/fixing-polygons-in-osm/map/#18.55/51.50511/-0.18786 
Requires manual cleanup - Not trivial. Possibly invalid, incomplete or junk data. Might need retracing, removal or adding members to incomplete relation.
Not trivial. Might need to be remapped.
Most of the features with issues seem to be small landuse areas. I tried to find if there was anything significant on the map that was invalid. Here are a few:
landuse=forest or natural=wood http://osmlab.github.io/fixing-polygons-in-osm/map/#11.45/-4.3603/28.9007 http://osmlab.github.io/fixing-polygons-in-osm/map/#10.67/8.0507/18.6335waterway=riverbank or natural=water http://osmlab.github.io/fixing-polygons-in-osm/map/#14.15/7.8692/16.5565 http://osmlab.github.io/fixing-polygons-in-osm/map/#11.77/-3.0661/17.0305boundary=protect_area boundary=national_park http://osmlab.github.io/fixing-polygons-in-osm/map/#4.7/16.570/1.839landuse=meadow landuse=farmlandhttp://osmlab.github.io/fixing-polygons-in-osm/map/#7.89/47.360/4.341 http://osmlab.github.io/fixing-polygons-in-osm/map/#11.76/47.0182/0.2869A rather inspiring story came to my notice last week on World Environment day from Chethan during lunch, that of a lady who mothered thousands of trees just like her own children in rural India, not too far away from here in Bengaluru.
Saalumarada Thimmakka has planted over 8,000 trees, the most famous being 384 banyan trees on a 4km stretch of road from her home village Hulikal to Kudoor.
A bunch of us mappers Chethan, Maanya and Ramya plan to take a ride out of the city to meet this amazing lady and plant her banyan trees on the map!
Found this oddity through Chetan, that I never recalled seeing before.
At first thought someone squared a border, but this was created 4 years ago and is a disputed border area between Chile and Argentina.
Not been able to find many maps that show the border like this, but here is some interesting reading:
The Southern Patagonia Ice Fields from ISS. Source
| NYC had an import of over 1 million building footprints and 900,000 addresses in 2014 from the New York City Department of Information Technology and Telecommunications (DoITT). The DoITT GIS releases an updated shapefile of the footprints every quarter, and the latest version can be accessed here: Building footprints | Address points |
Open datasets like these are a great opportunity to explore how OSM can be used as a bridge between authoritative information and that crowdsourced by citizens. Two years after the import, it is interesting to see how the OSM data compares with the latest official footprints. The interesting questions to ask is:
Both of these are pretty challenging questions and requires some careful data comparison and conflation. manings and I were trying our hand at answering these questions and here is our progress.
Grab the latest NYC footprints from DoITT and the NYC OSM extract from Mapzen.
Use QGIS to filter only the buildings from the OSM extract and save it as a separate shapefiles to make the analysis slightly faster.
A simple way to quickly see a difference between the geometries in the two datasets is to do a visual diff by overlapping the layers with different colors. Geometries that don’t overlap will show the color of the underlying layer.
Green=missing in DoITT; Yellow= OSM overlaps DoITT; Red=missing in OSM. (interactive map)
This is a purely visual comparison and with some eyeballing, we noticed not much has changed. The green buildings on the New Jersey side are outside the import area and do not exist in the DoiTT dataset. We know have a few more questions that lie unanswered:
Both the above questions can be answered if we know which dataset is more updated, and the only reliable method to find out is to visit the site and ground truth the information.
Next we will try to detect changes in building configurations, where buildings might have combined or been split from its parent. This can be done by comparing the centroids of the two footprint datasets and check if they match. To do this, we can first extract the centroid of all the OSM footprints, and using a point in polygon analysis, find out how many centroid intersect with every DoITT footprint.
Black=0 OSM footprints at location; Grey=1 OSM footprint at location; Yellow= 2 OSM footprints at location; Red=2+ OSM footprints at location
This method can isolate the footprints that need to be added into OSM, but its not a simple insertion as there may be cases where a single building might need to be split into smaller buildings
This also detected overlapping polygons in OSM, and cases where multiple buildings need to be combined into larger buildings
In cases where the buildings have transformed their shape drastically, the centroid may not overlap and get flagged as a missing building
Minor shape changes don’t get flagged
Stats - Total footprints in DoITT dataset: 1,082,433 - No existing OSM footprint at location (PNTCNT=0): 1,223 footprints to add - 1 existing OSM footprint at location (PNTCNT=1): 1,072,975 footprints not changed or with minor geometry changes - 2 existing OSM footprints at location (PNTCNT=2): 1,661 footprints to be merged or updated - More than 2 existing OSM footprints at location (PNTCNT>2): 251 footprints to be merged or updated
| View Map | JOSM/iD TMS https://api.mapbox.com/styles/v1/planemad/cilja03bn002bccm7chcpw604/tiles/{z}/{x}/{y}?access_token=pk.eyJ1IjoicGxhbmVtYWQiLCJhIjoiemdYSVVLRSJ9.g3lbg_eN0kztmsfIPxa9MQ |
The above analysis can still miss out on minor shape changes. To find out which buildings were modified since the last import, we tried to compare shape changes using the nycdoitt:bin footprint id as primary reference key.
For each nycdoitt:bin polygon from OSM and NYC:
This process confirms the centroid diffs in our test area and can further identify shape changes where, the buildings were not deleted/added but shapes were improved, etc.
This is only true if the nycdoitt:bin id has not changed. However, we noticed that the building id from the NYC changes a lot and is not unique to each building. For example, in our test area, we found many duplicate entries (i.e. 1000000 = 106; 2000000 = 1482; 3000000 = 2052; 4000000 = 10119; 5000000 = 2623). Taginfo confirms this thus, using the nycdoitt:bin id is not the best approach to do this comparison.
–
Clearly none of these approaches give a clear picture of how to conflate the two datasets. Our next attempt would be at using a more traditional approach using PostGIS to intersect geometries and compare the differences for a manual review. We’re hoping to have a reusable workflow that can be used by open data agencies like NYC DoITT to use updates from OSM and also give back to the map more often.
Would be great to hear about alternate approaches for data conflation and strategies that could be explored in the future. You can look forward to another update on the spatial adventures of Maning and I soon.
Iquitos is a city of 370,000 in the middle of Amazon jungles of Peru without a road connection with the outside world. Found it while investigating routing failures in OSRM.
Here’s a great travelogue of this fascinating place while trying to research possible car ferry connections. There are none.
Maps lead you to interesting places :)
Hello Kitty
Quality assurance in OSM is a pretty hot topic. The first question when first faced with the concept of a map that anyone can edit is - how can one guarantee the quality of the data?
In a constantly transforming world, the fact is that its impossible to gaurantee any map in the world is accurate unless it was physically surveyed recently. And the reason why OpenStreetMap has such high quality of map data for many parts of the world is precisely because this project allows anyone to update the map so easily. And to ensure quality, we just need more users of the map for that location, an empower them to update anything that does not match reality.
For this to work effectively its obvious we need to make it as simple as possible for map users to spot mistakes and make an update. Some interesting data incidents that were caught by the community recently:
This is just a small list of examples that the data team at Mapbox stumbles into while ivestigating issues reported in our map feedback system. We have been using an amazing tool by willemarcel called osmcha which acts as a handy dashboard to review and investigate changesets. By looking for words like reverted or vandalismin the comments, its possible to identify changesets that corrected a previous mistake.
The interesting observation here is that many of these issues were accidental changes by the mapper which could have been easily avoided if they were more careful or had there been more sanity checks in the software. Most of the incidents have been fixed, usually by alerting the mapper and undoing their changes.
Something to be concerned about is the time it takes for an issue to be detected, which ranges upto a month for something as major as the label for a major city like Buffalo being deleted. But once the mistake is spotted, action and recovery is swift, usually a few minutes. This points us to an interesting question on why does it take so long for issues to be identified? And what happens to issues that are invisible on the map style?
While may mistakes are easy to visually identify in the map style especially if they involve large features like cities, forests, lakes or motorways, there are probably thousands of issues on smaller features involving inconsistent access tags, broken geometries or relations that would never be spotted or fixed unless an expert mapper stumbled on them. And there will be numerous cases of incorrect data intentionally being added to the map that will go unnoticed till a local mapper finds it.
Find whats wrong in the map universe with osmose
This is where tools for quality assurance of the OSM data come into play to address the issue. Some of the more popular ones are Osmose, Maproulette, Improveosm, OSM Inspector and keepright. These tools have greatly helped to bring in some sense of quality monitoring to the map, and could become a lot more powerful when part of an integrated validation environment.
This was one of the reasons that prompted us to create osmlint which makes it possible to run a global analysis of OSM data in a few minutes. Anyone want to know how many buildings in the world are shaped like cats? give it a try ;)
Another aspect to consider while identifying data issues is the mapper behaviour. Currently its common for mappers to check the edit count of a contributor to evaluate how experienced or trusted the user is. Pascal Neis has done some amazing research on user behaviour and the popular tool HDYC provides a detailed contribution profile for any mapper.
An open question is if aspects such a user reputation and experience play a part in validation workflows, and to what extent they can be relied upon to catch problematic behaviour like spamming and vandalism.
Now is a great time to get involved in the OSM data validation puzzle with the growing number of people using the map outside the OSM website through services like Strava and Github. There has also been some major technology leaps that allows us to process more data much faster than before opening up new technical approaches for solving the puzzle.
One of the big question for our team at Mapbox is - How can we make it as simple as possible for anyone who uses the map to validate it? Reaching this goal would greatly enhance the accuracy of the map data keeping it as close to ground reality as practical.
Have you thought of OSM validation before? What other big questions should we all be thinking about?
Smile, you are on OSM
One of the main reasons I was convinced of the power of OSM when I first started to edit 8 years ago was the possiblity of seeing maps in my native language Tamil, spoken by 70 million people, which was till then unheard of. This is a huge factor in making technology relevant for people on the ground and breaking the notion that one needs to be literate in English or the Latin alphabet to be able to use modern tools like interactive maps.
There have been numerous experiments at making multlingual maps in India, I remember Yahoo maps having Indic streetmaps for India in 2007, but was soon discontinued. Google Maps for India is primarily in English and even in OSM we use the name tag to record the name in the Latin alphabet for consistency. India has 22 languages, none of them universally understood by the whole country. Even the national survey agency makes only English maps, with a select few created in Hindi.
Hardly anyone in India even knows that OSM can handle regional languages, simply because its not visible anywhere on the map. After some recent interest from the community in making regional language maps for openstreetmap.in, I decided to give this a shot to make a multlingual place map for India using OSM and Mapbox Studio that I have been playing with recently.
First step is to get some sweet OSM data. For India, we are looking for a places that have one of the regional language name tags name:XX tag apart from the standard name tag in English. This overpass query extracts such places with names in either Malayalam, Tamil, Telugu, Kannada, Bengali or Hindi. There are around 21,000 such places in the Indian subcontinent. You can easily modify the query for your own language.
Once the data is exported as a geojson, you can upload it as a new Mapbox Tileset that can be styled later.
If you are comfortable with the commandline, you can extract the data, reformat and upload to Mapbox with these terminal commands.
I started of with a new Mapbox Studio style based on the ‘Emerald’ template, which is a great starting point to design a custom fully featured OSM based map. There are plenty of guides online to help you get started.
The official boundaries of India are slighlty different from whats displayed by default on OSM due to the dispute in Kashmir. Legally it is required to depict the entire erstwhile Kingdom of Kashmir as an undisputed part of India. To do this, I had the claimed boundaries extracted from various sources and have created a custom dataset that can be added to a Mapbox style.
The custom boundaries were added as a new layer in the style with a paint property similiar to the existing boundaries. With some tweaking, it looks close to perfect.
The defacto (OSM) and official boundaries of India
Just like the custom boundaries, its rather simple to add the overpass datasets that was uploaded earlier. For convenience, I made duplicates of the existing place label layers, and modified the data source for each of them to use the uploaded places instead of the mapbox-streets dataset provided as the default.
By changing the text field to use for labelling, we can instantly see the language change.
This is great, but does not solve the problem of allowing a user to switch between multiple regional languages of their choice.
Once the map was published in Studio, its possible to embed it easily into a custom html page using Mapbox GL JS. Inspired by the language switcher example that how its possible to switch between the default languages supported by Mapbox, I could extend the concept to switch the text field dynamically for the other custom layers in the style. The end output works like the animation posted at the top.
The code is available here and the switchable map should soon be live on openstreetmap.in. A non switchable version of the style can be previewed here.
This should fill a huge void faced by the OSM India community in being able to use the map in their native language and also help us showcase the power of the project to local Government agencies.
Would be great to hear feedback and further ideas you may have to extend this process to your own community.
Always satisfying to go into the map and add some sugar. Especially in the town centre.
This is the main station in Thiruvananthapuram a city of close to 1 million in India.
