This blog post is part of a series of blog posts about the new OSM file format “OMA”. This is the last post. At the end of the article you’ll find links to the other blog entries.

 

I will end this series of blog posts about the OMA file format with a short summary of all the feedback I have received and a brief outlook. But first I’d like to give you a real-life example that uses OMA files.

 

A Real-Life Example

Over a year ago I wrote a renderer suitable for displaying micromapped areas. I used it to generate tiles at zoom level 20 for the restricted area of my micromapping project.

Now I have adapted this program to use OMA files and recomputed the tiles for the area of my micromapping project. The program can be found on GitHub; and there’s a slippy map showing the tiles.

For example, this is what the station forecourt of Hilden looks like:

 

Summary

… See full entry

This blog post is part of a series of blog posts about the new OSM file format “OMA”. This is the seventh post. At the end of the article you’ll find links to the other blog entries.

 

To create and use blocks and slices, you need to know the type of an element - is it a highway, a landuse or a barrier. And if it is a highway, is it a primary, a secondary or a footway.

 

Keys

Apart from relations where the type key stores it, this information is not available within OSM data. Again, it has to be guessed, and there is much discussion on the forums and elsewhere about the “main keys”, as they are sometimes called. Several apps, such as most OSM editors, contain such a list, but the lists are not always the same.

The approach I took was this: I counted all the keys of all the tags of all the elements and took the one that appeared most often in this list. I considered this the first main key and removed all elements with this key from the set of elements. Then I repeated this process.

After a while some “dirt” remains: elements that do not show a clear “main key”. I think if you start digging around in that dirt, you might find some hidden gems, that is, main keys that aren’t used very often, because the items they represent do not exist very often in the real world. But I didn’t.

I repeated this process twice,¹ once for nodes and once for ways. You have to keep an open mind, because sometimes the top key is clearly not a “main key”, for example name pops up after a while. Addresses needed some special treatment, because there is no address key. Addresses are stored as a collection of addr:* keys. Fortunately the addr:housenumber is almost always present, so I used that as the key for addresses.²

 

Values

… See full entry

This blog post is part of a series of blog posts about the new OSM file format “OMA”. This is the sixth post. At the end of the article you’ll find links to the other blog entries.

 

The Oma file format uses a set of bounding boxes to sort the data geographically. The default set used by the converter consists of three grids.

 

The Default Grids

The first grid uses a mesh size of 1 degree in both directions between -45° and 45° latitude. Towards the poles the mesh size increases and finally the area around the poles is put into a single bounding box.

The grid looks like this:

The second grid uses a mesh size of 10 degrees in both directions between -80° and 80° latitude. It does not include the areas around the poles.

This grid looks like this:

… See full entry

This blog post is part of a series of blog posts about the new OSM file format “OMA”. This is the fifth post. At the end of the article you’ll find links to the other blog entries.

 

When relations where introduced (in 2008, if I remember correctly) I was a big fan of them. They sounded great and promising. Meanwhile I think, it was a mistake to add relations, and if I where to decide I would probably remove them from OSM and replace them with something better.

Let’s have a closer look: Nodes and ways (and areas) are simple: A geometry with some properties. Relations do not share this simplicity. They share the properties, but they do not have their own geometry. Instead, they contain references to some elements. This makes things complicated.

I could write several articles about relations and why I think they are a bad idea, but this article is about relations in Oma files, so I’ll leave it at that for now.

 

Multipolygons

Instead, I’m going to take a closer look at what is currently done with relations. According to taginfo, more than half of all relations are multipolygons, that is, a collection of areas. Since we’ve already got areas in Oma files, this is easy to handle: All multipolygons are converted to one or more areas.

… See full entry

This blog post is part of a series of blog posts about the new OSM file format “OMA”. This is the fourth post. At the end of the article you’ll find links to the other blog entries.

 

The real subject of this blog series is the newly developed file format. So far I have mainly talked about the tools for creating and using the format, because the format itself is a dry subject. But now it’s time to dive more deeply into the format itself. I will not go into all the details, because I think that apart from some freaks like me, people are not interested in all the details. If you are, take a look at the specs.

 

Fast Access

OpenStreetMap data consists of a set of elements. Some of these elements are nodes, some are ways, and some are relations. You can think of OSM as a big storehouse where all the elements are scattered around:

Obviously it’s hard to find what you’re looking for, if there is no order.

The Order of OSM Files

… See full entry

This blog post is part of a series of blog posts about the new OSM file format “OMA”. This is the third post. At the end of the article you’ll find links to the other blog entries.

 

Until now you’ve got a general idea of what the Oma file format is, and an idea of how to use it. But you do not know, where to get an Oma file from.

Well, I hope, that sooner or later someone like Geofabrik will provide a daily updated planet.oma and some excerpts. That would make sense, because converting the data takes a lot of resources, and it would be a waste if everyone had to do it themselves.

But until we have such a distributor, you have to convert OSM files to Oma files on yourself. I have written a converter for this purpose. It’s written in Java and should be easy to use.

 

The Converter

You need a copy of oma.jar. If your are using Linux (or any other Unix operating system) you just have to type the following command:¹

java -Xmx<some number>G -jar oma.jar <some osm file>

The -Xmx part tells the Java Virtual Machine to use <some number> gigabytes of memory. For example, my computer has got 4GB of main memory, so I’ll use -Xmx3G, reducing the available memory by 1GB, because the operating system needs some memory too.

The osm file mentioned in the command, can be one of .osm, .o5m or .pbf².

Well, that’s about it. The program will read the file and start the conversion. This can take a long time, and hopefully it won’t crash.

Huh, crash? I wish I could give you better news, but unfortunately I have not been able to write a program that does never crash. The reason for this is that Java gives no guarantees or means of dealing with out-of-memory situations.³ So: If you have enough memory (and disk space), a crash should never happen, but if you have only limited memory, a crash might be possible.⁴

… See full entry

Please note: This blog post is part of a series of blog posts about the new OSM file format “OMA”. This is the second post. At the end of the article you’ll find links to the other blog entries.

 

This time I will give you an example of how to query Oma files. I wrote a prototype of a library for working with Oma files. I called it OmaLibJava.

To explain how to use this library, let’s say, we want to create an overview map of all the power facilities in a certain town.

 

The Classical Approach

The classical approach with OSM files would be, to first reduce the size of the file by creating a smaller file containing only the data of interest. This is typically done in two steps: remove everything that is not a power facility / remove everything that is not in the town. The order of these two steps is not important for the result, but might have a huge impact on the duration of the process.

Although it is not necessary – or even counterproductive – this can easily be done with Oma files too. For example, the following small Java program extracts all power facilities of Germany:

import java.io.*;
import de.kumakyoo.omalibjava.*;

public class ExtractPower
{
    public static void main(String[] args) throws IOException
    {
        Extractor e = new Extractor("germany.oma");
        e.addExtract(new BlockFilter("power"),"power.oma");
        e.run();
    }
}

Running this program on germany.oma on my computer takes 8.5 seconds. It creates the file power.oma, which is about 22MB in size and contains all power facilities of Germany.¹

Let’s have a closer look at the program: the library contains a class called Extractor, which reads an Oma file (here germany.oma) and writes several extracts simultaneously (here only one, called power.oma).

An important part is the filter BlockFilter("power"). This tells the extractor to keep only elements with the key power. I’ll tell you more on filters in a moment. But first I want to show you why you don’t need this step.

 

… See full entry

Whenever I use files containing OSM data, I’m faced with two major problems. These problems are inherent in the OSM data model and inherited by the common file formats (.osm, .o5m, .pbf).

The first of these problems is about accessing the data: As a result of the data model users are forced to use either huge amounts of memory or a lot of time. Often even both.

The second problem is even worse: Quite often you have to guess properties of the data, which means using heuristics. But by their very nature, they can lead to wrong results. The most prominent example is probably the question, if a closed way element represents a linear object or an area.

To overcome these problems I invented a new file format. The main idea: Convert the data once (accepting the drawbacks caused by the two problems) and end up with data that can be processed fast, using only little memory. I also tried to keep the file size small and the file format simple.

I called the new format “OMA format” like “Open MAp”. It’s accompanied by a human readable version called “OPA format” like “Open PAper”. (Oma and Opa are the German words for grandma and grandpa).

It took me about a year to design the Oma file format, and write a converter and a library for querying the new file format. I still consider it in experimental state, as I would like to get some feedback before releasing a stable version. Hence this post.

All this is a lot of stuff and I can’t go into all the details in just one blog post. For this reason I’ve decided to keep this article brief and base it on a single example. In the coming weeks, I will be writing more articles about the new file format, which will provide more in-depth information.

 

Scattered Data in OSM Files: The Viktorstraße in Wuppertal

… See full entry