Maybe visualize where you have GeoTIFF image tiles and where you have mask tiles. It could be that the GeoTIFFs just don’t cover all of the areas you extracted masks for.

Otherwise try to reproduce this with a small GeoTIFF and/or a smaller area.

And maybe try the gdal2tiles approach and see if the output is different.

You need to debug this a bit - could be multiple problems.

.results is the jq filter already. You can pipe the result of the http query into a file and then do jq '..' file or do http .. | jq '..'.

Check https://stedolan.github.io/jq/manual/

You need to run the jq command either on a json file or pipe the output from the http query directly into it. Please see the jq documentation and quick start.

If you don’t want to install robosat on your own, we provide both CPU as well as GPU images you can docker-run directly: https://hub.docker.com/r/mapbox/robosat/

Regarding your problem cutting out an area from an osm base map: it sounds like either the cutting is creating invalid polygons or there are invalid polygons in the base map. In addition it sounds like cutting out an area does not properly keep the way locations that’s why you are running into the location problem

osmium-tool’s extract comes with sane default strategies to prevent all of these issues.

Wnat you can try if you are having a hard time installing osmium-tool is to use the mason package manager to get a pre-built binary:

mkdir ./mason
curl -sSfL https://github.com/mapbox/mason/archive/v0.17.0.tar.gz \
    | tar --gunzip --extract --strip-components=1 --exclude="*md" --exclude="test*" --directory=./mason

./mason/mason install osmium-tool 1.7.1
export PATH="$(./mason/mason prefix osmium-tool 1.7.1)/bin:$PATH"

osmium extract \
  --bbox -84.6673,33.5219,-84.0427,34.0829 \
  -o atlanta.osm.pbf \ 
  georgia-latest.osm.pbf

I think osmium-tool comes with documentation for how to build it. For protozero and libosmium, I would just compile them from source, too, and not rely on the package manager. If you are running into compilation issues I would open a ticket with these libraries.

You don’t strictly need osmium-tool I just used it because I had it around. You can use any tool to cut out a smaller area out of a larger pbf base map.

The problem with using rs download against the OpenAerialMap endpoint is they provide a different endpoint for each underlying GeoTIFF. This means you need to rs download from many endpoints and you will see lots of errors.

The alternative is to download the raw GeoTIFFs to your machine and run either gdal2tiles or the small tiler tool I wrote here to cut out tiles from the GeoTIFFs.

You will find the GeoTIFF URLs in the OpenAerialMap API response (see the description above using the /meta endpoint and the jq transformations).

Hope that helps

You will need two URLs with tokens. One for the compare map to load styles and one for the tile server to fetch satellite or aerial imagery from to run segmentation on it.

You will need to adapt the center locations for the compare map (for both the before and after map) here. The after map is then adding a raster layer automatically requesting tiles from the rs serve tileserver here.

Again, I recommend using the rs predict batch prediction tool.

Your --url argument is an URL with an access token, right? Like

--url https://api.mapbox.com/v4/mapbox.satellite/{z}/{x}/{y}@2x.webp?access_token=TOKEN

The serve tool is currently fixed at zoom level 18 since we only used it for initial debugging. If you are not using z18 you have to change it here:

https://github.com/mapbox/robosat/blob/6c7f547d2dd303f8a68b4fbd2ba60583348d7816/robosat/tools/serve.py#L53-L55

You will definitely see many 404s since the serve tool tileserver will only respond with e.g. z18 tiles but the map will request e.g. z16, z17, z18, z19, all at the same time.

In addition you probably want to adapt the map template’s initial location, zoom levels, and so on:

https://github.com/mapbox/robosat/blob/6c7f547d2dd303f8a68b4fbd2ba60583348d7816/robosat/tools/templates/map.html#L46-L64

I recommend using the rs predict tool for proper batch-prediction. The rs serve tool is really just a simple debugging tile-server and needs a bit of getting into the code.

You will get the warning when the downloader was not able to download tiles from the list of tile coordinates you give it. This is probably due to the tile endpoint not providing imagery for all your tile ids.

https://wiki.openstreetmap.org/wiki/Aerial_imagery is a good place to start when looking for imagery sources.

Agree, for larger tiling jobs I recommend using proper tools like gdal2tiles.

I can’t debug statements like “contains garbage images”. What does your GeoTIFF look like? Do you have a small self-contained example where I can reproduce this issue? What do the Slippy Map tiles look like? You can give gdal2tiles a try; the tiler script was just a quick way for me to tile my GeoTIFFs.

It doesn’t matter where you get the bounding box from; I used http://geojson.io since it’s convenient to use. And yes, it is currently working for me.

The bounding box is only used for cutting out a smaller base map from a larger .osm.pbf and downloading aerial images in that area. After that we work with data extracted from the base map and the aerial imagery as is.

Not sure why you’d want to run the RoboSat toolchain in Jupyter Notebooks, but I guess there is nothing stopping you from doing that. You have to install the RoboSat tools and its dependencies then you can use the robosat Python package. The rs commands are really just a small shell script expanding to python3 -m robosat.tools cmd args.

We train on (tile, mask) pairs, that’s right.

But for prediction we buffer the tile on the fly (e.g. with 32px overlap on all sides), predict on the buffered tile which now captures the context from its eight adjacent tiles, and then crop out the probabilities for the original tile again. This results in smooth borders across tiles in the masks. The probabilities you are seeing above might still not match 100% at the borders, but that’s fine.

Here is the original tile and how the buffering during prediction affects it.

original tile in the middle, four corners from adjacent tiles added for buffering

fully buffered tile with context from its eight adjacent tiles

Without this buffering approach you would clearly see the tile borders, correct.

IoU is still below 0.8

If you reach an IoU of 0.8 that’s pretty amazing to be honest. Here’s why. There are two sides contributing to the IoU metric: your predictions can be off but worse also the OpenStreetMap “ground truth” geometries can be off. Even with a perfect model you won’t reach an IoU of 1.0 since the OpenStreetMap geometries can - and often are - coarse, or have a slight misalignment, or are not yet mapped in OpenStreetMap etc.

Here’s an interesting experiment: randomly sample from your dataset. Manually annotate the image tiles generating fine-detailed masks. Now calculate the IoU metric on your manually generated fine-detailed masks aand the automatically generated masks from OpenStreetMap. This will be your IoU upper bound you can reach.

Also see this graphic to get a feel for the IoU metric.

Training this without GPU takes almost a week :-)

Agree, without a GPU training will be slow. That said, I made some changes recently which should speed things up considerably:

• https://github.com/mapbox/robosat/pull/65 - Simplifies the training setup; now you no longer have to manually tune learning rate and more importantly epochs at which to decay the learning rate. This should give you great training results without any tuning.

• https://github.com/mapbox/robosat/pull/46 - We are using an encoder-decoder architecture. Before we were training both the encoder as well as the decoder from scratch. This changeset brings in a pre-trained ResNet encoder, resulting in faster training times per epoch, less epochs needed, less memory needed, higher validation metrics, and faster prediction runtime.

If you want to give it a try with current master you should see improvements for your use-case.

The amount of images you will need for training can vary a lot and mostly depends on

• the imagery quality, and if it’s from the same source or not
• how good and precise the masks for training are
• the zoom level
• the variety in the images, and if it’s from the same area or totally different
• the time and processing you can invest

For example the more hard-negative iterations you do the better the model can distinguish the background class. But hard-negative mining also takes quite a while. Same with the automatically created dataset: you can manually clean it up it but it is quite time-intensive.

In addition you could do more data augmentation during training to further artificially embiggen the dataset, you could do test-time augmentation where you predict on the tile and it’s 90/180/270 degree rotations and then merge the predictions, you could train and predict on multiple zoom levels, and so on.

I would say it also depends on your use-case. For detecting building footprints like in this guide a couple of thousand image are fine to get the rough shapes. It’s definitely not great for automatic mapping but that is not my intention in the first place.

Regarding trained models: I recently added an ONNX model exporter to RoboSat which allows for portable model files folks can use with their backend of choice. I could publish the trained ONNX model for this guide since I did it on my own time. The Mapbox models I am not allowed to publish as of writing this.

If there is community interest maybe we can come up with a publicly available model catalogue hosting ONNX models and metadata where folks can easily upload and download models?

Can you create an issue in the https://github.com/mapbox/robosat repository. I haven’t seen rs extract erroring out; there may be some warnings due to invalid polygon shapes but that should be it. Let’s follow up on a ticket.

@Rory: here are some rough numbers: with a single GPU you can expect training on ~20k images to take in the order of multiple days. Prediction on a GPU will take between 350 ms – 750 ms per 512x512 image tile - depending on how much overlap you buffer per tile to handle tile boundaries. Doing the same on CPUs and especially your laptop will probably slow things down by a factor between 10x – 50x at least.

What you can do is prepare all the datasets and get ready for training. Then spawn up a AWS p2 instance (install nvidia drivers, cuda, cudnn) and train on there for a few days. Save the model, shut down the instance, predict on CPUs. You could also look into getting a GTX 1080 TI but it’s only really worth it if you want to do these sort of things on a large scale for a couple of months at a time running 24/7.

There are “bottlenecks” in the road network which the partitioner detects. You can think of it as answering the question: which roads do I have to cut out of the road network to split it in half. Then we optimize for the minimum amount of roads to cut out and we balance the cuts to divide the road network “roughly” in half on every recursion level. This gives us roads we know cars have to go through when driving from one cell to the other.

Here are two maps from almost three years ago when I prototyped the partitioner and did these visualizations at a Geofabrik hack weekend in Karlsruhe. I remember we were wondering if the partitioner would cut old east and west Berlin in half.

The first one shows multiple cuts for Berlin. We run multiple cuts with a different slope in parallel and take the best cut on every level. Then we recurse down repeating the process in parallel for both sides.

Map - Berlin

And a single cut how to split California in half removing the minimum amount of roads.

Map - California

Note: this was from a prototype and probably does not match what you will currently get from our actual implementation in the Open Source Routing Machine.

Here is the current implementation:

• https://github.com/Project-OSRM/osrm-backend/tree/master/include/partitioner
• https://github.com/Project-OSRM/osrm-backend/tree/master/src/partitioner

You can definitely make use of these partitions for visualization or other purposes.

Also check out the Inertial Flow paper.

Thanks, I just added a note to the original diary post.