DX, API Design & Documentation

Great ambitions with fewer dollars: DEM Net Elevation API

111views

How I have built an modern API platform, cloudless, servermore with zero budget

The project

Great Digital Elevation Models (DEM) have become public a few years ago. As no open source API nor library existed in the C#/.net field, DEM Net Elevation API was created to fill the gap.

The API produces 3D textured terrain in seconds from a single method call.

It’s a .Net open source project with an Asp Net Core Web API backend.

APIs are developer friendly, they are built to be consumed.

A pet project built as an enterprise platform

Key requirements :

  • Modern enterprise-grade Web API following Microsoft guidelines
  • Handle lots of data with no setup : a single 30m elevation dataset is 500GB with 33,600 tile files. All datasets : 1,7 TB)
  • No tile cache : built a lightweight indexing system that would live in memory
  • Fast generation : a model must be generated live with no offline processing or post-editing
  • Cross platform : must run on PC / Mac / Linux

Technology stack :

.Net Core 3.1 / C#

Cloud is the new Roma : all tech roads lead to it…

  • Almost every .net article is using Azure bits at some point.

Pros :

  • Azure is great for coders very well integrated to dev tools and frameworks
  • Azure offers generous free tiers to develop, deploy and test
  • Azure allows easy supervision and monitoring
  • A lot of boilerplate and bootstrapping is done automatically (logging, injection)
  • Azure PaaS allows splitting up app responsibilities into independent parts

Cons :

  • Costs : going to Standard tiers is expensive : $1000 / month for elevationapi. (I need a huge and fast data storage)
  • Dependency : Not easy to move out or change cloud provider
  • If you want to pay less, you’ll have to tie you code to the cloud, split your architecture into cloud-specific micro-services.

Breaking the rules : the anti-hype is cheaper 

I chose to go the old way : a single dedicated server with muscles : 4 cores, 32GB RAM, 4TB HDD

Cons :

  • You are on you own : server updates, downtimes, hackers, monitoring, … this is all on you
  • You won’t scale to reach the 1M simultaneous user peak
  • Single point of failure

Pros :

  • No external tier dependency
  • Performance boost compared to Azure Standard workers
  • Lots of disk space
  • Total control
  • Dev / Staging / Production slots for the same price
  • Usage cost is load-independent
  • Pricing : around 80$ / month – no extra fees.

Think ahead : do better with less

  • Do better with less: I had to optimize parts of the code for better performance and scalability, monitor usage, use rate limiting on critical paths
  • Do it the “cloud” way : architecture as microservices, think every subsystem as distributable, (shared cache abstraction), easing the pain to a future cloud migration
  • After nearly one year of production, I see how it could move to the cloud from an architect point of view
  • Remove any “magic library” dependencies, add external dependencies with scrutiny
  • Rely on the framework for core tech : dependency injection, logging, profiling. 
  • Externalize any hard coded parameter to a config file
  • No database, no state, SOLID principles

End-to-end development : wearing one hat at a time

Building a full stack platform challenges every stack layer, and helps you code cleaner and faster. But wear only one hat at a time !

  • Set high standards for the front end, asking the best from your API
  • Set high standards for the web API, asking the best from your core library
  • Stress-test the API, find edge cases, be your own client asking for clarity, documentation, unified models
  • Don’t build a front-specific WebAPI, see where it should be in 5 years, build it prone to evolve, imagine what others could built upon it.
  • Web API : don’t trust the caller, give self explanatory messages, hide exceptions details, add a Swagger and build unified responses : models defines your business.
  • RESTful is a state of mind, follow the principles, don’t go too deep.
  • Try to be stateless when possible, keeps code simpler and easier to maintain
  • Build an organized Postman collection
  • Write / rewrite / simplify
  • Log and perf trace critical paths

Open source

Open source was obvious

  • Nothing existed on GitHub, this had to be done, open.
  • Making source code public is challenging : Issues posted, docs, samples, tests.
  • Open source means collaborations and networking

And the fear of predation is real

 

  • A free elevation service suffered from predation and went to a freemium model
  • I decided to advertise on what’s differentiating the API from the others. I serve accurate elevations with nice features, but the real new thing was live 3D terrain models.
  • I used the great AspNetCoreRateLimit from stefanprodan, to avoid predatory usage

Side project means no pressure, no deadline, just fun and tech

  • I shared models on Sketchfab, Twitter, and started to gain attention
  • Many folks asked for awesome features I’d never imagine, sketching step by step the future of the API
  • Talented engineers offered their help and joined the project (triangulation, devops, bug fix)
  • A generous sponsor paid a big feature and the resources needed!
  • Sketchfab published an article on the API exporter

Some Numbers

  • Website https://elevationapi.com started on September 2019
  • 1.6 TB of DEM data hosted at elevationapi.com
  • 22,000 models generated (250 GB)
  • People from 160 countries have tested the API yet
  • 50,000 lines of code
  • SketchFab: 150 followers, 25K views, 800+ likes, 60M triangles, 100+ models (and counting!)

What it does : some visuals

Mycenaean Atlas Project – ElevationAPI – OpenStreetMap and contributors, Imagery: Thunderforest
ElevationAPI – AW3D30 dataset from OpenTopography / JAXA
New York City with OSM Buildings – Data: openstreetmap.com and contributors, imagery : Stamen Design
GPX Elevation profile – Multimapas powered by ElevationAPI
BrGeo.org – Chapada Diamantina ( https://sketchfab.com/3d-models/modelo-digital-do-terreno-chapada-diamantina-25b03dd4f19f46dc903557660670aa17 )
Montagne Sainte Victoire STL model – AW3D30 dataset OpenTopography / JAXA  – 3D printed by Manu Fiancette

Santiago de Chile with the Cerro Plomo barely visible / elevation profile (Source: http://veg-warriors-in-chile.blogspot.com/2016/02/cerro-el-plomo-5434m-asl.html)
Chile-Peru trench, South America, ETOPO1 dataset – this is the dataset used in every physical low scale maps
Bike ride round trip in Provence by JD ( https://sketchfab.com/3d-models/gpx-track-as-3d-plane-path-f4645889161848f4b471cd0a102cfc79 ) 
First result with 50 m TIN: a lot less triangles while the shape aspect is preserved.
160+ countries (Google Analytics)

Espace Killy ski resort (Val d’Isère, Tignes)
Xavier Fischer

Xavier Fischer

Authoring DEM net Elevation API
Software Engineer, specialized in .Net and Geographical Information Systems (GIS), an active open source contributor. (GeoJSON.Net co-owner)

APIdays | Events | News | Intelligence

Attend APIdays conferences

The Worlds leading API Conferences
with 9 Conferences in 2019:

Singapore, Zurich, Helsinki, Amsterdam, San Francisco, Sydney, Barcelona, London, Paris.

Get the API Landscape

The essential 450 companies

Get the API Landscape
Industry Reports

Download our free reports

The State Of Api Documentation: 2017 Edition
  • State of API Documentation
  • The State of Banking APIs
  • GraphQL: all your queries answered
  • APIE Serverless Architecture