How I have built an modern API platform, cloudless, servermore with zero budget

The project

Great Digital Elevation Models (DEM) have become public a few years ago. As no open source API nor library existed in the C#/.net field, DEM Net Elevation API was created to fill the gap.

The API produces 3D textured terrain in seconds from a single method call.

It’s a .Net open source project with an Asp Net Core Web API backend.

APIs are developer friendly, they are built to be consumed.

A pet project built as an enterprise platform

Key requirements :

• Modern enterprise-grade Web API following Microsoft guidelines
• Handle lots of data with no setup : a single 30m elevation dataset is 500GB with 33,600 tile files. All datasets : 1,7 TB)
• No tile cache : built a lightweight indexing system that would live in memory
• Fast generation : a model must be generated live with no offline processing or post-editing
• Cross platform : must run on PC / Mac / Linux

Technology stack :

.Net Core 3.1 / C#

Cloud is the new Roma : all tech roads lead to it…

• Almost every .net article is using Azure bits at some point.

Pros :

• Azure is great for coders very well integrated to dev tools and frameworks
• Azure offers generous free tiers to develop, deploy and test
• Azure allows easy supervision and monitoring
• A lot of boilerplate and bootstrapping is done automatically (logging, injection)
• Azure PaaS allows splitting up app responsibilities into independent parts

Cons :

• Costs : going to Standard tiers is expensive : $1000 / month for elevationapi. (I need a huge and fast data storage)
• Dependency : Not easy to move out or change cloud provider
• If you want to pay less, you’ll have to tie you code to the cloud, split your architecture into cloud-specific micro-services.

Breaking the rules : the anti-hype is cheaper 

I chose to go the old way : a single dedicated server with muscles : 4 cores, 32GB RAM, 4TB HDD

Cons :

• You are on you own : server updates, downtimes, hackers, monitoring, … this is all on you
• You won’t scale to reach the 1M simultaneous user peak
• Single point of failure

Pros :

• No external tier dependency
• Performance boost compared to Azure Standard workers
• Lots of disk space
• Total control
• Dev / Staging / Production slots for the same price
• Usage cost is load-independent
• Pricing : around 80$ / month – no extra fees.

Think ahead : do better with less

• Do better with less: I had to optimize parts of the code for better performance and scalability, monitor usage, use rate limiting on critical paths
• Do it the “cloud” way : architecture as microservices, think every subsystem as distributable, (shared cache abstraction), easing the pain to a future cloud migration
• After nearly one year of production, I see how it could move to the cloud from an architect point of view

• Remove any “magic library” dependencies, add external dependencies with scrutiny
• Rely on the framework for core tech : dependency injection, logging, profiling. 
• Externalize any hard coded parameter to a config file
• No database, no state, SOLID principles

End-to-end development : wearing one hat at a time

Building a full stack platform challenges every stack layer, and helps you code cleaner and faster. But wear only one hat at a time !

• Set high standards for the front end, asking the best from your API
• Set high standards for the web API, asking the best from your core library
• Stress-test the API, find edge cases, be your own client asking for clarity, documentation, unified models
• Don’t build a front-specific WebAPI, see where it should be in 5 years, build it prone to evolve, imagine what others could built upon it.
• Web API : don’t trust the caller, give self explanatory messages, hide exceptions details, add a Swagger and build unified responses : models defines your business.
• RESTful is a state of mind, follow the principles, don’t go too deep.
• Try to be stateless when possible, keeps code simpler and easier to maintain
• Build an organized Postman collection
• Write / rewrite / simplify
• Log and perf trace critical paths

Open source

Open source was obvious

• Nothing existed on GitHub, this had to be done, open.
• Making source code public is challenging : Issues posted, docs, samples, tests.
• Open source means collaborations and networking

And the fear of predation is real

• A free elevation service suffered from predation and went to a freemium model
• I decided to advertise on what’s differentiating the API from the others. I serve accurate elevations with nice features, but the real new thing was live 3D terrain models.
• I used the great AspNetCoreRateLimit from stefanprodan, to avoid predatory usage

Side project means no pressure, no deadline, just fun and tech

• I shared models on Sketchfab, Twitter, and started to gain attention
• Many folks asked for awesome features I’d never imagine, sketching step by step the future of the API
• Talented engineers offered their help and joined the project (triangulation, devops, bug fix)
• A generous sponsor paid a big feature and the resources needed!
• Sketchfab published an article on the API exporter

Some Numbers

• Website https://elevationapi.com started on September 2019
• 1.6 TB of DEM data hosted at elevationapi.com
• 22,000 models generated (250 GB)
• People from 160 countries have tested the API yet
• 50,000 lines of code
• SketchFab: 150 followers, 25K views, 800+ likes, 60M triangles, 100+ models (and counting!)

What it does : some visuals