Great ambitions with fewer dollars: DEM Net Elevation API
How I have built an modern API platform, cloudless, servermore with zero budget
Great Digital Elevation Models (DEM) have become public a few years ago. As no open source API nor library existed in the C#/.net field, DEM Net Elevation API was created to fill the gap.
The API produces 3D textured terrain in seconds from a single method call.
It’s a .Net open source project with an Asp Net Core Web API backend.
APIs are developer friendly, they are built to be consumed.
A pet project built as an enterprise platform
Key requirements :
- Modern enterprise-grade Web API following Microsoft guidelines
- Handle lots of data with no setup : a single 30m elevation dataset is 500GB with 33,600 tile files. All datasets : 1,7 TB)
- No tile cache : built a lightweight indexing system that would live in memory
- Fast generation : a model must be generated live with no offline processing or post-editing
- Cross platform : must run on PC / Mac / Linux
Technology stack :
.Net Core 3.1 / C#
Cloud is the new Roma : all tech roads lead to it…
- Almost every .net article is using Azure bits at some point.
- Azure is great for coders very well integrated to dev tools and frameworks
- Azure offers generous free tiers to develop, deploy and test
- Azure allows easy supervision and monitoring
- A lot of boilerplate and bootstrapping is done automatically (logging, injection)
- Azure PaaS allows splitting up app responsibilities into independent parts
- Costs : going to Standard tiers is expensive : $1000 / month for elevationapi. (I need a huge and fast data storage)
- Dependency : Not easy to move out or change cloud provider
- If you want to pay less, you’ll have to tie you code to the cloud, split your architecture into cloud-specific micro-services.
Breaking the rules : the anti-hype is cheaper
I chose to go the old way : a single dedicated server with muscles : 4 cores, 32GB RAM, 4TB HDD
- You are on you own : server updates, downtimes, hackers, monitoring, … this is all on you
- You won’t scale to reach the 1M simultaneous user peak
- Single point of failure
- No external tier dependency
- Performance boost compared to Azure Standard workers
- Lots of disk space
- Total control
- Dev / Staging / Production slots for the same price
- Usage cost is load-independent
- Pricing : around 80$ / month – no extra fees.
Think ahead : do better with less
- Do better with less: I had to optimize parts of the code for better performance and scalability, monitor usage, use rate limiting on critical paths
- Do it the “cloud” way : architecture as microservices, think every subsystem as distributable, (shared cache abstraction), easing the pain to a future cloud migration
- After nearly one year of production, I see how it could move to the cloud from an architect point of view
- Remove any “magic library” dependencies, add external dependencies with scrutiny
- Rely on the framework for core tech : dependency injection, logging, profiling.
- Externalize any hard coded parameter to a config file
- No database, no state, SOLID principles
End-to-end development : wearing one hat at a time
Building a full stack platform challenges every stack layer, and helps you code cleaner and faster. But wear only one hat at a time !
- Set high standards for the front end, asking the best from your API
- Set high standards for the web API, asking the best from your core library
- Stress-test the API, find edge cases, be your own client asking for clarity, documentation, unified models
- Don’t build a front-specific WebAPI, see where it should be in 5 years, build it prone to evolve, imagine what others could built upon it.
- Web API : don’t trust the caller, give self explanatory messages, hide exceptions details, add a Swagger and build unified responses : models defines your business.
- RESTful is a state of mind, follow the principles, don’t go too deep.
- Try to be stateless when possible, keeps code simpler and easier to maintain
- Build an organized Postman collection
- Write / rewrite / simplify
- Log and perf trace critical paths
Open source was obvious
- Nothing existed on GitHub, this had to be done, open.
- Making source code public is challenging : Issues posted, docs, samples, tests.
- Open source means collaborations and networking
And the fear of predation is real
- A free elevation service suffered from predation and went to a freemium model
- I decided to advertise on what’s differentiating the API from the others. I serve accurate elevations with nice features, but the real new thing was live 3D terrain models.
- I used the great AspNetCoreRateLimit from stefanprodan, to avoid predatory usage
Side project means no pressure, no deadline, just fun and tech
- I shared models on Sketchfab, Twitter, and started to gain attention
- Many folks asked for awesome features I’d never imagine, sketching step by step the future of the API
- Talented engineers offered their help and joined the project (triangulation, devops, bug fix)
- A generous sponsor paid a big feature and the resources needed!
- Sketchfab published an article on the API exporter
- Website https://elevationapi.com started on September 2019
- 1.6 TB of DEM data hosted at elevationapi.com
- 22,000 models generated (250 GB)
- People from 160 countries have tested the API yet
- 50,000 lines of code
- SketchFab: 150 followers, 25K views, 800+ likes, 60M triangles, 100+ models (and counting!)
What it does : some visuals