API Lifecycle Management

Data Abstraction using APIs

114views

This is an article by Vikram Vasudevan, founder of Ekahaa solutions. They specialize in data abstraction solutions.

When it comes to API, there has always been a lot of focus on API management, monetization, performance monitoring, caching, and things like that. But few people talk about a key, possibly underrated aspect of APIs – Abstraction in APIs. When you build APIs, you connect to different types of data sources, and it has its own share of complexity. Abstraction, in general, is hiding the complexities of the underlying sub-system. In the case of APIs, there are many different levels of abstraction.

What is Abstraction?

In computing terms, abstraction is a way of hiding the working details of a sub-system. For example, when you see a restaurant, you look at the ambiance and want to eat there. You do not see the complexity of how it was built or is run.

In a database context, we create views, procedures, functions, etc. It abstracts the complexity of the underlying data model. And for somebody who’s using the View, they don’t see what it takes to build that View. That in itself is a big abstraction at a database level. API itself is a big abstraction.

You have a beautiful web application. What you and the users see is the front end. Behind that is a large infrastructure, micro-services, gateways, etc. It is a highly complex system.

After the advent of the cloud, the cloud tools you use are another form of abstraction. When you ask to deply an application to GCP cloud, it spawns your cloud computing system, creates your databases, sets up your networking, etc. This is infrastructure abstraction.

Key principles of abstraction

DRY – Don’t Repeat Yourself – Eliminate repetition of software patterns. Redundancies are okay, as far as performance enhancements go, but don’t repeat your code everywhere as maintenance and technical debt will build up.

WET – Write Everything Twice – only if that’s the best way to solve your problem.

AHA – Avoid Hasty Abstractions – sometimes the best way is to not abstract at all.

MASK – End users do not need to know the complexities. Hide it from them.

Abstraction of Datasources and APIs

In today’s world, there are many different types of data sources. You have the SQL data sources, legacy systems, no SQL data sources, the graph DPS, the influx time series databases, ERPs, etc. So, having a system that can effectively talk to all of these data sources without understanding the underlying semantics would be a big challenge. This is one of the key challenges regarding data source abstraction. 

When you build a product, you have to talk to many different types of data sources. So there’s a need to interface with data sources. Every product has its way of dealing with this. Essentially, it boils down to four aspects of data source abstraction. 

Four aspects of Datasource Abstraction

Unification – you need to unify your data by bringing multiple data sources together. You must provide a singular language through which you can talk to all of your different types of data sources.

Fortification – You must fortify your data sources by wrapping them in a governance layer. You must govern your data sources through a single data access layer with policy-driven access control at API, row, and field levels.

Restification – Provide a Restful API interface with the flexibility of GraphQL. Perform CRUD operations on data sources.

Democratization – Data is useful only if it is shared. So you need to democratize your data. Propagate data securely across disparate systems seamlessly based on an event, a schedule, or adhoc.

API Building Lifecycle

The API building lifecycle is very complex. The steps are as follows –

  1. Identify a data source
  2. Setup API infrastructure
  3. Write code to define API
  4. Write code to define security policies
  5. Use SDKs to inject code to extract documentation.
  6. Create separate repositories for code versioning.
  7. Plan and perform testing of APIs
  8. Deploy the APIs

Summary

An API-Driven Data Abstraction Platform needs to support API building, not just defining or documenting APIs.

API building on the fly is a key ingredient of API driven data abstraction platform. It has to be able to connect to major data sources that are in use today. At the same time, it has to keep up with newer data sources that keep getting spawned daily. We must keep up to date on the developments in the data world. This is where the community-driven character model comes in very handy. Auto-discovery of resources from databases is important. The availability of customization is important for advanced users. So whenever you make changes, you need to ensure backward compatibility so that existing API consumers don’t have a negative impact. 

Always have SSL support and a multi-tiered security model.

API Management is necessary to ensure that APIs are not scattered. You can either build your solutions or use an available product.

And last but not least are data management and governance capabilities

Benefits of API-Driven Data Abstraction

  • Faster API Releases – Rapid API development and testing allow frequent releases to your customers.
  • TCO Reduction – Massive savings in API development and infra costs
  • Operational Efficiency – Manage data democratization and governance and all your data pipeline activities using one platform.
Vikram Vasudevan
Vikram Vasudevan is a software technology aficionado, a deeply hands-on architect with over two decades of experience designing and building applications. He strongly believes his growth is inclusive of the growth of the team around him. He is a tool-agnostic creator, an innovator and a pioneer. An astute learner and a staunch believer in learning on the job. Vikram's broad technology spectrum ranges all the way from building highly performant & highly available OLTP applications to operational datastores & datawarehouses in one band, from data quality to process automation in another. From test automation to devops in yet another band. Vikram is an accomplished cloud architect, well-versed with major cloud providers such as Microsoft Azure, Google Cloud Platform and Amazon Web Services. Above all, Vikram likes to leave his mark on this world as a good human being, a down-to-earth person, someone who respects everyone else around him equally regardless of their position/stature. Vikram's passion to build stuff eventually paved the way for the launch of his first entrepreneurial venture "Ekahaa Solutions" in 2020. Vikram is the proud creator of Abstracta™, a data abstraction platform. Vikram expects to work in an environment that is politics-free, unrestricted and one that fully supports and backs his innovative instincts, trusts him and provides him the freedom to research and explore, provides him a platform for showcasing his creativity, with the understanding that he will do his very best to reduce costs to the company and pioneer them to the next level of technological maturity.

APIdays | Events | News | Intelligence

Attend APIdays conferences

The Worlds leading API Conferences:

Singapore, Zurich, Helsinki, Amsterdam, San Francisco, Sydney, Barcelona, London, Paris.

Get the API Landscape

The essential 1,000+ companies

Get the API Landscape
Industry Reports

Download our free reports

The State Of Api Documentation: 2017 Edition
  • State of API Documentation
  • The State of Banking APIs
  • GraphQL: all your queries answered
  • APIE Serverless Architecture