National Australia Bank (NAB) is Australia’s largest business bank. We work with small, medium, and large businesses. We have more than 30,000 people serving 9 million customers in Australia, New Zealand, and worldwide. We will focus on technical aspects of API and event discovery and share our experience and findings using async API and open API spec for search and discovery.
API and Event Ecosystem at NAB
For the last couple of years, we have been working on building capability throughout our organization and services as APIs and events.
We have many business domains, as shown in the diagram. All these business domains need to provide the capability to our customers, suppliers, regulators, or partners. They do that through APIs built on top of our business services, typically built in Spring, Spring Boot, and Java. We use our APIs for queries and commands to ensure that business capability. For significant state changes, we use Kafka events and wrap up all that with Identity Management.
For specific development for a particular experience, we use Node and Graph QL to build on top of our core business services.
It can get hard to understand further development with all these business domains, APIs, and events. The pointers below will help you understand this better.
Understanding the capability that’s available within our tech platforms is helpful when putting together a business case. Most of them are surfaced through APIs and events.
Understanding what’s there today or what’s in the pipeline is quite valuable when we want to chase an opportunity.
Once a business case has been created, it’s time to build. The API is built using an API developer portal. After the API is built and in production, when there is an issue, it’s pretty handy to be able to search for the interface that is causing the problem. Typically, there’s a URI, some error codes, that help the developers determine where the error is to analyze and fix. Troubleshooting at field and error levels is quite useful before getting to the next stage.
For improved security, it is imperative to understand the risk position that you’re at. It is essential to understand the interface, look at the data that’s been exposed, and then take appropriate actions.
Using Open API and Async API to drive discovery of APIs and events
At NAB, we use our internal development portal. This portal has metadata that is required to drive API discovery for the users. We have used various options to search for API resources. If you’re searching for a customer or an account, you can search by keyword. You can further dive down and filter and drill down precisely to what you want. We have a couple of ways to do that: Namespace (business-domain), category, gateway, API Status (developed, under development, planned), environment, and domain. Internally, this helps us find what we know we’re looking for.
Search and Discovery using OPENAPI and REST
When we look at Open API, you typically have to fill up the API title field. I want to highlight that when you have hundreds or thousands of specifications, you need to start having strong naming standards for consistency because we use the API title to drive our searches on keywords. We model the title around resources, so resource-orientated titles are better than having an API spec per operation basis.
Some people don’t know that you can use annotations in an open API spec. We wanted to be able to filter and intake by a category, no matter which business user is exposing this API. So being able to put our tags and annotations for our needs is quite helpful. So definitely use them when and where applicable. Inside an API spec, you can have multiple operations. We like to have an API spec around a resource. So, when we use our API products for our open banking, we have a spec around a resource with multiple operations to help the user discover related APIs.
Once we search for a product API on our internal developer portal, I may need to do something with the products. I may need to get some details, fields, and methods available. We found that grouping related operations together with the resource helps drive the discovery of where you want to go and how to use it. It helps create better products and experiences, and outcomes for our business.
We have stated that we can have multiple operations per spec, but it is interesting to know how many operations you have per spec. We know that humans have a limit on the number of operations they can comprehend at a given time. So, we’ve put a limit on about ten operations per spec.
Now, let’s jump into the next level of detail. We have API resources, operations, and products within the API spec. But it’s all about the data, so you need to document your fields very well. You need to show how to use them; you need to explain their format; you need to show examples. And it needs to be consistent across all your specs so that your users have the trust to follow and find what they need.
Async API and events
We had quite a bit of success with API search and discovery. While doing that, we realized that we have a lot of events, and we want to expose to our business and our engineers to be able to build things and find things in the same way. In a similar vein, we know that different business domains produce events, and we want to manage them in the same type of lifecycle. But it is slightly different. Open API specs will not work for events.
You can search by keywords using maybe a resource, namespace, business domain, or category. But the way to drill down is different. For this, we use Kafka. We use various Kafka brokers, whether you’re running for an enterprise event broker versus a more specialized environment, whether it’s intended to be used outside of the domain or within. We use a basic API for this, but with some slight differences.
We found that just having a resource title isn’t enough because it is a real estate change. We’d like to have the title around the resource and the type of change. It’s a different way of modeling, a different way of putting this documentation together. It’s just another way of managing and editing. Annotations and Async API also need documentation for your customizations. When we were designing this, we were not sure whether to use the “subscribe” or “publish” options within the channels object. We decided on “subscribe.” “Subscribe” helps us describe what you need to do to consume this event. The last part of Async API, which was critical for us, was AVRO, which supports a native array. We use AVRO to serialize our data within our Kafka event streams. So, maintaining the one schema standard makes a lot of sense; it removes any chance of errors.
Importance of data quality
Bad data equals bad results. It makes your API specs useless. So, you know, these are the four things in terms of data quality that I encourage you to improve, to put in controls, to manage your accuracy –
- Accuracy – Make sure that what’s documented matches the implementation.
- Completeness – Make sure that your specs are complete.
- Currency – Make sure it is up to date with the latest release. Having some field names without examples is not complete; it’s not good enough; it has to be current and match what’s actually in production.
- Consistency – All specs must have the same level of quality.
- Open API Spec – Perfect for REST API Search and Discovery.
- Don’t overload your specs
- One RESTful resource per spec
- Multiple related operations within one spec (Maximum of 10)
- Async API – Great for Kafka Event Stream and Discovery
- Native AVRO support for schema consistency
- Consider the use of “Channels” object (Subscribe vs. Publish)
- Data Quality – Data quality control is key
- Put in controls to manage accuracy, completeness, currency, and consistency
- Test and Pivot
To summarize, understand your API search and discovery use cases within the scope of your ecosystem. Every company has a different ecosystem, so know what you’re going after. If you do need to customize, use annotations. Don’t go overboard. Focus on data quality and consistency. Finally, you want to build trust with your users and user bases.