In today’s evolving software landscape, the demand for efficient and scalable API design is more critical than ever. Declarative clients are gaining traction for their ability to simplify complex state management, offering a streamlined approach to interacting with APIs. This article delves into the principles of designing declarative-friendly APIs, exploring best practices, common pitfalls, and actionable patterns to enhance API usability.
About the Author
I am Yusuke Tsutsumi, formerly a Manager of the Terraform team at Google Cloud Platform (GCP) and later an API Design Tech Lead. My experience with integrating and managing APIs has shaped my understanding of declarative client needs and API design patterns. The insights shared here are drawn from my professional journey, aiming to help developers create more robust and user-friendly APIs.
Understanding Declarative Clients
Before diving into design patterns, it’s essential to differentiate between declarative and imperative clients.
- Imperative Clients: These clients require explicit instructions for each operation. For example, to ensure a virtual machine (VM) exists, you might use an SDK to get the VM instance, check if it exists, and then create or update it accordingly. This approach demands that developers manage the workflow logic and state transitions explicitly.
- Declarative Clients: In contrast, declarative clients allow developers to specify the desired state of the system without detailing the steps to achieve it. The client or underlying system handles the reconciliation process to align the current state with the desired state. This model simplifies complex state management, especially in environments with intricate dependencies like cloud infrastructures.
Examples of Declarative Clients
Several tools exemplify the declarative approach:
- Terraform: Allows users to define infrastructure as code, specifying resources and configurations declaratively.
- Pulumi and AWS CDK: Enable developers to write code in familiar programming languages to define and manage cloud infrastructure declaratively.
- Kubernetes: Manages containerized applications and services declaratively, ensuring that the cluster’s state matches the desired configurations.
These tools are prevalent in cloud environments due to the complexity of orchestrating multiple resources with interdependencies, such as VMs, storage, networking, and more.
The Reconciliation Loop
At the heart of declarative clients is the reconciliation loop:
- Define Desired State: The developer specifies the desired configuration for a resource.
- Retrieve Current State: The client gets the current state of the resource from the API.
- Compare States: The client performs a diff to identify discrepancies between the desired and current states.
- Apply Changes: If differences exist, the client creates, updates, or deletes resources to reconcile the state.
- Verification: The client retrieves the resource again to confirm that the desired state has been achieved.
This loop continues until the current state matches the desired state, ensuring consistency and correctness in the system.
Why Declarative-Friendly APIs Matter
Even if you don’t currently use declarative clients, designing APIs that are declarative-friendly offers broad benefits:
- Consistency: Promotes uniform behavior across different clients and integrations.
- Ease of Integration: Reduces the need for custom logic, making it easier for clients to interact with your API.
- Future-Proofing: As declarative tools become more widespread, having APIs that accommodate them can save significant rework and engineering time.
- Improved User Experience: Leads to better usability, reducing errors and the need for extensive documentation or support.
Design Patterns for Declarative-Friendly APIs
- Resource-Oriented Design (RESTful APIs)
APIs should expose resources that can be manipulated through standard HTTP methods like GET, POST, PUT, DELETE, and LIST. Consistency in these operations is crucial. Resources should have unique identifiers, and their relationships should be non-cyclic to avoid complex state management. - Handling Imperative Operations
Not all actions fit neatly into CRUD operations. For operations like restarting a VM or backing up a database, it’s acceptable to have dedicated methods rather than forcing them into the resource model. This separation maintains clarity and prevents complicating the declarative model. - Ensuring Strong Consistency
APIs should be strongly consistent, meaning that once a request completes, subsequent requests reflect the changes immediately. This consistency is vital for the reconciliation loop, as clients rely on accurate and up-to-date information to determine the next steps.
Dealing with eventual consistency requires strategies like:- Adding parameters to enforce strong consistency.
- Implementing consensus algorithms on the server side.
- Handling retries and delays server-side to abstract complexity from clients.
- User-Settable Identifiers
Allowing users to set resource identifiers simplifies resource management in declarative clients. Server-generated IDs can hinder the reconciliation process, as clients may not know how to reference resources created previously.
If server-generated IDs are necessary, consider:- Namespace IDs: Distinguish between user-settable and server-generated IDs, perhaps by prefixing server-generated IDs with a specific character like an underscore.
- Alternative Patterns: Implement methods that allow clients to reference resources without needing the exact ID upfront.
- Avoiding Mixed Ownership of Fields
Mixing user-managed and server-managed data in the same field can lead to conflicts and infinite loops in the reconciliation process. For instance, if both the client and server modify the same labels field, they might continually overwrite each other’s changes.
To mitigate this:- Duplex Model: Use separate fields for user-managed and server-managed data. For example, have an inputLabels for user input and outputLabels for server-generated data.
- Data Type Annotation: Introduce data types or annotations that inform clients about the nature of each field, indicating whether it’s user-managed, server-managed, or both.
- Avoid Hard-Coding Logic: Refrain from embedding special handling for specific fields in client logic, as it doesn’t scale and complicates maintenance.
Understanding the Reconciliation Loop Pitfalls
An infinite loop in the reconciliation process occurs when the client and server repeatedly overwrite each other’s changes due to unclear ownership of fields. This situation often arises from:
- Input Sanitization: The server modifies input data in unexpected ways.
- Default Values: Mismatches in default values between client expectations and server implementations.
- Data Type Mutations: Changes in data types or formats that aren’t recognized as equivalent by the client.
By clearly defining field ownership and behaviors, these issues can be avoided, ensuring a smooth reconciliation process.
An Ideal Declarative-Friendly API: Kubernetes Custom Resources
While no API is perfect, Kubernetes Custom Resource Definitions (CRDs) come close. Kubernetes exposes a RESTful API with consistent methods, and CRDs allow users to define their own resource types. Tools like kubectl can interact with any resource type because of the standardized API structure.
This level of standardization fosters a rich ecosystem of clients and tools that can manage resources declaratively, illustrating the benefits of designing APIs with declarative clients in mind.
Moving Forward: The AEP Project
Building on these principles, we are working on the AEP (API Engineering Practices) Project—an open-source initiative aimed at creating an API specification that incorporates these best practices. Unlike OpenAPI, which describes APIs as they are, AEP focuses on ensuring consistent semantics around CRUD operations, facilitating the generation of a diverse ecosystem of clients.
Contributors from organizations like Google, Roblox, IBM, Microsoft, and Netflix are collaborating on this project. We invite developers and API designers to join us, contribute, and help shape the future of declarative-friendly APIs.
Conclusion
Designing APIs that are declarative-friendly is not just about accommodating a particular client model; it’s about creating consistent, user-friendly interfaces that simplify development and integration. By adopting the patterns discussed—resource-oriented design, strong consistency, user-settable identifiers, and clear ownership of fields—you can enhance the usability of your APIs and prepare them for the evolving landscape of software development.
We encourage you to apply these patterns in your API designs, provide feedback, and participate in the ongoing conversation about API best practices.
