Leveraging Generative AI in API Testing

In this article, we explore the transformative power of Generative AI in API testing. As the Executive Director of Platform Engineering at Quest Diagnostics, I lead teams in DevSecOps, SRE, Quality Engineering, and Release Management with an API-first mindset. With over 24 years of experience across the finance, government, and healthcare sectors, I am passionate about APIs and microservices and their role in enhancing integrated customer experiences.

We’ll delve into three key areas:

1. Setting the Stage: An introduction to generative AI, its types, and adoption dimensions.
2. API Product Strategy: A high-level overview of the API lifecycle and how we leverage generative AI in API testing.
3. Summary and Key Takeaways: Early benefits observed and essential insights for implementation.

Setting the Stage: The Emergence of Generative AI

On November 30, 2022, OpenAI released ChatGPT to the public—a general-purpose chatbot running on fine-tuned versions of GPT-3.5. Within days, over a million users signed up, and in just two months, it amassed 100 million users, becoming the fastest-growing internet service ever. Generative AI is no longer a buzzword; it’s a transformative technology reshaping industries.

Understanding the AI Landscape

To comprehend the significance of generative AI, let’s examine how it fits into the broader AI ecosystem:

• Artificial Intelligence (AI): Utilizes computer science and data to enable problem-solving in machines.
• Machine Learning (ML): Techniques and algorithms that allow systems to learn from data, recognize patterns, and make predictions or decisions.
• Deep Learning: An ML technique layering algorithms into artificial neural networks that mimic the human brain.
• Natural Language Processing (NLP): Enables machines to comprehend, generate, and manipulate human language in its written form.
• Generative AI: Built on large language models (LLMs), it produces new content by learning patterns from existing data.

Types of Artificial Intelligence

We often hear about different AI types—Traditional AI, Generative AI, and Causal AI.

• Traditional AI: Focuses on processing and analyzing historical data, following specific rules to perform particular tasks exceptionally well, such as playing chess through pattern recognition.
• Generative AI: An extension of NLP based on neural networks, generating new ideas and content like images, music, and text. It excels in summarizing large unstructured data and creating narratives.
• Causal AI: In early stages of adoption, it predicts why certain outcomes occur by focusing on cause-and-effect relationships. Useful in healthcare for clinical trials and treatment interventions.

Adopting Generative AI: Project Dimensions

When identifying projects suitable for generative AI, we consider three major dimensions:

1. Business Strategy: Align AI, data, technology, and business strategies. Focus on operational stability, efficiency, and customer-centricity. Remember, not every problem requires AI.
2. Culture, People, and Leadership: Upskill the workforce, build trust, foster collaborative cultures, and establish communities of practice.
3. Ethics and Compliance Policies: Emphasize strong AI ethics to responsibly impact people through our products.

Based on these dimensions, we followed four steps in adopting generative AI:

1. AI Governance: Helps in making better business decisions, clarifies legal and compliance policies, and improves customer and employee feedback loops.
2. People and Process: Upskill the workforce with AI training aligned to legal and compliance requirements. Develop robust information architecture and management processes.
3. Identify Use Cases: Focus on repeatable tasks, areas for operational savings, engineering productivity, and customer pain points where generative AI adds value.
4. Choose the Right Model: Decide whether to build our own LLM or leverage existing models. Building our own is expensive and time-consuming, so we experimented with multiple vendor models and started with OpenAI’s ChatGPT model as an MVP.

API Product Strategy

Our API product strategy is influenced by multiple factors:

• Customers: What services do they need? How can we solve their problems?
• Social and Environmental Factors: How are people’s experiences changing? What are the major environmental changes?
• Competitors: How can we increase our market share? What differentiates our product?

Starting with Customer Pain Points

We begin by identifying customer pain points, keeping the customer at the center of everything. Understanding social and environmental challenges helps us build an outside-in thinking approach.

The API Lifecycle

A high-level overview of the API lifecycle includes:

1. API Product Opportunity and Prioritization: Collect opportunities, collaborate with business units, roadmap, assess, prioritize, and secure leadership approvals.
2. API Design: Use an API-first design approach, understand dependencies, and conduct API discovery to determine if we should develop new APIs or leverage existing ones.
3. Development: Decide whether to create, update, or utilize existing APIs. In healthcare, we review for PII (Personally Identifiable Information) and PHI (Protected Health Information), working closely with security, privacy, and legal teams.
4. API Testing: Leverage generative AI to enhance testing efficiency, which we’ll explore in detail.
5. Release: After thorough testing and security reviews, release the APIs, ensuring capacity requirements are met.
6. Consumption and Retirement: Coordinate with consumers, backend systems, and app developers to integrate APIs into mobile apps, web applications, IoT devices, and more.

Leveraging Generative AI in API Testing

Focusing on the API testing phase, we integrate generative AI to enhance efficiency and productivity.

The API Testing Process

1. API Product Specifications: Start with feature stories and acceptance criteria.
2. Select a Large Language Model (LLM): Experimented with multiple vendors and chose GitHub Copilot as a QA persona to help identify test cases.
3. Test Case Generation: AI assists in writing test cases in Gherkin format, preparing test environments, and creating service virtualizations and mock services.
4. Synthetic Data Generation: AI helps generate synthetic data based on API specifications.
5. Test Execution: Execute test cases and conduct defect triage.
6. Documentation: AI aids in creating rich documentation of test steps—a task that often consumed significant time or was neglected.
7. Integration: The process is fully integrated into the GitHub CI/CD pipeline, connected with Jira and Xray for test case management.
8. Monitoring and Iteration: Continuously monitor, write additional test cases, and provide recommendations for improvement.

Benefits Observed

• Operational Efficiency: Significant reduction in the testing lifecycle duration.
• Engineer Productivity: Improved by 30% due to automation and AI assistance.
• Cycle Time Reduction: Faster development and deployment cycles.
• Enhanced Documentation: Rich, detailed documentation aiding future development and maintenance.

Summary and Key Takeaways

Our journey in integrating generative AI into API testing has yielded substantial benefits, including operational efficiencies, productivity gains, and improved documentation.

Key Takeaways

1. Establish AI Governance: Educate teams and leadership on AI security, ethics, and compliance. Incorporate generative AI principles into your API governance model.
2. Identify Use Cases: Start small using a three-dimensional model (business strategy, culture and people, ethics and compliance). Avoid shadow IT; experiment within the organization.
3. Experiment and Iterate: Don’t wait—get started. Fail fast to learn and adapt. Adjust your strategies based on feedback and outcomes.

Conclusion

Generative AI holds immense potential in transforming API testing and development processes. By thoughtfully integrating AI technologies, organizations can achieve significant operational efficiencies, enhance productivity, and deliver superior customer experiences.

Thank you for reading. If you have any questions or would like to discuss further, feel free to connect with me on LinkedIn.