Introducing API-tests in an existing application

Allgemein

In the last few days I added API testing to a symfony application. In essence it was a rather straight forward experience but I stumbled over a few things and perhaps it can help you solve issues faster.

Preconditions

Currently I am working on an SaaS platform and last year we decided to extend our API documentation. We already have an API but the documentation so far was not really standardized up to that point so one of the things that we added was an OpenAPI specification.

That spec isn’T hand-coded but auto-created via the Nelmio OpenAPI package. On every merge to master the CI pipeline creates the documentation and commits that to another git repo so that it then is auto-updated at 2 third-party services so that users can see it.

What we encountered though in this last year was that every now and then our output actually changed due to some refactoring without us noticing and the OpenAPI spec actually wasn’t consistent with the actual API.

And nothing is worse than incorrect docs. Then rather no docs.

So I was thinking about how we can make sure that the docs resemble the actual API and how we can make sure that we will notice inconsistencies between the two.

One option would be to actually check each and every request and response that comes into and leaves the application. But on the production system that is far to processor and therefore cost-intense and – probably much more important – far too late in the process.

So I decided to instead add the API testing to our integration tests.

In those tests we use our own implementation of Symfonys KernelBrowser so it is easy to hook into the request() and the getResponse() method to first test the request and the response before then passing them on to do all the remaining, already existing, tests.

For the time being I only test whether the response matches. But that is mainly because the requests are handcrafted and are missing some fields or headers and that is a lot of manual work to fix. It’s planned but…

I found a pretty good tutorial from Ruben Rubio which I used as the groundwork.

For testing requests against an OpenAPI schema there is a package from The League of extraordinary PHP Packages so I decided to use that. Drawback: It is based on PSR7/15 Requests/Response, so I also need the symfony/psr-http-message-bridge to create a PSR7 response from a symfony response. And then integrate that into the KernelBrowser.

So the first thing I did was running

composer require –dev symfony/psr-http-message-bridge league/openapi-psr7-validator nyholm/psr7

Then I creates a class to do the heavy lifting. 

<?php

declare(strict_types=1);

namespace App\Validation\OpenAPI;

use League\OpenAPIValidation\PSR7\Exception\NoPath;
use League\OpenAPIValidation\PSR7\Exception\NoResponseCode;
use League\OpenAPIValidation\PSR7\Exception\Validation\AddressValidationFailed;
use League\OpenAPIValidation\PSR7\Exception\ValidationFailed;
use League\OpenAPIValidation\PSR7\OperationAddress;
use League\OpenAPIValidation\PSR7\ResponseValidator;
use League\OpenAPIValidation\PSR7\ValidatorBuilder;
use Nyholm\Psr7\Factory\Psr17Factory;
use Symfony\Bridge\PsrHttpMessage\Factory\PsrHttpFactory;
use Symfony\Component\HttpFoundation\Response;

final class OpenApiResponseAssert
{
    private PsrHttpFactory $psrHttpFactory;
    private ResponseValidator $validator;

    private static string $openApiSpecLocation = __DIR__ . '/path/to/openapi.yaml';

    public function __construct()
    {
        if (! file_exists(self::$openApiSpecLocation)) {
            throw new \RuntimeException(sprintf(
                'OpenApi spec file %s does not exist.',
                self::$openApiSpecLocation
            ));
        }
        $psr17Factory = new Psr17Factory();
        $this->psrHttpFactory = new PsrHttpFactory($psr17Factory, $psr17Factory, $psr17Factory, $psr17Factory);
        $this->validator = (new ValidatorBuilder())
            ->fromYamlFile(self::$openApiSpecLocation)
            ->getResponseValidator();
    }

    /** @throws ValidationFailed */
    public function __invoke(Response $response, string $route, string $method): void
    {
        if (null === $this->openApiResponseAssert) {
            return $this->client->getResponse();
        }

        $request = $this->client->getRequest();
        $response = $this->client->getResponse();

        if ($request->get('_route') === null) {
            return $response;
        }

        $methodInfo = ReflectionMethod::createFromMethodName($request->get('_controller'));
        if ($methodInfo->getAttributes(\OpenApi\Attributes\Response::class) === []) {
            return $response;
        }

        $route = $this->router->getRouteCollection()->get($request->get('_route'));


        ($this->openApiResponseAssert)($response, $route->getPath(), $request->getMethod());

        return $response;
    }
}

The probably most interesting part is that in this classes __invoke() method I create a PSR7 message from the Symfony message and then pass that to the validator.

To be failsafe we just ignore when the OpenAPI spec can’t be read but we will print out a warning when the tests run.

Also we check whether the route is set as attribute in the request. If not, then this is not something that we should check.

And then we check whether the attributes of the controller method actually contain an OpenAPI Response declaration. If that is not the case, then that endpoint is not documented to be part of the OpenAPI spec and we will most likely run into a path error. In such a case we just return the response without checking it first.

THe same effect has ignoring exceptions that might get thrown due to the invoked path not being available in the OpenAPI spec. The reasoning is that the application has much more paths than we have (and want) documented in the OpenAPI spec. So when we run a test that calls an endpoint that is not part of the spec I just want to ignore that the API validation doesn’t work. This approach was taken b efore using the more sophisticated (and complex) take with checking the Attributes. I could probably remove that now.

As the NoResponseCode extends the NoPath exception I had to catch that before to make sure that cases are reported where the path is available in the API but the encountered response-code is not documented in the API. We had a lot of 401 and 403 cases documented but the 400 and 404s (and some others) weren’t documented. Now they are 😀

Then I needed to get that invoked every time we call getRequest() in our tests. For that I extended our implementation of the KernelBrowser like this

<?php declare(strict_types=1);

namespace App;

use App\Validation\OpenAPI\OpenApiResponseAssert;

final class KernelBrowser
{
    private OpenApiResponseAssert|null $openApiResponseAssert = null;

    public function __construct() 
    {
        try {
            $this->openApiResponseAssert = new OpenApiResponseAssert();
        }  catch(Throwable $e) {
            trigger_error(sprintf(
                'API contract testing is not possible. %s' . PHP_EOL,
                $e->getMessage()
            ), E_USER_WARNING);
        }
    }

    public function getResponse(): object
    {
        $request = $this->client->getRequest();
        $response = $this->client->getResponse();
        if ($request->get('_route') === null) {
            return $response;
        }

        $route = $this->router->getRouteCollection()->get($request->get('_route'));

        ($this->openApiResponseAssert)($response, $route->getPath(), $request->getMethod());

        return $response;
    }
}

This class tries to create an OpenAPIResponseAssertion. Should that fail due to the specs not being available we just ignore that, generate a warning and continue.

And whenever a request is fethced from the class via the getRequestmethod we invoke the Assert-class should it exist. If not, we just return the response.

So far so good. This worked like a charm almost instantly and I could then check where Reality and the Docs differed. In this case I most of the time adapted the docs, sometimes though also the actual API. But that was a case by case decission.

Challenges

During that check though I encountered several oddities that presented real challenges where a solution needed some extra thought.

The main issue here is that the docs need to match an API that was introduced over time and without a formalized process.

Optional properties

One of the challenges for example was that we return an object that contains one key where the value can be one of several different objects. The perfect case for the oneOf operator. Until the tests failed because the object matched several of the possible options but it is only allowed to match one. What I didn’t realize at first is that OpenAPI allows additional properties. So when you have 2 objects where A “extends” B then every entity that matches B also matches A.

The solution here was to set additionalProperties on each of the objects definitions to false.

Empty Object

And then I stumbled upon one case where these different objects also included a totally empty object. Fine, declaring an empty object in OpenAPI is possible and setting the additionalProperties to false should do the trick, shouldn’t it?

It didn’t.

The issue here is that the underlying library doesn’t take the additionalProperties into account when checking the empty objet. So I “tricked” the system by adding an optional property named '' (yes! An empty string!) to the object and it worked.

Different DateFormats

OpenAPI supports a date-time format. Fine. We do return some datetime-information, so it seemed the perfect fit. Until I reallized it wasn’t.

OPenAPIs date-time format requires an offset to be present in the string. But we in some cases return a datetime string without an offset. So I had to create an extra format for that in our nelmio-open-api config-file:

nelmio_api_doc:
   documentation:
      components: 
         schemas:
              datetime:
                type: string
                pattern: "[0-9]{4}-[01][0-9]-[0-3][0-9]T[0-2][0-9]:[0-6][0-9]:[0-6][0-9]"

I also had to create another schema as in one case we return either a datetime string or a timestamp-integer, depending on a passed parameter. I solved that similarily in the config file:

nelmio_api_doc:
   documentation:
      components: 
         schemas:
              datetime_or_timestamp:
                oneOf:
                  - type: string
                    format: datetime
                  - type: integer
              date_time_or_timestamp:
                oneOf:
                  - type: string
                    format: date_time
                  - type: integer

Array or string

And in one case we return either a string or an array of strings – again depending on a passed parameter. I tried to solve this within the documentation of the object that this is part of but it turned out that this is not possible that way. I had to move that into the config file as well.

nelmio_api_doc:
   documentation:
      components: 
         schemas:
              message:
                example: '\u003Cp\u003EHi Mark, I need a projector for my presentation. Thanks\u003C\/p\u003E'
                oneOf:
                  - type: string
                  - properties:
                      markdown:
                        type: string
                      html:
                        type: string
                    type: object
                    additionalProperties: false
                additionalProperties: false

Here the message property can either be a string or an object containing the keys ‘html’ and ‘markdown’. but no other keys. And it has to be either of the two. No other information is possible.

Fin

With these settings and obscure schema I now have a validation of all responsens up and running while running the test suites. And should we for whatever reason change the API or the docs unintended we will get an immediate feedback as the tests break.

If you have any suggestions of feedback, feel free to add a comment or drop me a line.

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Find out more about Webmentions.)