Introduction

A Solid ecosystem can only thrive if different servers and applications can co-exist, and the way Solid achieves this is through open specifications. To guarantee that implementations interoperate as intended, we need rigorous tests that follow the spec closely. In a previous blog post we shared principles for a test suite that maximally supports developers, and thereby the entire ecosystem.

In partnership with Solid server developers, we’ve been building and testing an open source implementation of that design. With the completion of version 1.0 of this open source Conformance Test Harness (CTH), we aim to continue our support for the entire Solid community.

In this post, we will describe some of the key features of the CTH, how developers can take advantage of it, and how we as a community can together grow this test suite as Solid continues to evolve.

Release 1.0

Readable Tests

The Solid community consists of diverse developers, and everyone should be able to learn from the tests and contribute to them. This is why we separated the description of the tests from the framework that executes them. The result: tests that are easy to read and understand. If a test fails, you immediately know what went wrong.

Figure 1: An example of a test scenario

You can find many more examples of test scenarios in the repository.

Linking Tests to Specification

Whenever a test fails, developers want to understand why it should have worked. They should not get lost in test code or endless specification discussions. This is why the tests closely follow the structure of the specification document. Specification sections directly link to the individual Karate files that test the requirement via a test manifest, such that it is clear which tests cover what parts.

Figure 2: An example of a test manifest linking the test and the spec

Figure 3: The relationship between the test suite, test manifests (test suite descriptions) and specifications

Optional Features

Solid implementers can choose whether or not they implement certain optional parts of the specification. That’s part of what makes every implementation unique—yet interoperable! The tests support different servers by letting developers decide what optional tests should pass. At present, they do this by listing supported features in the test subject description file.

Approved Tests

A test suite is only as good as the quality of its individual tests. This is why the Solid Editors independently review approve each test to confirm it correctly applies the specification as written. The review status of each of the tests - unreviewed, accepted, assigned, approved, rejected, on hold - is maintained and published in the Test Manifest.

Reporting

Detailed reports provide developers with a confident path toward compatibility. The usability and readability of the dynamically generated reports of the test results is improved to highlight failed tests and the reasons for those failures:

Figure 4: High-level test results show developers how tests relate to the spec

Figure 5: Detailed test results help developers analyze client-server interactions

Authorization

Authorization tests can be run against servers implementing either Web Access Controls (WAC) or Access Control Policies (ACP) due to a universal access control layer (provided by CTH) which translates the core authorization controls to work with both implementations.

Dependencies

CTH runs from a Docker image so there are no other dependencies to worry about. This makes it easy to use in a variety of environments, whether testing a Solid server implementation, developing new tests, or incorporating it into CI workflows. Various options are described alongside the Docker image here.

Open Source

The CTH codebase and the implemented tests are being openly developed on GitHub. The repositories contain instructions for executing the CTH against several different Solid servers, and provide examples of how to write and submit tests.

Immediate Value

Solid implementers have already used pre-releases of CTH v1.0 to execute tests against their server implementations. They have responded very positively to the generated CTH reports, finding it to be a valuable tool which is helping them to improve the conformance of their servers to the Solid specification. CTH has also enabled collaborative conversations between all stakeholders, including server implementers, test developers, and specification authors, and supports the interoperability of servers across the Solid ecosystem.

“The [CTH] helps us make sure that the Community Solid Server adheres to the specification. Given the evolving nature of the Solid requirements, it informs us when changes are needed. We worked together with Pete to set it up in our CI environment, causing us to uncover some cases where the [CTH] was too strict. We’re happy with this bidirectional flow of support between servers and test suite.”

- Joachim Van Herwegen, lead developer of the Community Solid Server

Publishing of Test Reports

As a developer, you’re in control of what, how, and why you test. In a true decentralized spirit, we encourage developers to publish their own test reports whenever they feel ready to do so. After all, implementers know their own work best—and we love supporting all of you.

Next Steps

Even as we release CTH v1.0, we are already looking at improvements:

• Supporting Solid developers with additional and new parts of the specification. CTH will be extended to accommodate tests covering more of the Solid specification (e.g., HTTP compliance, new Solid specification functionality).
• Making it even easier to contribute to tests by simplifying the creation of Containers, and/or Resources, and setting access controls, through the use of higher-level library methods.
• Enabling choice of which subsets of the tests are run (e.g., approved only) to focus report contents.
• Helping testers understand the test results by incorporating comments in the reports. This could, for example, highlight discussions related to a test such as why it expects a particular response.
• Supporting increased test coverage by working through the specifications, methodically identifying and annotating any requirements that still need testing.

The ultimate goal of CTH is to empower developers with the confidence that their servers and apps will continue working sustainably. Ultimately, interoperability is a people thing: it’s all of us agreeing on how our implementations should behave toward each other, and verifying that they do. This is why we’re so excited to grow this test suite with all of you. Get involved by joining the conversation at https://gitter.im/solid/test-suite.