Solid Conformance Test Suite

In this post we describe an approach and proposed new architecture for a conformance test suite for the Solid Protocol and associated specifications.

Interoperability is one of the core underlying requirements for a thriving open Solid ecosystem. This is achieved through the Solid Protocol and the associated specifications. The Solid server, upon which all new and exciting interoperable apps will depend, is one of the central components of the Solid ecosystem. Once a Solid server conforms with the Solid specifications we can be confident that it will interoperate as expected in the distributed ecosystem. But how do we know that a server is conformant with the Solid specifications?

Enter the need for a high quality conformance test suite.  Test suites drive interoperability and are a key part of making sure standards are implemented correctly and consistently.

Like everything else on the web, we did not start this from a blank page. Much work has been done to date on investigating how to approach a test suite for the Solid protocol and there are even two initial implementations that provide some tests today. We have taken many learnings from these efforts and would like to acknowledge and thank Kjetil Kjernsmo, Michiel de Jong and everybody else who committed to the repositories and issues, for their open source contributions and work on Solid test suites.

Requirements

Initially we spent some time considering what would be required of a good test suite. The following sections outline the main requirements we ended up with. The existing implementations don’t meet these requirements so we would like to close that gap by taking a different approach to the test suite. We found it is relatively easy to take the existing tests from those implementations and make them available in the proposed implementation.

Traceability to specifications

It must be possible to automate the linking of significant units of information in specifications, such as requirements as well as security, privacy, accessibility and internationalisation considerations. In order to achieve this, we support the proposals on embedding RDFa in specifications and test suite description documents. This has been done with other specifications.

Declarative and easy to understand

If tests are difficult to understand, overly complex, or require high levels of expertise then the risk of incorrect tests will be high. Server implementations written to pass an incorrect test suite may drive interoperability on incorrect behaviour, which would have widespread, negative consequences for the entire Solid ecosystem. Therefore we should strive to make tests declarative and easy to understand to avoid incorrect tests in the test suite. This has the added benefit of making the test suite understandable to a larger number of people in the community. This should lead to both a larger number of contributors and a larger number of reviewers.

Specification requirement levels

The Solid specifications contain requirements of different levels using keywords as defined by BCP14. The test suite must be capable of determining the combinations of tests to execute based on the specifications and the capabilities of a given Solid server. This ensures that tests for optional features are only executed against servers that claim they implement those features.

Reporting

Extensive work has already been carried out on reporting formats by a working group and the resulting language, EARL, should be used for reports from conformance test suites. This will ensure that the data in the reports can be used by many different tools.

The reports generated by the test suite should provide high level pass/fail results, but it should also be possible to drill down to fine levels of detail to determine the exact tests that failed and why they failed. Of course, it must be easy to automatically make those reports available on the web.

Inspection

Developers writing Solid servers will benefit from detailed information associated with failing tests; therefore the request and response payloads between the client and server for each test executed should be accessible by the developer. This will make it easier for developers to debug their server code.

Maintainability

The test suite needs to evolve with the specifications as they mature. Therefore it is imperative that the system be easily maintainable, as it is likely to be modified and maintained by many different people over the years.

The system used to execute the tests should be decoupled from the tests themselves. This will ensure that most of the complexity can be encapsulated in a test harness while the tests themselves can be as simple as possible.

Obviously the test harness must be well architected and of a high quality to avoid the problems of brittle software and technical debt. This will ensure the test harness remains maintainable by the community for years to come. We propose that;

• The architecture for the test suite should be reviewed by the specification team and the community
• The code should conform to a published set of coding standards
• The code for the test suite should be accompanied by automated tests providing 100% branch coverage

Extensibility

As the Solid specifications evolve and as the Solid ecosystem matures there will be a need to add many more tests and different types of tests. It should be simple to extend the test suite with new tests for existing specifications and for new specifications without having to add more code to the test harness. This will reduce the risk of breaking existing tests and will enable more people in the community to contribute to the test suite.

Flexibility and ease of use

Different types of people will want to use the test suite for different purposes. For example, Solid server developers may want to include it in their continuous integration pipeline. This type of developer may want the suite to execute as fast as possible so will benefit from the ability to run many tests in parallel. Those deciding which Solid server to use may want to execute the test suite from the command line when experimenting with a server. Others may want to provide web pages with the results of running the test suite against multiple servers.

The test suite will also be used in many different environments such as different cloud environments, or an individual developer’s workstation. With all these considerations in mind the test suite should make minimal assumptions about its environment and should be easy to configure to use in many different ways.

Architecture

The following is an illustration of the proposed high level architecture of the test suite.

Test harness

The test harness controls the overall execution of a test suite. It is responsible for loading the test suite, locating the tests, creating and controlling test executors and generating test suite compliance reports.

The harness will provide different interfaces to the test suite such as a REST API and a command line interface, and will be platform agnostic.

Test executor

The test executor is responsible for executing tests against a Solid server. The harness can create multiple test executors where each is responsible for executing a set of tests. Test executors do not need to run on the same host as the test harness.

Test suite description document

This is an RDF document containing metadata about the test suite. The metadata includes;

• Prerequisites (e.g. dependencies)
• Requirements level from the specification (e.g., MUST, SHOULD, MAY, etc.)
• References to the appropriate section in the specification
• References to the actual tests

The test suite at any point in time may not cover all of the tests required for the specifications. The test harness will use the linkage between the test suite document and the RDF in the specifications to determine how much of the specifications are covered by the test suite.

Test cases

The test cases will be contained in a repository and may be grouped in various ways to enable logical sets of tests to be grouped cohesively.

Conformance report

Conformance reports will be generated using EARL thereby making them available for consumption by many different tools.

Component Architecture

The following is an illustration of the component architecture for the test harness and executor.

Test Language

We have already mentioned that we believe tests should be declarative and easy to understand. We are proposing to use Karate as the underlying test framework and test language. Tests are written in Gherkin, but with built-in support for HTTP interactions. We will add RDF format conversions and assertions. Gherkin has a successful history with a rich set of integrations and the Karate extensions are well suited to the needs of a Solid conformance test suite.

The following is a generic Karate example:

Feature: karate 'hello world' example

Scenario: create and retrieve a cat

Given url 'http://myhost.com/v1/cats'
And request { name: 'Billie' }
When method post
Then status 201
And match response == { id: '#notnull', name: 'Billie' }

Given path response.id
When method get
Then status 200

Now for an example of Solid tests in Karate:

 Background:
# setup function to create the resource/acl and get the access tokens
* def testContext = callonce read('this:protected-operations-setup.feature@name=setupAccessTo') { bobAccessModes: 'acl:Read' }
* configure headers = { Authorization: '#(testContext.bobAuthHeader)' }
* url target.serverRoot + testContext.resourcePath

# prepare the teardown function
* configure afterFeature = function() {Java.type('com.inrupt.solid.testharness.HttpUtils').deleteResourceRecursively(testContext.containerUrl, testContext.aliceAuthHeader)}

  Scenario: Bob can read the resource with GET
    When method GET
    Then status 200

  Scenario: Bob can read the resource with HEAD
    When method HEAD
    Then status 200

  Scenario: Bob can read the resource with OPTIONS
    When method OPTIONS
    Then status 204

  Scenario: Bob cannot PUT to the resource
    Given request '<> <http://www.w3.org/2000/01/rdf-schema#comment> "Bob replaced it." .'
    And header Content-Type = 'text/turtle'
    When method PUT
    Then status 403

  Scenario: Bob cannot PATCH the resource
    Given request 'INSERT DATA { <> a <http://example.org/Foo;> . }'
    And header Content-Type = 'application/sparql-update'
    When method PATCH
    Then status 403

  Scenario: Bob cannot POST to the resource
    Given request '<> <http://www.w3.org/2000/01/rdf-schema#comment> "Bob replaced it." .'
    And header Content-Type = 'text/turtle'
    When method POST
    Then status 403

  Scenario: Bob cannot DELETE the resource
    When method DELETE
    Then status 403

‍There is even the possibility of creating the Solid tests in RDF:

@prefix acl: <http://www.w3.org/ns/auth/acl# >.
@prefix dct: <>.
@prefix http:  <http://www.w3.org/2007/ont/http# >.
@prefix :  <https://solidProject.org/ns/karate# >.
@prefix tool:  <https://solidProject.org/ns/karateTools# >.
@prefix rdfs:  <http://www.w3.org/2000/01/rdf-schema# >.
@prefix stat:  <http://www.w3.org/ns/posix/stat# >.

@prefix alice:  <https://alice.example.com/profile/card# >.
@prefix bob:  <https://bob.example.com/profile/card# >.

 < > dct:title "Karate in Turtle test example";
  stat:size 3514;
  :lines 92 ;
  rdfs:comment """This is an attempt to just use RDF at the same level as the
Karate example, to make a fair comparison. :when  setup is shared between a
number of similar tests in the same suite, that is trivial in RDF.""" .

:test13 a :Feature; :title "Bob can only read an RDF resource to which he is only granted read access";

  :background [
    :prepare  <#AliceSetupAccess1 >;
    # call the setup function to create the resource/acl
    :url [ :concat ( [ :target :serverRoot] [:testContext :resourcePath ] )];

    # prepare the teardown function -- actually why not be more explicit
    :configure [ :afterFeature tool:deleteResourceRecursively;  :subject (
        [ is :containerUrl of :TestContext]
        [ is :testContext of  <#aliceAuthHeader > ] )
    ]
  ];

  :loggedInAs [ :user bob:me; :do [
  :scenario [ :title "Bob can read the resource with GET";
    :when [ http:method http:GET ];
    :then [ http:status 200 ]];

  :scenario [ :title "Bob can read the resource with HEAD";
    :when [ http:method http:HEAD ];
    :then [ http:status 200 ]
  ];
  :scenario [ :title "Bob can read the resource with OPTIONS";
    :when [ http:method http:OPTIONS ];
    :then [ http:status 200 ]
  ];
  :scenario [
    :title "Bob cannot PUT to the resource";
    :given [
      :payload  """ < >  <http://www.w3.org/2000/01/rdf-schema#comment > "Bob replaced it." .""";
      :header [ :contentType "text/turtle" ]
    ];
    :when [ http:method http:PUT ];
    :then [ http:status 403 ]
  ];
  :scenario [ :title "Bob cannot PATCH the resource";
    :given [
      :payload  "INSERT DATA {  < > a  <http://example.org/Foo > . }";
      :header [ :content-Type "text/turtle" ]
    ];
    :when [ http:method http:PATCH ];
    :then [ http:status 403 ]]
  ];
  :scenario [ :title "Bob cannot POST to the resource";
    :given [
      :payload  """ < >  <http://www.w3.org/2000/01/rdf-schema#comment > "Bob posted to it." .""";
      :header [ :content-Type "text/turtle" ]
    ];
    :when [ http:method http:POST ];
    :then [ http:status 403 ]]
  ];
  :scenario [ :title "Bob cannot DELETE the resource";
    :when [ http:method http:DELETE ];
    :then [ http:status 200 ]
  ] .


# Separate setup prep

 <#AliceSetupAccess1 > a :Preparation; :title """Set up a sample Turtle file for Alice with defined access set for Bob""";

  :background [

    :loggedInAs [ :user alice:me; :do [
       :operation http:PUT;
        :target    <//alice.example.com/foo.ttl >;
        :text   " # a test file"
    ], [
       :operation http:PUT;
        :targetACLOf    <//alice.example.com/foo.ttl >;
        :text   """
      aclPrefix
       + createOwnerAuthorization(target.users.alice.webID, resourcePath)
       + createBobAccessToAuthorization(target.users.bob.webID, resourcePath, bobAccessModes)
    """ ]
    ]
 ] .

Next Steps

We have already completed a proof of concept using the proposed architecture. The test harness was written in Java and dockerised, while all of the tests were written in Karate. We are now ready to bring the proposal to the Solid specification team and proceed with a production quality implementation.

Of course this will be an open source contribution to the Solid ecosystem and we would be delighted to get feedback and contributions from members of the community. The public repository will contain all of the source code and links to all the relevant material.

Thank you for taking the time to read this post and we look forward to working with you to ensure we all enjoy an interoperable Solid ecosystem.