OVERVIEW

Increasing automated testing is necessary in order to increase the effectiveness, efficiency and coverage of software testing. Currently testing SAML is a manual process that only occurs during release and feature testing. Because testing SAML is a manual process we only test against OneLogin and Okta.

Creating reusable automated tests will make testing and supporting additional SAML providers a much easier process. These tests will ensure a base level of supported functionality. As test coverage increases, supporting a new provider could be as simple as writing the test harness for the given provider. 

GOALS

1. Define the testing architecture for Automated SAML testing.

2. Define the necessary APIs for manipulating SAML data.

3. Define the base set of required tests 

SCOPE

In scope:

• Define a pluggable architecture to test SAML Environment.

• Provide a proof of concept.

...

• Defining implementation details for specific providers.  

• Define all tests and all potential data.

• Define implementation details for all required tests.

BACKGROUND READING

SPECIFICATIONS

There are three different pieces of functionality to make this work.

1. Architecture Definition

   1. Define test structure

   2. Define test data

   3. Define base tests to be implemented.

2. Provider Creation and Setup

3. Provider Test Creation and API definition

Architectural Definition

Test Structure

The test structure should be configured so that each provider can share the same test. Each provider will have to some of its own provisioning such as Setting up its configuration. At a minimum the SAML Configuration will need to be provided. A sample test flow is shown below. This test is doing a basic SAML login.

...

The okta_login_spec.js file is the test file specific to the Okta provider. This file will control the test flow for Okta tests. This file will be created specifically for the SAML provider being tested. The login_test

The okta_api_commands.js file will contain the apis for creating, modifying users and other settings required in the Okta system. This file will be created specifically for the SAML provider being tested.

The saml_test_commands.js file will contain the shared testing logic. Depending on the test some of these steps may not be necessary or others required. 

verifyProviderData - verify ProviderData is setup as expected. 

setConfiguration . A single test may call one or many commands to complete the test.

resetProviderData - Create, Reset, or Verify the data is correct on the SAML provider. If data is static and pre-created in the SAML provider, this step may not be necessary. Depending on the needs, the test will use a combination of apis from okta_api_commands.js.

updateConfiguration - set the Mattermost “SAML Settings” Configuration as required for specific provided. 

runLoginTests(testParameters) doSAMLLoginClick - call the shared test

visit() … find(‘a’).click - perform shared test operations

testParameters.samlLoginPage() - call back into the provider specific file in order to fill out and login via the providers page.

verifyLogin()...logout() - perform shared test operations

samlLogout() - logout of SAML Provider, if necessary.

resetData() - if data was modified for test, reset data if necessary.command to visit Mattermost Login and click the “SAML” button.

oktaSAMLLogin - Each specific provider will provide test steps for ensuring the Provider’s login page is displayed correctly, and the user is logged in successfully.

verifyMMLogin() - call the shared command to make sure the Mattermost user is successfully logged in and the correct page is displayed.

samlLogout() - logout MM user and ensure login page is displayed

Test Data

The SAML providers will have to have specific users with specific fields to be set appropriately for the accompanying test. This can be accomplished a couple different ways. That data can be created upfront in the SAML provider or each test suite can be responsible to create its own test data prior to each test. If the data in the SAML provider is static, that is it won’t be updated during a test run and be the same from test run to test run. It is suggested to create that data upfront. However, if the test requires the update of the SAML data, for instance to test if SAML fields update properly. Then the data should be either be verified or deleted and recreated before the test run.

If test data is required to be added to the Mattermost database. That data can be added in the following file - ../mattermost-server/cmd/mattermost/commands/sampledata.go.

Base Tests

• SAML User Login - New user

• SAML User Login - Existing user

• SAML Guest Login - New user

• SAML Guest Login - Existing user

• SAML Admin Login - New user

• SAML Admin Login - Existing user

Once the base tests have been completed, the manual release tests will be next.
https://docs.google.com/spreadsheets/d/18nQEYeHRQCkHPRmiDZLd-z8DyMg1TlDfDu-c0qlK8Vs/edit#gid=932310946

Required SAML APIs

Create User - create a new user in SAML Server

...

• Certificates - will need loaded

• SAML Provider settings may need updated, for server name to callback.
  

Proof of Concept

In order to ensure that the above specification is at least feasible, a proof of concept was developed.

https://github.com/sbishel/mattermost-webapp/tree/saml-test-pocpoc2

The main files for the POC are located in the directory as follows -
..e2e/cypress.json

..e2e/cypress/fixtures/saml_users.json

e2e/cypress/integration/enterprise/login.saml/okta_login_testsspec.js

• contains the tests that are shared across providers.

./okta_logine2e/cypress/support/index.js

e2e/cypress/support/okta_api_commands.js

• contains the api functions for Okta administration.

./okta_login/okta_login_spec.js

...

e2e/cypress/support/saml_test_commands.js

The test expects the user to already exist in Mattermost, it won’t. Therefore, the first time through, select the default team. The next time the test will pass. It’s just a POC.

CREDITS