Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Version History

Version 1 Next »

Target release

Q1 2020

Epic

https://mattermost.atlassian.net/browse/MM-20777

Edition

N/A

Document status

READY

OVERVIEW

Increasing automated testing is necessary in order to increase the effectiveness, efficiency and coverage of software testing. Currently testing SAML is a manual process that only occurs during release and feature testing. Because testing SAML is a manual process we only test against OneLogin and Okta.

Creating reusable automated tests will make testing and supporting additional SAML providers a much easier process. These tests will ensure a base level of supported functionality. As test coverage increases, supporting a new provider could be as simple as writing the test harness for the given provider. 

GOALS

  1. Define the testing architecture for Automated SAML testing.

  2. Define the necessary APIs for manipulating SAML data.

  3. Define the base set of required tests 

SCOPE

In scope:

  • Define a pluggable architecture to test SAML Environment.

  • Provide a proof of concept.

Not in scope:

  • Defining implementation details for specific providers.  

  • Define all tests and all potential data.

  • Define implementation details for all required tests.

BACKGROUND READING


SPECIFICATIONS

There are three different pieces of functionality to make this work.

  1. Architecture Definition

    1. Define test structure

    2. Define test data

    3. Define base tests to be implemented.

  2. Provider Creation and Setup

  3. Provider Test Creation and API definition

Architectural Definition

Test Structure

The test structure should be configured so that each provider can share the same test. Each provider will have to some of its own provisioning such as Setting up its configuration. At a minimum the SAML Configuration will need to be provided. A sample test flow is shown below. This test is doing a basic SAML login.

The okta_login_spec.js file is the file specific to the Okta provider. The login_test.js file will contain the shared testing logic. Depending on the test some of these steps may not be necessary or others required. 

verifyProviderData - verify ProviderData is setup as expected. 

setConfiguration - set the Mattermost “SAML Settings” Configuration for specific provided. 

runLoginTests(testParameters) - call the shared test

visit() … find(‘a’).click - perform shared test operations

testParameters.samlLoginPage() - call back into the provider specific file in order to fill out and login via the providers page.

verifyLogin()...logout() - perform shared test operations

samlLogout() - logout of SAML Provider, if necessary.

resetData() - if data was modified for test, reset data if necessary.

Test Data

The SAML providers will have to have specific users with specific fields to be set appropriately for the accompanying test. Rather than creating that data upfront, it is each test suites responsibility to create the provider data prior to each test. Since the data will need to be verified and possibly reset before each test run, there doesn’t not seem to be a reason to create that data up front. However, this could differ from provider to provider.

If test data is required to be added to the Mattermost database. That data can be added in the following file - ../mattermost-server/cmd/mattermost/commands/sampledata.go.

Base Tests

  • SAML User Login - New user

  • SAML User Login - Existing user

  • SAML Guest Login - New user

  • SAML Guest Login - Existing user

  • SAML Admin Login - New user

  • SAML Admin Login - Existing user

Required SAML APIs

Create User - create a new user in SAML Server

Update User - update user information in SAML server

Deactivate User - deactivate a user in SAML server

Retrieve User - retrieve user information from SAML server

Provider Creation and Setup

The system should allow the user to easily create new SAML Providers to test. That process should be as follows: 

  • Create Test Account on SAML Provider

In order to complete ongoing, automated testing, a test account will need to be created on the given SAML provider. Those accounts already exist for Okta and OneLogin.

  • Create Application Setup on SAML Provider.

In order to use the SAML Provider, it will need to be configured. A new application should be created in the provider. This configuration is usually a manual process and differs based on the provider. Here are links to instructions  for configuring the currently support providers.

If creating a new SAML provider, be sure to document the setup steps for others to follow. Be sure the following attributes are included at a minimum. This may change as more tests are developed.

  • Email

  • Username

  • IsGuest

  • IsAdmin

  • UserType

  • FirstName

  • LastName

  • Import Base Data into SAML Provider (May be done at test time instead)

Whether this should be an import or a call in before() is dependent on the type of API support the SAML provider. Whichever method is used, the ability to “reset” the data will be necessary.

The requirements for the base data are:

  • 25 “Normal” SAML Users

    • IsGuest = false

    • UserType = null

  • 2 “Guest” Users

    • IsGuest = true && UserType = null

    • IsGuest = false && UserType = “guest”

    • Allows testing two different types of filters

  • 2 “Admin” Users (currently under development)

    • IsAdmin = true && UserType = null

    • IsAdmin = false && UserType = “admin”

    • Allows testing two different types of filters

  • CSV file, may require updating for specific

  • Write _spec.js file for new SAML Provider

For each provider a separate test file will be created. It will set Config setting, define the API functions and Other needed properties before calling the shared test.

  • Config Settings

    • SAMLSettings.Enable = true

  • API Functions (separate file)

    • CreateUser

    • UpdateUser

    • DeactivateUser

    • RetrieveUser

    • ResetUser

  • Properties

    • signin-submit textbox name

    • SAMLProviderLogin()

Known Issues
These tests may have to run as separate suites from the current tests. The reason is that these tests will require the cypress.json file to have the following setting. 

"chromeWebSecurity": false
From the POC testing, it appears this setting cannot be set at runtime and we don’t want this setting = false for most testing. 

ExperimentalSettings.UseNewSAMLLibrary

All of the tests should be run twice, once using the existing SAML Library and one using the New SAML Library. However, that configuration setting is only loaded on server start, it does not change on the fly. Therefore, the base configuration will need to be updated and the server restarted.

Certificates will need to be created, stored and installed as part of configuration setup. We don’t want new certificates for each run as that would require setting up providers prior to each test run.

Proof of Concept

In order to ensure that the above specification is at least feasible, a proof of concept was developed.

https://github.com/sbishel/mattermost-webapp/tree/saml-test-poc

The main files for the POC are located in the directory
../e2e/cypress/integration/enterprise/login

./login_tests.js

  • contains the tests that are shared across providers.

./okta_login/okta_api.js

  • contains the api functions for Okta administration.

./okta_login/okta_login_spec.js

  • contains the test code for Okta. Ensures user exists in SAML provider.

The test expects the user to already exist in Mattermost, it won’t. Therefore, the first time through, select the default team. The next time the test will pass. It’s just a POC.

CREDITS


  • No labels