Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Version History

« Previous Version 4 Next »

Document version 0.0

Status: Test development in progress

Document Version

Description

Date

0.0

Initial version

05/11

References

Incident Response User Stories

Epic - https://mattermost.atlassian.net/browse/MM-24554 , https://mattermost.atlassian.net/browse/MM-24784

v0.3.0 tickets - https://mattermost.atlassian.net/issues/?jql=project%20%3D%20MM%20AND%20fixVersion%20%3D%20%22IR%20(v0.4.0)%22

Tests for earlier versions:
v0.1: Test Plan - Incident Response v0.1.0

v0.2: Test Plan - Incident Response v0.2.0

v0.3: Test Plan - Incident Response v0.3.0

Tests for Channel Export plugin: Test Plan - Channel Export v0.0.1

Summary

This document details the tests for Incident Response plugin v0.4.0. The tests are derived from the user stories in Incident Response overview. The MVP of the Incident Response plugin will be tested against all tests included in this test plan.

Scope

This document covers the cases for the user stories as outlined in the Incident Response MVP Overview for v.0.4.0 of the plugin only. v0.4.0 includes features that works in conjunction with the Channel Export plugin, the tests for which as a stand-alone plugin have been covered in the test plan for the plugin (see reference above). Thus, the tests in this document will validate the channel export features that are tied to the Incident Response plugin.

Glossary

  • Admin - A user with system administrator privileges.

  • Normal user - A user with non-system administrator privileges.

  • Test Server - A list of test server versions used in testing including Mattermost server and Marketplace Server

  • Main menu - Hamburger menu on the LHS.

  • Incident Response icon - The Incident Response plugin icon that appears on the channel header

  • Commander - The user who owns an incident. The user who starts the incident is designated the commander by default.

  • Incident member/incident channel member - A user who has been added to the incident channel.

  • Non-incident user - A user who has not been added to the incident channel.

  • Active incident - An incident that has not been ended.

  • Ongoing incident - Same as Active Incident.

  • Inactive incident - An incident that has been ended.

  • Incident RHS - The RHS sidebar that opens upon clicking the incident response icon and displays incidents.

  • Incident Backstage - The backstage view that shows the incident list.

  • Playbook Backstage - The backstage view that shows the playbook list.

Assumptions

The tests in this test plans are written with the assumption that:

  1. The tests outlined in this document is to be executed against the Incident Response plugin v0.4.0.

  2. This version of the plugin can be installed and used in Mattermost instances that aren’t equipped with any enterprise license.

  3. The Incident Response plugin can function as a standalone plugin that does not require to talk to any other plugin.

  4. The Incident Response plugin will function in Mattermost version 5.12 and later

  5. The tests will be executed in Mattermost version 5.23.

  6. Testing is primarily done on the webapp, with spot checks on the desktop app, RN mobile app and mobile web browser app.

  7. The tests in this document are executed as a normal user (see glossary above) unless otherwise specified.

  8. The test cases that pass for a normal user will also pass for a system administrator.

  9. Plugin uploads is enabled in server instance

    1. “EnableUploads” option under “PluginSettings” in mattermost-server’s config.json is set to 'true'.

  10. The channel export feature defined in the user story is dependent on the existence of the Channel Export plugin being installed and enabled in the test server.

Setup

Document Setup

For the practicality of running tests easily, test cases are arranged in sequential order when necessary, following the preceding test case.

Test Setup

The following setup will be necessary in order to begin testing:

  • A Mattermost v5.23 test server.

  • The test server is equipped with an E10 license for the plugin to be installed.

  • Incident Plugin v0.4.0.

Test Server:

Test Server (upgrade tests):

Build Hash:

Test Date:

Tests

Test Area - Plugin Setup

Test ID

Test Case

Test Procedure

Result

Notes

1

Admin can install plugin in a non-EE instance

  1. Log in to a non-enterprise Mattermost, as admin

  2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

  3. Verify that the plugin can be installed.

2

Admin can install plugin in an E10 instance

  1. Log in to Mattermost that has an E10 license, as admin.

  2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

  3. Verify that the plugin is successfully installed.

3

Admin can install plugin in an E20 instance

  1. Log in to Mattermost that has an E20 license, as admin

  2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

  3. Verify that the plugin can be installed.

 

4

Admin can enable plugin

  1. Once installed in the “Installed Plugins” section, find the Incident Response plugin v0.2.0 and click on the “Enable” button.

  2. Verify that the plugin starts normally without errors.

 

5

Plugin functionality is available to admin

  1. Switch to a normal channel.

  2. Type /incidentin the post input box.

  3. Verify that /incidentis listed as a slash command option.

  4. Verify that the channel header has Incident Response icon for RHS.

  5. Create a post.

  6. Click post action menu for the post.

  7. Verify that Start Incident is available as a post action menu option.

 

6

Plugin functionality is available to normal user

  1. Log in as a non-admin user.

  2. In a normal channel, type /incidentin the post input box.

  3. Verify that /incidentis listed as a slash command option.

  4. Verify that the channel header has Incident Response icon for RHS.

  5. Create a post.

  6. Click post action menu for the post.

  7. Verify that Start Incident is available as a post action menu option.

 

7

Admin can disable plugin

  1. Log out and log back in as admin.

  2. From System Console > PLUGINS > Plugin Management, find the Incident Response plugin.

  3. Click on the “Disable” button.

  4. Verify that the plugin is disabled.

  5. In the main channel’s post input box, type /incident.

  6. Verify that /incident option is not present in the slash command list.

 

8

Admin can remove plugin

  1. From System Console > PLUGINS > Plugin Management, find the Incident Response plugin.

  2. Click on the “Remove” button.

  3. Verify that the plugin is no longer installed in the instance.

 

9

Plugin upgrades normally from v0.3 to v0.4

  1. Install older version of the plugin.

  2. Upgrade to the latest version.

  3. Verify that the plugin upgrade happens without any error.

10

Incidents started with the older plugin version is still active and available in v0.3

  1. Start a few incident with plugin version 0.3.

  2. Upgrade plugin to v0.4.

  3. Open up the incident RHS to see the incident list.

  4. Verify that incidents started with version 0.3 are still available in v0.4.

 

11

Incidents ended with plugin v0.2 are not available in v0.3

  1. Start a few incident with plugin version 0.3.

  2. Leave some as active and end a few of them.

  3. Upgrade plugin to v0.4.

  4. Open up the incident RHS to see the incident list.

  5. Verify that incidents started with version 0.3 are still available in v0.4.

  6. Verify that incidents ended in step 2 are not visible in the list.

 

12

Playbook started with the older plugin version is still available upon plugin upgrade

  • Login as an admin user

  • Enable the Incident Response plugin

    • Navigate to System Console > PLUGINS > Plugin Management.

    • In “Installed Plugins” find Incident Response and click “Enabled”.

  • Log out.

Test Area - Monitor

As a user of a team, I can see a list of all past and current incidents within the team so that the information is not lost.

(Epic: https://mattermost.atlassian.net/browse/MM-24554?jql=project%20%3D%20MM%20AND%20fixVersion%20%3D%20%22IR%20(v0.4.0)%22 )

Test ID

Test Case

Test Procedure

Result

Notes

Clicking playbook button on RHS opens Playbook backstage by default

Backstage shows a tab for incidents

Clicking on “Incidents” tab on backstage LHS highlights the Incidents tab

Clicking on “Incidents” tab on backstage LHS loads the list of all incidents

The incidents list view displays the team name the incidents belong to

The incident list in the backstage displays the following columns: name, status, start timestamp, end timestamp and the commander

The incident list in the backstage shows all active incidents that belong to the current team only

https://mattermost.atlassian.net/browse/MM-24555

The incident list in the backstage shows all inactive incidents that belong to the current team only

https://mattermost.atlassian.net/browse/MM-24555

Active incidents show an “Ongoing” status in the Status column in the backstage

Inactive incidents (ended incidents) show an “Ended” status in the Statuscolumn in the backstage.

The incident list in the backstage shows updated end timestamp for an incident

An active incident shows “Ongoing” in End Date column

The incident list in the backstage shows updated commander for an incident

The incident names in the “Name” column render as clickable links

Long incident name wraps in the “Name” column displaying the entire incident name

As the incident plugin, the channel is automatically archived when the incident ends so that it reduces clutter.

Channel is archived when ending an incident with slash command

Channel is archived when ending an incident from RHS

The incident channel is no longer available on the LHS once the incident is ended

The incident channel shows up under archived channel once the incident is ended

Test Area - Coordinate

As a user, I can export channel transcript from the incident detail page so that it can be saved for record

(Epic: https://mattermost.atlassian.net/browse/MM-24784?jql=project%20%3D%20MM%20AND%20fixVersion%20%3D%20%22IR%20(v0.4.0)%22 )

Test ID

Test Case

Test Procedure

Result

Notes

When Channel Export plugin is not installed, the “Export Incident Channel” link is inactive and appears in greyscale

To admin

To incident member

To non-member

When Channel Export plugin is not installed, hovering over the “Export Incident Channel” link shows a message to “install the channel export plugin”

To admin

To incident member

To non-member

Install the channel export plugin to support exporting this incident.

When Channel Export plugin is installed in an instance with no EE license, the “Export Incident Channel” link is inactive and appears in greyscale

To admin

To incident member

To non-member

When Channel Export plugin is installed in an instance with no EE license, hovering on the “Export Incident Channel” link shows a message that “E20 license is required”

To admin

To incident member

To non-member

Exporting an incident channel requires a Mattermost Enterprise E20 license.

When Channel Export plugin is installed in an instance with an E10 license, the “Export Incident” link is inactive and appears in greyscale

To admin

To incident member

To non-member

When Channel Export plugin is installed in an instance with an E10 license, hovering on the “Export Incident Channel” link shows a message that “E20 license is required”

To admin

To incident member

To non-member

Exporting an incident channel requires a Mattermost Enterprise E20 license.

When Channel export plugin is installed but not enabled, the “Export Incident Channel” link is inactive and appears in greyscale

To admin

To incident member

To non-member

When Channel Export plugin is installed but not enabled, hovering over the “Export Incident Channel” shows “enable the plugin” message

To admin

To incident member

To non-member

Enable the channel export plugin to support exporting this incident.

When Channel Export plugin is installed and enabled, an active “Export Incident Channel” link and icon is available in the backstage incident list view

When Channel Export plugin is installed and enabled, hovering over the “Export Incident Channel” link shows a message that the action will download the channel content

An incident member can export the incident channel by clicking “Export Incident Channel” link

Link is not in greyscale

Shows action when clicked

Clicking the “Export Incident Channel” link downloads the csv file of the incident channel

The exported csv file captures the current status of the incident

Admin can export the incident without being a member of the incident channel

“Export Incident Channel” button shows as a grey-inactive button to non-incident user

Should not be able to click

A non-incident user sees a “no access” message upon hovering on the “Export Incident Channel” button

You do not have access to this incident channel. Contact the commander to export this incident.

Verify the appearance of the “Export Incident Channel” link and icon appear in dark theme

Verify the appearance of the “Export Incident Channel” link and icon appear in other themes

Exporting an ended incident channel from backstage

As the incident commander, I receive an incident summary and a link to more info after ending an incident so that I know where to go for post-mortem.
https://mattermost.atlassian.net/browse/MM-24409

Test Area - Measure

As a user, I can see a summary of the following on incident detail page so that I can get the gist at a glance:

  • Channel, start time, end time, duration, commander, number of people in the channel

(Epic: https://mattermost.atlassian.net/browse/MM-24554?jql=project%20%3D%20MM%20AND%20fixVersion%20%3D%20%22IR%20(v0.4.0)%22 )

Test ID

Test Case

Test Procedure

Result

Notes

User cannot view incident summary if the user is not an incident member

From the incidents list in the backstage, user can navigate to incident summary upon clicking the incident name of an active incident if the user is an incident member

From the incidents list in the backstage, user can navigate to incident summary upon clicking the incident name of an inactive incident

Incident summary page displays the incident name as the page heading

Incident summary page has a “<“ button that brings a user back to the incidents list view

Incident summary page bears a link to the incident channel

Hovering on the incident channel link shows a tooltip suggesting that it’s a shortcut to the incident channel

Clicking on incident channel link of active incidents brings the user to the incident channel

Clicking on incident channel link of inactive incidents, when ”ExperimentalViewArchiveChannels”:truebrings the user to the incident channel

https://mattermost.atlassian.net/browse/MM-24408

Clicking on incident channel link of inactive incidents, when ”ExperimentalViewArchiveChannels”:falseshows a message that “the channel is not accessible, contact system admin”

???

Incident summary shows the current status of the incident

Incident summary shows the current commander of the incident

For active incidents, incident summary shows the duration of how long the incident has been active for

For inactive incidents, incident summary shows the duration of how long the incident remained active for before the incident ended

For active incidents, incident summary shows the current number of members in the incident channel

For an inactive incident, incident summary shows the number of member in the incident channel when the incident was ended

For active incident, incident summary shows the total number of messages in the incident channel

For inactive incident, incident summary shows the total number of messages in the archived channel of the inactive incident

Test Area - Tests/bugs outside of user stories

Test ID

Test Case

Test Procedure

Result

Notes

Playbook name too long

Playbook name empty

Clicking checklist item timestamp should permalink jump to corresponding post

https://mattermost.atlassian.net/browse/MM-24423

Add a message indicating no checklist items in Incidents RHS

https://mattermost.atlassian.net/browse/MM-24502

Support fuzzy-searching incident name via REST API

https://mattermost.atlassian.net/browse/MM-24557

Display incident checklist timeline graph

https://mattermost.atlassian.net/browse/MM-24577

Support incident list pagination

https://mattermost.atlassian.net/browse/MM-24580

Support incident list sorting

https://mattermost.atlassian.net/browse/MM-24581

Omit playbooks table header if empty

https://mattermost.atlassian.net/browse/MM-24721

Reduce in-channel checklist notifications

https://mattermost.atlassian.net/browse/MM-24731

  • No labels