Test Plan - Incident Response v0.5

0.0

Initial version

05/14

0.1

Test

Test ID

Test Case

Test Procedure

Result

Notes

1

Admin can install plugin in a non-EE instance

1. Log in to a non-enterprise Mattermost, as admin

2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

3. Verify that the plugin can be installed.

2

Admin can install plugin in an E10 instance

1. Log in to Mattermost that has an E10 license, as admin.

2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

3. Verify that the plugin is successfully installed.

3

Admin can install plugin in an E20 instance

1. Log in to Mattermost that has an E20 license, as admin

2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

3. Verify that the plugin can be installed.

4

Admin can enable plugin

1. Once installed in the “Installed Plugins” section, find the Incident Response plugin v0.2.0 and click on the “Enable” button.

2. Verify that the plugin starts normally without errors.

5

Plugin functionality is available to admin

1. Switch to a normal channel.

2. Type /incidentin the post input box.

3. Verify that /incidentis listed as a slash command option.

4. Verify that the channel header has Incident Response icon for RHS.

5. Create a post.

6. Click post action menu for the post.

7. Verify that Start Incident is available as a post action menu option.

6

Plugin functionality is available to normal user

1. Log in as a non-admin user.

2. In a normal channel, type /incidentin the post input box.

3. Verify that /incidentis listed as a slash command option.

4. Verify that the channel header has Incident Response icon for RHS.

5. Create a post.

6. Click post action menu for the post.

7. Verify that Start Incident is available as a post action menu option.

7

Admin can disable plugin

1. Log out and log back in as admin.

2. From System Console > PLUGINS > Plugin Management, find the Incident Response plugin.

3. Click on the “Disable” button.

4. Verify that the plugin is disabled.

5. In the main channel’s post input box, type /incident.

6. Verify that /incident option is not present in the slash command list.

8

Admin can remove plugin

1. From System Console > PLUGINS > Plugin Management, find the Incident Response plugin.

2. Click on the “Remove” button.

3. Verify that the plugin is no longer installed in the instance.

9

Plugin upgrades normally from v0.3 to v0.5

1. Install version 0.3 of the plugin.

2. Upgrade to the latest version.

3. Verify that the plugin upgrade happens without any error.

10

Plugin upgrades normally from v0.4 to v0.5

1. Install version 0.4 of the plugin.

2. Upgrade to the latest version.

3. Verify that the plugin upgrade happens without any error.

11

Incidents started with the prior plugin versions are still active and available in v0.5

1. Start a few incident with plugin version 0.4.

2. Upgrade plugin to v0.5.

3. Open up the incident RHS to see the incident list.

4. Verify that incidents started with version 0.4 are still not available in v0.5.

12

Incidents ended with prior plugin versions are not available in v0.5

1. Start a few incident with plugin version 0.4.

2. Leave some as active and end a few of them.

3. Upgrade plugin to v0.5.

4. Open up the incident RHS to see the incident list.

5. Verify that incidents started with version 0.4 are still not available in v0.5.

6. Verify that incidents ended in step 2 are not visible in the list.

13

Playbook started with the older plugin version is still available upon plugin upgrade

1. Create a few playbooks with plugin version 0.4.

2. Upon plugin upgrade, verify that the playbooks created in step 1 are still not available in the backstage.

Test ID

Test Case

Test Procedure

Result

Notes

14

Incident created without selecting a playbook is public by default

1. As user-1, bring up the incident creation modal.

2. Provide a channel name.

3. Do not select anything from the Playbook dropdown.

4. Click “Start Incident”

5. Do not add any other user to the incident channel.

6. As user-2, click the incident icon in channel header.

7. Verify that the incident is visible in the RHS incident list.

8. Launch incident backstage.

9. Verify that the incident is visible in the incident list.

15

User can make a public incident

1. As user-1, create a playbook as follows:

   1. Launch the playbook backstage.

   2. Click “New Playbook”.

   3. Provide name of the playbook - “Public PB 1“

   4. Enter a few checklist items.

   5. Click “Save”.

2. Switch to the main channel view.

3. Create an incident by selecting the playbook “Public PB 1”.

4. Do not add any other user to the incident channel.

5. As user-2, verify that the incident is visible in the incident backstage.

6. Click on the incident.

7. Verify that the incident details can’t be viewed.

16

User can make a private incident

1. As user-1, launch the playbook backstage.

2. Click “New Playbook”.

3. Provide a playbook name “Private-PB-1”.

4. Keep the “Create Public Incident” option disabled.

5. Add some checklist items.

6. Save playbook.

7. Create an incident by selecting the playbook “Private-PB-1”.

8. Login as user-2.

9. Verify that the incident is not visible in the incident RHS.

10. Launch incident backstage.

11. Verify that the incident is not visible in the incident list.

17

Converting a public incident channel into private, makes the incident private

1. As a sysadmin, create a public incident “Pub-1”.

2. Log in as user-1.

3. Join the “Pub-1” incident channel to become a participant.

4. Convert it into a private incident

   1. Go to Pub-1 channel

   2. From the channel header, click the dropdown

   3. From the menu, select “Convert to Private Channel”

   4. In the “Convert public to a Private Channel” confirmation box, select “Yes..”

5. Verify access in backstage

   1. Launch incident backstage.

   2. Verify that the incident is visible in the incident list.

   3. Click on the incident.

   4. Verify that the incident details are accurately displayed.

18

Non participant cannot access a previously public incident converted to a private incident

Continue the test from above.

1. Log in as user-2.

2. Verify incident is not accessible in backstage.

   1. Launch incident backstage.

   2. Verify that the incident is not visible in the incident list.

19

Non participant cannot access a new private incident

1. As user-1, bring up the incident modal.

2. Give it the channel name “Pr-In-1”.

3. From the Playbook dropdown, select “Private-PB-1”.

4. Click “Start Incident”.

5. Do not add anyone else to the incident channel.

6. Login as user-2.

7. Verify that the incident “Pr-In-1” is not visible in the incident RHS.

8. Launch incident backstage.

9. Verify that the incident “Pr-In-1” is not visible in the incident list.

20

Private incidents cannot be searched in backstage by non-participants.

Continue from the above test.

1. In the incident backstage, search for “Pub-1”.

2. Verify that “Pub-1” incident is not present in the search result.

3. Search for “Pr-In-1” in the backstage.

4. Verify that “Pr-In-1” incident is not present in the search result.

21

Public incidents are visible only within the incident’s team

1. As user-1, create a public incident in Team X.

2. Launch incident backstage.

3. Verify that the incident is visible in the incident list.

4. Switch to Team Y.

5. Launch the incident backstage.

6. Verify that the incident is not visible in the incident list.

7. As user-2, go to Team X.

8. Launch the incident backstage.

9. Verify that the incident is visible in the incident list.

10. Switch to Team Z.

11. Verify that the incident is not visible in the incident list.

22

Private incidents are visible only within the incident’s team

1. As user-1, create a private incident using playbook with private setting on.

2. Launch incident backstage.

3. Verify that the incident is visible in the incident list.

4. Switch to Team Y.

5. Launch the incident backstage.

6. Verify that the incident is not visible in the incident list.

7. As user-2, go to Team X.

8. Launch the incident backstage.

9. Verify that the incident is not visible in the incident list.

10. Switch to Team Z.

11. Verify that the incident is not visible in the incident list.

23

Public incidents cannot be searched in a different team’s incident backstage

1. As user-1, create a public incident “Pub C” in Team Y.

2. Launch incident backstage.

3. Verify that the incident is visible in the incident list.

4. Search for “Pub C” in the backstage.

5. Verify that “Pub C” is present in the search result.

6. Switch to Team X.

7. Launch the incident backstage.

8. Verify that the incident is not visible in the incident list.

9. Search for “Pub C” in the backstage.

10. Verify that “Pub C” is not present in the search result.

24

Non participants can view public incident details in RHS

Continue from the above test.

1. As user-1, add checklist items to Pub C.

2. Check at least 1 item on the checklist.

3. As user-2 go to Team Y.

4. From the public channels list in the LHS, find the incident channel for “Pub C” and click.

5. Verify that the Pub C incident details view opens up in the RHS.

6. Verify that user-2 can see the checklist items as created in step 1.

7. Verify that user-2 can see the correct state of checklist items as modified in step 2.

25

Non participants cannot edit incident details in RHS of a public incident unless added as the incident channel’s member

Continue from the above test.

1. As user 2, go to Team Y.

2. View Pub C’s details on the RHS.

3. Verify that the checklist items are greyed out (inactive).

4. Verify user-2 cannot add items to the checklist.

5. Verify user-2 cannot check off checklist items.

6. Verify user-2 cannot uncheck checklist items.

Need to find out more about the mechanism to view the incident details on RHS if the details view is going to be tightly coupled with the channel view.

26

Non participant cannot access view private incident details in RHS

27

System administrator can see a private incident

1. As user-1, create a private incident “Pr A”.

2. Add a few checklist items.

3. Log in as sysadmin.

4. Verify that sysadmin can see the private channel “Pr A” on the LHS.

5. Launch incident backstage.

6. Verify that sysadmin can see “Pr A” in the incident list.

28

System administrator can join a private incident

Continue from the above test.

1. As sysadmin, in the main channel view, find and click “Pr A” on the LHS.

2. Verify that sysadmin can successfully join the channel.

29

System administrator can view a private incident details

Continue from the above test.

1. As sysadmin, open up the incident RHS.

2. Verify that sysadmin can see the incident details for “Pr A” on the RHS.

3. Launch incident backstage.

4. Click on “Pr A”.

5. Verify that sysadmin can see the details of “Pr A” correctly in the details view in backstage.

30

System administrator can check and uncheck items in private incidents

Continue from the above test.

1. As sysadmin, open up the incident RHS to view the details of “Pr A”.

2. Check an item from the checklist.

3. Verify that sysadmin can successfully check an item off the list.

4. Uncheck an item from the checklist.

5. Verify that sysadmin can successfully uncheck an item off the list.

6. Launch the incident backstage.

7. Find and click “Pr A” to view the details.

8. Verify that the items checked off and unchecked are reflected properly in the graph.

31

System administrator can add and remove checklist items in a private incident

Continue from the above test.

1. As sysadmin, add checklist item “Admin check“ to “Pr A”.

2. Verify that “Admin check” is successfully added to the checklist.

3. Remove one of the items added earlier by user-1 from the checklist.

4. Verify that the item is successfully removed.

5. Launch the incident backstage.

6. Find and click “Pr A” to view the details.

7. Verify that the items added and removed are reflected properly.

32

System administrator can end a private incident

Continue from the above test.

1. As sysadmin, open the incident details of “Pr A” on the RHS.

2. Click “End Incident”.

3. Verify that the incident is ended successfully.

4. Launch the incident backstage.

5. Verify that the incident end state under “Status” is properly reflected in the list view.

6. Verify that the incident end state is properly reflected in the incident details view.

33

Team administrator can see a private incident

1. As user-1, create a private incident “Pr B”.

2. Add checklist items “check 1” and “check 2”.

3. Log in as a team admin.

4. Verify that team admin can see the private channel “Pr B” on the LHS.

5. Launch incident backstage.

6. Verify that team admin can see “Pr B” in the incident list.

34

Team administrator can join a private incident

Continue from the above test.

1. As team admin, in the main channel view, find and click “Pr B” on the LHS.

2. Verify that team admin can successfully join the channel.

35

Team administrator can view a private incident details

Continue from the above test.

1. As team admin, open up the incident RHS.

2. Verify that team admin can see the incident details for “Pr B” on the RHS.

3. Launch incident backstage.

4. Click on “Pr B.

5. Verify that team admin can see the details of “Pr B” correctly in the details view in backstage.

36

Team administrator can check and uncheck items in private incidents

Continue from the above test.

1. As team admin, open up the incident RHS to view the details of “Pr B”.

2. Check both items “check 1” and “check 2” from the checklist.

3. Verify that team admin can successfully check the item off the list.

4. Uncheck item “check 2” from the checklist.

5. Verify that team admin can successfully uncheck the item off the list.

6. Launch the incident backstage.

7. Find and click “Pr B” to view the details.

8. Verify that the items checked off and unchecked are reflected properly in the graph.

37

Team administrator can add and remove checklist items in a private incident

Continue from the above test.

1. As team admin, add checklist item “check 3“ to “Pr B”.

2. Verify that “check 3” is successfully added to the checklist.

3. Remove “check 2” from the checklist.

4. Verify that the item is successfully removed.

5. Launch the incident backstage.

6. Find and click “Pr B” to view the details.

7. Verify that the items added and removed are reflected properly.

38

Team administrator can end a private incident

Continue from the above test.

1. As team admin, open the incident details of “Pr B” on the RHS.

2. Click “End Incident”.

3. Verify that the incident is ended successfully.

4. Launch the incident backstage.

5. Verify that the incident end state under “Status” is properly reflected in the list view.

6. Verify that the incident end state is properly reflected in the incident details view.

Test ID

Test Case

Test Procedure

Result

Notes

39

System administrator can create a public playbook

1. Log in as a sysadmin.

2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

3. Click “New Playbook”.

4. Enable the setting for “Create Public Playbook”.

5. Provide “SA PB 1” for the Playbook Name.

6. Enable the setting for “Create Public Incident” to allow the playbook to create public incidents by default.

7. Add the following checklist items:

   1. Start test

   2. Edit test

   3. End test

8. Click “Save Playbook”.

9. Verify that upon saving, “SA PB 1” exists in the playbook list view in backstage.

Is step #4 correct?

40

System administrator can create a private playbook

1. Log in as a sysadmin.

2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

3. Click “New Playbook”.

4. Enable the setting for “Create Public Playbook”.

5. Provide “SA PB 2” for the Playbook Name.

6. Disable the setting for “Create Public Incident” to allow the playbook to create private incidents by default.

7. Add the following checklist items:

   1. Start test

   2. Edit test

   3. End test

8. Click “Save Playbook”.

9. Verify that upon saving, “SA PB 2” exists in the playbook list view in backstage.

#4 correct?

41

Team admin can create a public playbook

1. Log in as a team admin.

2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

3. Click “New Playbook”.

4. Enable the setting for “Create Public Playbook”.

5. Provide “TA PB 3” for the Playbook Name.

6. Enable the setting for “Create Public Incident” to allow the playbook to create public incidents by default.

7. Add the following checklist items:

   1. Start test

   2. Edit test

   3. End test

8. Click “Save Playbook”.

9. Verify that upon saving, “TA PB 3” exists in the playbook list view in backstage.

#4 correct?

42

Team admin can create a private playbook

1. Log in as a team admin.

2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

3. Click “New Playbook”.

4. Enable the setting for “Create Public Playbook”.

5. Provide “TA PB 4” for the Playbook Name.

6. Disable the setting for “Create Public Incident” to allow the playbook to create private incidents by default.

7. Add the following checklist items:

   1. Start test

   2. Edit test

   3. End test

8. Click “Save Playbook”.

9. Verify that upon saving, “TA PB 4” exists in the playbook list view in backstage.

#4 correct?

43

A non-admin user cannot create a playbook

1. Log in as a non-admin user.

2. Launch playbooks backstage.

3. Verify that there is no “New Playbook” button in the playbooks backstage.

Is #3 correct?

44

A non-admin user cannot create a private playbook

N/A due to the test above this one. Is this correct?

45

Private playbook is initially only visible to playbook creator

1. Log in as a non-admin user.

2. Launch playbooks backstage.

3. Verify that playbooks “SA PB 2” and “TA PB 4” are not visible in the backstage.

4. Switch to a normal channel view.

5. Bring up the incident creation dialog.

6. Verify that playbooks “SA PB 2” and “TA PB 4” are not available in the Playbook dropdown.

7. Log in as the team admin.

8. Launch Playbooks backstage.

9. Verify that “SA PB 2” is not visible in the backstage.

10. Switch to normal channel view.

11. Bring up the incident creation dialog.

12. Verify that “SA PB 2” is not available in the Playbook dropdown.

13. Log in as the sysadmin.

14. Launch Playbooks backstage.

15. Verify that “TA PB 4” is not visible in the backstage.

16. Switch to normal channel view.

17. Bring up the incident creation dialog.

18. Verify that “TA PB 4” is not available in the Playbook backstage.

Are steps 13-18 correct?

Are private playbooks visible to/accessible by system admins? Or do they need to be invited to the playbook to access them as well?

46

Private playbook is only available within the team it was created in

1. Log in as sysadmin.

2. Switch to different team than the one where “SA PB 1” and “SA PB 2” were created.

3. Launch the Playbook backstage.

4. Verify that the playbooks “SA PB 1” and “SA PB 2” are not visible in the backstage.

5. Switch to normal channel view.

6. Bring up the incident creation dialog.

7. Verify that “SA PB 1” and “SA PB 2” are not available in the Playbook backstage.

47

Public playbook is visible to all the members of a team

1. Create a new team “Team B”.

2. Don’t add any user to the team.

3. Create a public playbook - “Pub-PB1”.

4. Verify no other user (non-admin team member, guest) can see “Pub-PB1” in backstage since they are not added to Team B”.

5. Add user-1 to Team B.

6. Log in as user-1.

7. Verify that user-1 can see “Pub-PB1” in backstage.

8. Verify that user-1 can see “Pub-PB1” in incident creation dialog’s playbook dropdown.

9. Add user-2 to Team B.

10. Create a new public playbook - “Pub-PB2”.

11. Verify that both user-1 and user-2 can see “Pub-PB2” in backstage.

12. Verify that both user-1 and user-2 can see “Pub-PB2” in incident creation dialog’s playbook dropdown.

48

Public playbook can be edited by member user of the team

Continue from the above tests.

1. Log in as user-1.

2. In Team B, launch the Playbook backstage.

3. Find the playbook “Pub-PB2”.

4. Click Edit for the playbook.

5. Enable the setting for “Create Public Incident”.

6. Add item “new task”.

7. Click “Save Playbook”.

8. Verify that the edit has been saved properly.

9. Click “Edit” again.

10. Verify that the edits made in step 5 & 6 persisted.

49

Team member can “invite” other team members to manage public playbook

Not sure about steps to invite other members to manage playbook.

50

Team member can create an incident using public playbook

Continue from the above test.

1. Log in as user-2.

2. Bring up the incident creation modal.

3. Create a new incident with playbook “Pub-PB2”.

4. Open the incident details.

5. Verify that the incident created is a public incident.

6. Verify that the incident created has all settings and checklist items as edited in the above test.

51

Public playbook can be deleted by team member

1. Log in as user-2.

2. Launch Playbook backstage.

3. Find “Pub-PB2”.

4. Click “Delete” for the playbook.

5. Click “Delete Playbook” in the confirmation modal.

6. Verify that the playbook is gone from the backstage list view.

7. Switch to normal channel view.

8. Bring up the incident creation modal.

9. Verify that “Pub-PB2” is not available in the playbook dropdown.

Private playbook cannot be edited by a non-admin team member

1. Note the playbook name and playbook-id of the private playbook “TA SB 4”

2. Log in as user-1.

3. Issue a PUT request to http://localhost:8065/plugins/com.mattermost.plugin-incident-response/api/v1/playbooks/[playbook-id]` with a payload change to add a checklist item.

4. Verify that the request cannot be completed.

5. As team admin, verify that the PUT request made in 3 did not take effect.

Using a direct PUT to test here because the user will not be able to find the private playbook in the UI.

Private playbook cannot be deleted by member user of the team

Continue from the above test.

1. Log in as user-1.

2. Issue a DELETE request to http://localhost:8065/plugins/com.mattermost.plugin-incident-response/api/v1/playbooks/[playbook-id].

3. Verify that the request cannot be completed.

4. As team admin, verify that t

Team member cannot “invite” other team members to manage private playbook

Team member cannot create an incident using private playbook

Test ID

Test Case

Test Procedure

Result

Notes

Test ID

Test Case

Test Procedure

Result

Notes

Test ID

Test Case

Test Procedure

Result

Notes

Test ID

Test Case

Test Procedure

Result

Notes

Test UI in light theme

Test UI in dark theme

Test UI in Monokai theme

{"sidebarBg":"#2f3136","sidebarText":"#8f9297","sidebarUnreadText":"#dcddde","sidebarTextHoverBg":"#35363c","sidebarTextActiveBorder":"#35363c","sidebarTextActiveColor":"#dcddde","sidebarHeaderBg":"#282b30","sidebarHeaderTextColor":"#818386","onlineIndicator":"#64b285","awayIndicator":"#eea941","dndIndicator":"#b74b47","mentionBg":"#7289da","mentionBj":"#ffffff","mentionColor":"#dcddde","centerChannelBg":"#36393e","centerChannelColor":"#dcddde","newMessageSeparator":"#f75252","linkColor":"#7289da","buttonBg":"#43b581","buttonColor":"#ffffff","errorTextColor":"#fd5960","mentionHighlightBg":"#36393e","mentionHighlightLink":"#7289da","codeTheme":"monokai"}

Test ID

Test Case

Test Procedure

Result

Notes