Test Plan - Incident Response v0.3.0

0.0

Initial version

04/23

0.1

Updated with feedbacks

04/29

0.2

Added test 60

04/30

0.3

Test in progress

05/01

1.0

Tests done

05/01

Test Area - Plugin Setup

1

Admin can install plugin in a non-EE instance

1. Log in to a non-enterprise Mattermost, as admin

2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

3. Verify that the plugin can be installed.

Tested on local. Installed from beta marketplace at https://api.private-beta.integrations.mattermost.com/

2

Admin can install plugin in an E10 instance

1. Log in to Mattermost that has an E10 license, as admin.

2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

3. Verify that the plugin is successfully installed.

Tested on local. Installed from beta marketplace at https://api.private-beta.integrations.mattermost.com/

3

Admin can install plugin in an E20 instance

1. Log in to Mattermost that has an E20 license, as admin

2. From System Console > PLUGINS > Plugin Management, upload the Incident Response plugin.

3. Verify that the plugin can be installed.

4

Admin can enable plugin

1. Once installed in the “Installed Plugins” section, find the Incident Response plugin v0.2.0 and click on the “Enable” button.

2. Verify that the plugin starts normally without errors.

5

Plugin functionality is available to admin

1. Switch to a normal channel.

2. Type /incidentin the post input box.

3. Verify that /incidentis listed as a slash command option.

4. Verify that the channel header has Incident Response icon for RHS.

5. Create a post.

6. Click post action menu for the post.

7. Verify that Start Incident is available as a post action menu option.

6

Plugin functionality is available to normal user

1. Log in as a non-admin user.

2. In a normal channel, type /incidentin the post input box.

3. Verify that /incidentis listed as a slash command option.

4. Verify that the channel header has Incident Response icon for RHS.

5. Create a post.

6. Click post action menu for the post.

7. Verify that Start Incident is available as a post action menu option.

7

Admin can disable plugin

1. Log out and log back in as admin.

2. From System Console > PLUGINS > Plugin Management, find the Incident Response plugin.

3. Click on the “Disable” button.

4. Verify that the plugin is disabled.

5. In the main channel’s post input box, type /incident.

6. Verify that /incident option is not present in the slash command list.

8

Admin can remove plugin

1. From System Console > PLUGINS > Plugin Management, find the Incident Response plugin.

2. Click on the “Remove” button.

3. Verify that the plugin is no longer installed in the instance.

9

Plugin upgrades normally from v0.2.3 to v0.3

1. Install older version of the plugin.

2. Upgrade to the latest version.

3. Verify that the plugin upgrade happens without any error.

Tested on https://new-522.test.mattermost.cloud/

Upgrading from v0.2 to v0.3 causes the RHS to go blank

MM-24751: Upgrading the plugin from 0.2 to 0.3 changes the incident list to a blank RHSClosed

10

Incidents started with the older plugin version is still active and available in v0.3

1. Start a few incident with plugin version 0.2.

2. Upgrade plugin to v0.3.

3. Open up the incident RHS to see the incident list.

4. Verify that incidents started with version 0.2 are still available in v0.3.

11

Incidents ended with plugin v0.2 are not available in v0.3

1. Start a few incident with plugin version 0.2.

2. Leave some as active and end a few of them.

3. Upgrade plugin to v0.3.

4. Open up the incident RHS to see the incident list.

5. Verify that incidents started with version 0.2 are still available in v0.3.

6. Verify that incidents ended in step 2 are not visible in the list.

Test Area - Monitor

As a non-member of the incident channel, I’m prompted to contact the commander if I want to join.

13

A non-incident member viewing incident details on incident RHS sees a message to contact the commander to participate in the incident

Continue from the above test.

1. While viewing the incident details on RHS, verify that at the bottom of the RHS, the user sees a message that instructs them to contact the commander to participate in the incident.

14

‘Contact commander’ message on incident details view on RHS shows the correct commander

Continue from the above test.

1. Verify that the ‘contact commander’ message at the bottom of incident RHS shows the username of user A as commander in the message.

2. Verify that clicking on the commander’s username opens user A’s profile popover.

15

A non-incident member cannot end an incident

Continue from the above test.

1. Verify that user B does not have an “End Incident” button on the incident RHS details view.

16

If already channel member, contact commander message is not shown

Continue from the above test.

1. Login as user A.

2. In the incident channel started in the above tests, add user B as a member.

3. Logout and login as user B.

4. Verify that user B has the incident channel from step 2 above listed on the channels sidebar.

5. Open the incident RHS and find the incident above and click to see the details view.

6. Verify that the incident channel now appears in the details view.

7. Verify that the ‘contact commander’ message no longer appears at the bottom of the incident RHS.

17

A non-commander channel member can end an incident

Continue from the above test.

1. As user B, verify that the incident details view in the RHS can see an “End Incident” button.

18

Playbook icon is present in Incident RHS header

1. From the channel header, click on the incident icon to open the incident RHS.

2. Verify that the RHS header has a playbook icon.

3. Click on any incident to go to the incident details view.

4. Verify that the RHS header has the playbook icon.

19

Add a tooltip for the plus icon (New Incident) in the RHS

1. Click on the incident icon on the channel header.

2. Find the “+” icon on the incident RHS header.

3. Hover the mouse on the “+” icon.

4. Verify that a tooltip appears that signifies that it is for starting a new incident.

MM-24553: Add tooltip for RHS buttons Closed

20

Add a tooltip for the book icon (Playbooks) in the RHS

1. Click on the incident icon on the channel header.

2. Find the playbook icon on the incident RHS header.

3. Hover the mouse on the book icon.

4. Verify that a tooltip appears that signifies that it is for starting a new playbook.

MM-24553: Add tooltip for RHS buttons Closed

Test Area - Coordinate

• As a user, I can configure incident playbooks with a checklist to optionally save time setting up future incidents.

21

Clicking on the Playbook icon brings user to the Playbooks backstage

1. Login as a normal user.

2. From the channel header, click on incident icon to open incident RHS.

3. From the RHS header, click on the playbook icon.

4. Verify that the the screen view changes to a full-modal playbooks backstage.

22

“Playbook” is highlighted in backstage when user is brought there after clicking playbook icon

Continue from the above test.

1. Verify that upon clicking the playbook icon when the user is brought to the playbooks backstage, the “Playbooks” section on the left sidebar is highlighted.

23

When no playbook exists, the backstage shows “there are no playbooks defined yet”

1. Ensure that there are no predefined playbooks yet.

2. Click on the playbook icon to open up the Playbooks backstage.

3. Verify that the main Playbooks view notifies the user with a message that “there are no playbooks defined yet”.

4. Verify that a “New Playbook” button appears on the Playbooks backstage.

24

Clicking on New Playbook icon lets user add playbook name and checklist items

Continue from the above test.

1. Bring up the Playbooks backstage if not already open.

2. Click on the “New Playbook” icon.

3. Verify that the view changes from “Playbooks” to “New Playbook”.

4. Verify that an input field for Playbook Name exists.

5. Verify that user can see “Add new checklist item” button.

25

Canceling new playbook does not save the playbook

Continue from the above test.

1. Click on “New Playbook” button.

2. Enter “Playbook 1” for Playbook name.

3. Add a few checklist items.

4. Click “Cancel”.

5. Verify that the user is brought back to the Playbooks view on the backstage.

6. Verify that “Playbook 1” does not exist as a playbook.

7. Repeat steps 1 - 3.

8. Without saving, from the top, click on the '<' button.

9. Verify that the user is brought back to the Playbooks view on the backstage.

10. Verify that “Playbook 1” does not exist as a playbook.

26

New Playbook > Cancel > Discard Changes on confirmation dialog does not save playbook

Exits out of New Playbook view

27

New Playbook > Cancel > Cancel does not

Does not exit out of New Playbook view

28

Clicking on “ ← Back to Mattermost” brings user back to normal channel

Continue from the above test.

1. From the Playbooks backstage, click on the “ ← Back to Mattermost” button.

2. Verify that the user is brought back to Mattermost’s channel view.

29

Incident creation dialog shows playbook option

1. Bring up the incident RHS by clicking the incident icon on channel header.

2. Click on the “+” icon to start a new incident.

3. Verify that in the Incident Details interactive dialog, in addition to Commander and Channel Name, a Playbook dropdown is also present.

30

Not selecting any option from Playbook dropdown allows for a new incident creation

Continue from the above test.

1. Enter “I-A” for Channel Name.

2. Do not select anything for Playbook.

3. Click “Start Incident”.

4. Verify that the incident “I-A” got created and appears on the list on RHS.

31

“Playbook” in incident start modal only has “None” in dropdown when there is no playbook

1. Click on the “+” icon to start a new incident.

2. Enter “NI-1” for Channel Name.

3. Click on the Playbook dropdown.

4. Verify that the dropdown only contains “None” as an option.

32

Selecting “None” from Playbook dropdown does not create any playbook

Continue from the above test.

1. Select “None” from the playbook dropdown.

2. Click “Start Incident”.

3. Verify that a new incident “NI-1” is now started.

4. Click on the playbook icon to go to the Playbook backstage.

5. Verify there is no playbook.

33

A new playbook can be saved from the Playbook backstage

Continue from the above test.

1. Click on the “New Playbook” button.

2. Enter “PB-1” for Playbook name.

3. Add “Started”, “In progress”, and “Analysis” as checklist items.

4. Click “Save Playbook”.

5. Verify that the backstage “Playbooks” view shows PB-1 listed.

34

A playbook can be edited from the Playbook backstage

Continue from the above test.

1. Click on “edit” for PB-1 in the backstage.

2. Click “edit” for the checklist.

3. Add a new item “Resolved” at the end of the checklist.

4. Click “done”.

5. Click “Save Playbook”.

6. Click “edit” for PB-1 again.

7. Verify that the list shows all 4 items: “Started, “In progress”, “Analysis” and “Resolved”.

35

Playbooks created from backstage show up on the incident creation interactive dialog’s Playbook dropdown

Continue from the above test.

1. Create one more playbook, “PB-2” from the backstage (following steps 1-4 from the above test case).

2. Add different checklist items and save.

3. Go back to normal MM view.

4. Bring up the Incident Details interactive dialog (using any method).

5. Click the Playbook dropdown icon.

6. Verify that dropdown includes both “PB-1” and “PB-2” as options.

36

Selecting predefined playbook auto-populates an incident’s checklist

Continue from the above test.

1. Enter “I-B” for Channel Name.

2. Pick PB-1 for Playbook.

3. Click “Start Incident”.

4. Once the incident gets started, find the incident “I-B” in the incident list on RHS.

5. Click on incident I-B to see the details view.

6. Verify that the checklist has all items previously defined for PB-1 (“Started”, “In progress”, “Analysis” and “Resolved”) listed.

37

Checklist edited on incident RHS does not change the playbook checklist on backstage

Continue from the above test.

1. In incident RHS, click “edit” for the checklist started in the above test.

2. Delete “Started”.

3. Add “Generate Report” as the 3rd item on the checklist.

   1. Click “Add new checklist item”.

   2. Add “Generate Report” and press Enter.

   3. Drag “Generate Report” between “Analysis” and “Resolved”.

4. Click “done”.

5. Click on the playbook icon to go to the backstage.

6. In the backstage, click on “Edit” for PB-1.

7. Verify that the changes made in steps 2 and 3 are NOT reflected.

8. Verify that the checklist items on backstage remain the same as it was before the edits from step 1 of this test (with “Started”, “In progress”, “Analysis” and “Resolved”).

38

Checklist edited on backstage reflects properly during incident creation

Continue from the above test.

1. On Playbook backstage, click “edit” for “PB-2”.

2. Edit the checklist such that the items are now as follows:

   1. “Unplug device”

   2. “Wait 5 seconds”

   3. “Plug device again”

   4. “Done”

3. Switch to normal MM view.

4. Start an incident using any method.

5. In the Incident interactive dialog, enter “I-3” for Channel Name.

6. Enter “PB-2” from Playbook dropdown.

7. Click “Start Incident”.

8. Find the incident on the RHS list.

9. Verify that the checklist items appear as changed in step 2.

39

Checklist edited on backstage does not affected previously started incident's checklist

Continue from the above test.

1. On Playbook backstage, click “edit” for “PB-2”.

2. Edit the checklist such that the items are now as follows:

   1. “Turn off device”

   2. “Unplug”

   3. “Wait 3 seconds”

   4. “Plug back again”

   5. “Turn on device”

3. Start an incident using any method.

4. In the Incident interactive dialog, enter “I-4” for Channel Name.

5. Enter “PB-2” from Playbook dropdown.

6. Click “Start Incident”.

7. Find the incident on the RHS list.

8. Verify that the checklist items appear as changed in step 2.

9. Find incident “I-3” from the above test.

10. View details.

11. Verify that the checklist has not changed as edited in step 2.

40

A “delete playbook confirmation” dialog pops up when deleting a playbook

Continue from the above test.

1. Go to Playbook backstage by clicking on the playbook icon on RHS.

2. From the list, find playbook “PB-2”.

3. Click “delete” for “PB-2”.

4. Verify that a confirmation pop up appears.

MM-24464: Delete playbook in backstageClosed

41

A “delete playbook” can be canceled

Continue from the above test.

1. In the confirmation dialog click the “Cancel” button.

2. Verify that the playbook is not deleted and it is still there in the playbook list.

3. Switch to normal MM view.

4. Start an incident using any method.

5. In the incident interactive dialog, verify that the playbook still appears on the Playbook dropdown.

42

A playbook can be deleted from the backstage

Continue from the above test.

1. Go to Playbook backstage by clicking on the playbook icon on RHS.

2. From the list, find playbook “PB-2”.

3. Click “delete” for “PB-2”.

4. In the confirmation dialog click “Confirm”.

5. Verify that the playbook no longer appears in the Playbooks list.

6. Switch to normal MM view.

7. Start an incident using any method.

8. In the incident interactive dialog, verify that the playbook no longer appears on the Playbook dropdown.

43

A deleted playbook does not affect previously started incident using that playbook

Continue from the above test.

1. In the incident RHS, find incidents I-3 and I-4 from above tests.

2. Verify that the checklists for I-3 and I-4 are not changed.

44

A playbook created in Team X is not visible for Team Y’s incident creation

1. As user A, go to Team X.

2. Create a playbook “PB-X”.

3. Verify that PB-X is accessible in the Playbook dropdown during incident creation.

4. Switch to Team Y.

5. Start an incident using any method.

6. Verify that “PB-X” is not present in the Playbook dropdown.

45

A playbook created in Team X is not visible in Team Y’s Playbook Backstage

1. As user A, go to Team X.

2. Create a playbook “PB-X”.

3. Verify that PB-X is accessible in the Playbook dropdown during incident creation.

4. Go to Playbook backstage.

5. Verify that PB-X is available in playbooks list.

6. Switch to Team Y.

7. Go to playbook backstage.

8. Verify that PB-X is not available in playbooks list.

• As a member of the incident channel, I can change the incident commander from the RHS so that it remains accurate.

46

A commander can see a dropdown for users under “Commander” in details view

1. Login as a normal user A.

2. Start a new incident.

3. Open the incident RHS by clicking on incident icon on the channel header.

4. Find the incident started in step 2.

5. Click on the incident to see the detailed view.

6. Verify that the username under commander shows a dropdown icon.

47

A non-incident member cannot see a dropdown for users under “Commander” in details view

Continue from the above test.

1. Login as user B.

2. Open up the incident RHS.

3. Find the incident started by user A in above tests.

4. Click on the incident to see details view.

5. Verify that the username under “Commander” is not a dropdown menu.

48

An non-commander incident member can see a dropdown for users under “Commander” in details view

49

Incident channel members are listed in the user dropdown list

Continue from the above test.

1. Login as user A.

2. Start a new incident.

3. In the incident channel, add users B and C.

4. Open up the incident RHS.

5. Find incident from step 2 and click to see incident details.

6. Click the user dropdown under Commander.

7. Verify that both users B and C show up on the list.

50

An incident channel member can be searched by a commander in the user dropdown list

Continue from the above test.

1. In the search box (in the users dropdown), type the username for user C.

2. Verify that the list reduces to show only user C.

Can be searched by username, or full name or nick name.

51

A non-incident channel member cannot be searched in the user dropdown list

Continue from the above test.

1. In the search box, type the username for user Y, who is not yet added to the incident channel.

2. Verify that a message shows that the user could not be found.

52

A new user can be designated as a new commander

1. As user A, start a new incident.

2. Add user B to the incident channel.

3. Open the incident RHS.

4. Find the incident from step 1 and click to see the details.

5. Click the user dropdown below Commander.

6. Select user B.

7. Verify that the commander for the incident changes to user B.

8. Verify a bot message is posted to the incident channel that the user A changed the commander to user B.

9. Log in as user B.

10. Find the incident in the RHS.

11. Verify that user B appears as the commander in incident list view on the RHS.

12. Click on the incident to see the details.

13. Verify that user B is listed as the commander.

53

An incident channel member can edit checklist of the incident

1. As user A, start a new incident.

2. Add user C to the incident channel.

3. Log in as user C.

4. Open incident RHS.

5. Find the incident from step 1.

6. Click to view details.

7. Verify user C can edit the checklist.

54

An incident channel member can see a dropdown for users under “Commander” in details view

Continue from the above test.

1. Under commander, verify user C can see user dropdown.

2. Verify search works for user C in the user dropdown.

3. Verify user C can select users from the user dropdown.

55

An incident channel member can end an incident

Continue from the above test.

1. Click on the “End Incident” button for the incident.

2. Verify user C can end the incident.

56

A non-incident member cannot see a dropdown for users under “Commander” in details view

Continue from the above test.

1. Log in as user B and find the above incident (where user C was added) in the incident RHS.

2. Click to see details.

3. Verify that user B cannot see a ‘user dropdown’ under “Commander”.

57

Old commander cannot edit checklist of the incident if kicked from incident channel

1. As user A start an incident.

2. Add user B to incident channel.

3. Assign user B as the incident commander.

4. Log in as user B.

5. Go to the incident channel of the incident from step 1.

6. Kick user A from the channel.

7. Open incident RHS and find the incident.

8. Click to view details.

9. Verify user A does not appear in the user dropdown under commander (even in search).

10. Log out and log in as user A.

11. Open incident RHS and find the incident.

12. Click to see details.

13. Verify user A cannot edit the checklist.

58

Old commander cannot end an incident if kicked from incident channel

Continue from the above test.

1. Verify that user A cannot end the incident.

Test Area - Measure

• As the incident plugin, a message is posted to the incident channel when the commander is changed.

59

A bot message is posted to the incident channel when commander is changed.

1. Log in as user A.

2. Start a new incident.

3. Add user B to the incident channel.

4. From the incident RHS, find the incident started in step 2 and see the details.

5. From the user dropdown, pick user B.

6. Verify that in the incident channel, the Incident bot posted a message that “user A changed the incident commander from user A to user B”.

• As a user, I can review when each checklist item was (last) completed so that I can identify the steps that are taking the most time.

60

Checklist item displays when the item was checked off

1. As user A, start a new incident.

2. Add checklist items and save.

3. Invite user B to the incident channel.

4. From the checklist created in step 2, checkoff the first two items.

5. Verify that a timestamp appears next to both the checklist items on the RHS of when the item was checked.

6. Verify that the timestamps appear accurately for user B for both checklist items.

61

Checklist item displays the latest time when the item was checked off

Continue from the above test.

1. Uncheck the first item.

2. Verify that the timestamp beside the checklist item is no longer available.

3. Check the first item again.

4. Verify that the new timestamp appears beside the checklist item.

5. Verify that timestamps appear accurately for user B for both checklist items.

62

Items checked more than a day ago shows the day along with timestamp of when the item was checked

Continue from the above test.

1. Leave the items checked for a few days (or spoof checked time).

2. After a few days, check the timestamp of the checklist item.

3. Verify that the timestamp also includes the day when the item was checked.

Verified on community-daily

Test Area - Tests/bugs outside of user stories

63

Incident created in one team is visible in another team

1. Create an incident in team A.

2. Open incident RHS.

3. Find the incident from step one and click on it to see the details.

4. With the RHS still open in details view, switch teams to team B.

5. Open incident RHS.

6. Verify incident created in team A is not visible in team B.

MM-24716: Incident created in one team is visible in another teamClosed

64

Incident list on RHS is empty in Safari

MM-24743: Incident list on RHS is empty in SafariClosed

65

User is not always taken to the incident channel after starting an incident

https://mattermost.atlassian.net/browse/MM-24745

66

Playbooks deleted in different tab are still available in incident creation modal if the modal was opened before playbook deletion

MM-24747: Playbooks deleted in different tab are still available in incident creation modal if the modal was opened before playbook deletionClosed

67

Clicking on incidents for detailed view causes blank screen

https://mattermost.atlassian.net/browse/MM-24748

68

Tab navigation in playbook backstage "Edit Playbook" view highlights the entire checkbox container

MM-24752: Tab navigation in playbook backstage "Edit Playbook" view highlights the entire checkbox containerClosed

69

Incident icon on the header closes RHS if clicked while viewing reply thread

MM-24754: Incident icon on the header closes RHS if clicked while viewing reply threadClosed