Versions Compared

Version Old Version 16 New Version Current
Changes made by

Prapti Shrestha (Deactivated)

Prapti Shrestha (Deactivated)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Document version 1.1

Status: Test Completed

Document Version

Description

Date

0.0

Initial version

05/14

0.1

Initial review from the team

06/15

1.0

Test run for 0.5 complete

07/24

1.1

Test doc cleaned up

07/27

...

  1. The tests outlined in this document is to be executed against the Incident Response plugin v0.45.0.

  2. This version of the plugin can be installed and used in Mattermost instances that aren’t equipped with any enterprise license.

  3. The Incident Response plugin can function as a standalone plugin that does not require to talk to any other plugin.

  4. This version of the Incident Response plugin will function properly only in the latest version of Mattermost.

  5. The tests will be executed in the latest version of Mattermost.

  6. Testing is primarily done on the webapp, with spot checks on the desktop app, RN mobile app and mobile web browser app.

  7. The tests in this document are executed as a normal user (see glossary above) unless otherwise specified.

  8. The test cases that pass for a normal user will also pass for a system administrator.

  9. Plugin uploads is enabled in server instance

    1. “EnableUploads” option under “PluginSettings” in mattermost-server’s config.json is set to 'true'.

  10. The channel export feature defined in the user story is dependent on the existence of the Channel Export plugin being installed and enabled in the test server.

  11. Private channels cannot be converted into public channel, hence private incidents cannot be converted to public incidents.

...

Test Area - Limit incident access to only participants

As an incident participant, I can make an incident and its channel private so that non-participants don’t know it exists let alone access its content.

Test ID

Test Case

Test Procedure

Result

Notes

14

Incident cannot be created without selecting a playbook

  1. As user-1, bring up the incident creation modal.

  2. Provide a channel name.

  3. Do not select anything from the Playbook dropdown.

  4. Do not input anything for Channel Name.

  5. Click “Start Incident”.

  6. Verify that incident is not created.

    1. Verify that the Incident Details modal remains open.

    2. Verify that “This field is required.” message appears below Playbook dropdown.

    3. Verify that “This field is required.” message appears below Channel Name input box.

  7. Enter a Channel Name.

  8. Leave the Playbook field empty.

  9. Repeat steps 5 and 6.

 Pass

15

User can make a public incident

  1. As user-1, create a public playbook as follows:

    1. Launch the playbook backstage.

    2. Click “New Playbook”.

    3. Provide name of the playbook - “Public PB 1“.

    4. Enable the “Create Public Incident” option.

    5. Enter a few checklist items.

    6. Click “Save”.

  2. Switch to the main channel view.

  3. Create an incident by selecting the playbook “Public PB 1”.

  4. Do not add any other user to the incident channel.

  5. As user-2, verify that the incident is visible in the incident backstage.

  6. Click on the incident.

  7. Verify that the incident details can’t be viewed.

  8. Verify that the incident details can be viewed.

 Pass

#7 is a bug.

Fixed in master with https://github.com/mattermost/mattermost-plugin-incident-response/pull/219

16

User can make a private incident

  1. As user-1, launch the playbook backstage.

  2. Click “New Playbook”.

  3. Provide a playbook name “Private-PB-1”.

  4. Keep the “Create Public Incident” option disabled.

  5. Add some checklist items.

  6. Save playbook.

  7. Create an incident by selecting the playbook “Private-PB-1”.

  8. Log in as user-2.

  9. Launch incident backstage.

  10. Verify that the incident is not visible in the incident list.

 Pass

17

Converting a public incident channel into private, makes the incident private

  1. As a sysadmin, create a public incident “Pub-1”.

  2. Log in as user-1.

  3. Join the “Pub-1” incident channel to become a participant.

  4. Convert it into a private incident

    1. Go to Pub-1 channel

    2. From the channel header, click the dropdown

    3. From the menu, select “Convert to Private Channel”

    4. In the “Convert public to a Private Channel” confirmation box, select “Yes..”

  5. Verify access in backstage

    1. Launch incident backstage.

    2. Verify that the incident is visible in the incident list.

    3. Click on the incident.

    4. Verify that the incident details are accurately displayed.

 Pass

18

Non participant cannot access a previously public incident converted to a private incident

Continue the test from above.

  1. Log in as user-2.

  2. Verify incident is not accessible in backstage.

    1. Launch incident backstage.

    2. Verify that the incident is not visible in the incident list.

 Pass

19

Non participant cannot access a new private incident

  1. As user-1, bring up the incident modal.

  2. Give it the channel name “Pr-In-1”.

  3. From the Playbook dropdown, select “Private-PB-1”.

  4. Click “Start Incident”.

  5. Do not add anyone else to the incident channel.

  6. Log in as user-2.

  7. Verify that the incident “Pr-In-1” is not visible in the incident RHS.

  8. Launch incident backstage.

  9. Verify that the incident “Pr-In-1” is not visible in the incident list.

 Pass

20

Private incidents cannot be searched in backstage by non-participants.

Continue from the above test.

  1. In the incident backstage, search for “Pub-1”.

  2. Verify that “Pub-1” incident is not present in the search result.

  3. Search for “Pr-In-1” in the backstage.

  4. Verify that “Pr-In-1” incident is not present in the search result.

 Pass

21

Public incidents are visible only within the incident’s team

  1. As user-1, create a public incident in Team X.

  2. Launch incident backstage.

  3. Verify that the incident is visible in the incident list.

  4. Switch to Team Y.

  5. Launch the incident backstage.

  6. Verify that the incident is not visible in the incident list.

  7. As user-2, go to Team X.

  8. Launch the incident backstage.

  9. Verify that the incident is visible in the incident list.

  10. Switch to Team Z.

  11. Verify that the incident is not visible in the incident list.

 Pass

22

Private incidents are visible only within the incident’s team

  1. As user-1, create a private incident using playbook with private setting on.

  2. Launch incident backstage.

  3. Verify that the incident is visible in the incident list.

  4. Switch to Team Y.

  5. Launch the incident backstage.

  6. Verify that the incident is not visible in the incident list.

  7. As user-2, go to Team X.

  8. Launch the incident backstage.

  9. Verify that the incident is not visible in the incident list.

  10. Switch to Team Z.

  11. Verify that the incident is not visible in the incident list.

 Pass

23

Public incidents cannot be searched in a different team’s incident backstage

  1. As user-1, create a public incident “Pub C” in Team Y.

  2. Launch incident backstage.

  3. Verify that the incident is visible in the incident list.

  4. Search for “Pub C” in the backstage.

  5. Verify that “Pub C” is present in the search result.

  6. Switch to Team X.

  7. Launch the incident backstage.

  8. Verify that the incident is not visible in the incident list.

  9. Search for “Pub C” in the backstage.

  10. Verify that “Pub C” is not present in the search result.

 Pass

24

System administrator cannot see a private incident

  1. As user-1, create a private incident “Pr A”.

  2. Add a few checklist items.

  3. Log in as sysadmin.

  4. Launch incident backstage.

  5. Verify that sysadmin cannot see “Pr A” in the incident list.

 Pass

Won’t see private channel, but if permalink known, can simply browse to channel and self-join.

25

System administrator can join a private incident

Continue from the above test.

  1. As sysadmin, use the permalink to the incident created in the above test.

  2. Verify that sysadmin can successfully join the channel.

 Pass

Won’t be in LHS, but definitely can join channel with permalink.

26

System administrator can view a private incident details

Continue from the above test.

  1. As sysadmin, open up the incident RHS.

  2. Verify that sysadmin can see the incident details for “Pr A” on the RHS.

  3. Launch incident backstage.

  4. Click on “Pr A”.

  5. Verify that sysadmin can see the details of “Pr A” correctly in the details view in backstage.

 Pass

27

System administrator can check and uncheck items in private incidents

Continue from the above test.

  1. As sysadmin, open up the incident RHS to view the details of “Pr A”.

  2. Check an item from the checklist.

  3. Verify that sysadmin can successfully check an item off the list.

  4. Uncheck an item from the checklist.

  5. Verify that sysadmin can successfully uncheck an item off the list.

  6. Launch the incident backstage.

  7. Find and click “Pr A” to view the details.

  8. Verify that the items checked off and unchecked are reflected properly in the graph.

 Pass

28

System administrator can end a private incident

Continue from the above test.

  1. As sysadmin, open the incident details of “Pr A” on the RHS.

  2. Click “End Incident”.

  3. Verify that the incident is ended successfully.

  4. Launch the incident backstage.

  5. Verify that the incident end state under “Status” is properly reflected in the list view.

  6. Verify that the incident end state is properly reflected in the incident details view.

 Pass

29

Team administrator cannot see a private incident

  1. As user-1, create a private incident “Pr B”.

  2. Add checklist items “check 1” and “check 2”.

  3. Log in as a team admin.

  4. Verify that team admin cannot see the private channel “Pr B” on the LHS.

  5. Launch incident backstage.

  6. Verify that team admin cannot see “Pr B” in the incident list.

 Pass

Team admins can’t see a private incident. Test changed from can to cannot.

30

Team administrator can check and uncheck items in private incidents

Continue from the above test.

  1. Invite team admin to the “Pr B” incident channel.

  2. As team admin, open up the incident RHS to view the details of “Pr B”.

  3. Check both items “check 1” and “check 2” from the checklist.

  4. Verify that team admin can successfully check the item off the list.

  5. Uncheck item “check 2” from the checklist.

  6. Verify that team admin can successfully uncheck the item off the list.

  7. Launch the incident backstage.

  8. Find and click “Pr B” to view the details.

  9. Verify that the items checked off and unchecked are reflected properly in the graph.

 Pass

Team admins can’t see a private incident. Invalid test.

31

Team administrator can end a private incident

Continue from the above test.

  1. As team admin, open the incident details of “Pr B” on the RHS.

  2. Click “End Incident”.

  3. Verify that the incident is ended successfully.

  4. Launch the incident backstage.

  5. Verify that the incident end state under “Status” is properly reflected in the list view.

  6. Verify that the incident end state is properly reflected in the incident details view.

 Pass

Test Area - Configure playbooks to create public/private incidents

As an incident manager, I can configure playbooks to create either public or private incidents so that it defaults to the correct permission when executed.

Test ID

Test Case

Test Procedure

Result

Notes

32

Playbook can be configured to create a public incident

  1. Log in as team admin.

  2. Launch playbook backstage.

  3. Click on “New Playbook”.

  4. Enter “PB 1” for Playbook Name.

  5. Enable the button to “Create Public Incident”.

  6. Add checklist items “item 1” , “item 2” and “item 3”.

  7. Click “Save Playbook”.

  8. Switch to normal channel view.

  9. Bring up the incident creation dialog.

  10. Create incident “New Public PB1”.

  11. From the playbook dropdown, select “PB 1”.

  12. Click “Start Incident”.

  13. Verify that the incident created has the correct checklist items.

  14. Log in as user-2.

  15. Launch incident backstage.

  16. Verify user-2 can see “New Public PB1” in the incident list.

 Pass

33

Playbook can be configured to create a private incident

  1. Log in as team admin.

  2. Launch playbook backstage.

  3. Click on “New Playbook”.

  4. Enter “PB 2” for Playbook Name.

  5. Ensure the button to “Create Public Incident” is disabled.

  6. Add checklist items “item a” , “item b” and “item c”.

  7. Click “Save Playbook”.

  8. Switch to normal channel view.

  9. Bring up the incident creation dialog.

  10. Create incident “New Public PB2”.

  11. From the playbook dropdown, select “PB 2”.

  12. Click “Start Incident”.

  13. Verify that the incident created has the correct checklist items.

  14. Log in as user-2.

  15. Launch incident backstage.

  16. Verify user-2 cannot see “New Public PB1” in the incident list.

 Pass

34

Playbook can be edited to change public incident creation setting to private

Continue from the above tests.

  1. Log in as user-1.

  2. Launch playbook backstage.

  3. Find playbook “PB 1”.

  4. Click “Edit”.

  5. Disable the button to “Create Public Incident”.

  6. Click “Save Playbook”.

  7. Switch to normal channel view.

  8. Bring up the incident creation dialog.

  9. Create incident “Incident PB3”.

  10. From the playbook dropdown, select “PB 1”.

  11. Click “Start Incident”.

  12. Verify that the incident created has the correct checklist items.

  13. Log in as user-2.

  14. Launch incident backstage.

  15. Verify user-2 cannot see “Incident PB3” in the incident list.

 Pass

35

Playbook can be edited to change private incident creation setting to public

Continue from the above tests.

  1. Log in as user-1.

  2. Launch playbook backstage.

  3. Find playbook “PB 2”.

  4. Click “Edit”.

  5. Enable the button to “Create Public Incident”.

  6. Click “Save Playbook”.

  7. Switch to normal channel view.

  8. Bring up the incident creation dialog.

  9. Create incident “Incident PB4”.

  10. From the playbook dropdown, select “PB 2”.

  11. Click “Start Incident”.

  12. Verify that the incident created has the correct checklist items.

  13. Log in as user-2.

  14. Launch incident backstage.

  15. Verify user-2 can see “Incident PB4” in the incident list.

 Pass

Test Area - Prioritize incidents that the user is a participant of

As an incident participant, I can easily tell apart the incidents that are relevant to me so that it’s I can get there more quickly.

Test ID

Test Case

Test Procedure

Result

Notes

36

Incidents in backstage will be listed in priority order

  1. Log in as user-1.

  2. Add user-2 to incident “New Public PB 1”.

  3. Log in as user-2.

  4. Create a private incident “U2 Pr 1”.

  5. Launch incident backstage.

  6. Verify that the incidents are listed in the following order:

    1. U2 Pr 1 (user-2 is the creator)

    2. New Public PB 1 (user-2 is the member of)

    3. Incident PB4 (public incident that user-2 is not a member of)

  7. Switch to a normal channel view.

  8. Create a new public incident “U2 Pu”.

  9. Launch incident backstage.

  10. Verify that the incidents are listed in the following order:

    1. U2 Pu (the latest incident created by user-2)

    2. U2 Pr 1

    3. New Public PB 1

    4. Incident PB 4

N/A

Is the order correct in step 6 and 10?

Prioritizing incident was for RHS incident list. Since incident list is removed from the RHS, this test is invalid.

Test Area - Limit the ability to create playbooks to Team Administrators

As a system administrator, I can restrict who can create playbooks so that the incident response process can be better standardized.

Test ID

Test Case

Test Procedure

Result

Notes

37

System administrator can create a public incident playbook

  1. Log in as a sysadmin.

  2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

  3. Click “New Playbook”.

  4. Enable the setting for “Create Public Playbook”.

  5. Provide “SA PB 1” for the Playbook Name.

  6. Enable the setting for “Create Public Incident” to allow the playbook to create public incidents by default.

  7. Add the following checklist items:

    1. Start test

    2. Edit test

    3. End test

  8. Click “Save Playbook”.

  9. Verify that upon saving, “SA PB 1” exists in the playbook list view in backstage.

 Pass

38

System administrator can create a private incident playbook

  1. Log in as a sysadmin.

  2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

  3. Click “New Playbook”.

  4. Enable the setting for “Create Public Playbook”.

  5. Provide “SA PB 2” for the Playbook Name.

  6. Disable the setting for “Create Public Incident” to allow the playbook to create private incidents by default.

  7. Add the following checklist items:

    1. Start test

    2. Edit test

    3. End test

  8. Click “Save Playbook”.

  9. Verify that upon saving, “SA PB 2” exists in the playbook list view in backstage.

 Pass

39

Team admin can create a public-incident playbook

  1. Log in as a team admin.

  2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

  3. Click “New Playbook”.

  4. Ensure the “Create Public Incident” setting is enabled.

  5. Provide “TA PB 3” for the Playbook Name.

  6. Enable the setting for “Create Public Incident” to allow the playbook to create public incidents by default.

  7. Add the following checklist items:

    1. Start test

    2. Edit test

    3. End test

  8. Click “Save Playbook”.

  9. Verify that upon saving, “TA PB 3” exists in the playbook list view in backstage.

 Pass

40

Team admin can create a private-incident playbook

  1. Log in as a team admin.

  2. Launch the Playbooks backstage (either from main menu or incident RHS header’s playbook button).

  3. Click “New Playbook”.

  4. Ensure the “Create Public Playbook” setting is disabled.

  5. Provide “TA PB 4” for the Playbook Name.

  6. Disable the setting for “Create Public Incident” to allow the playbook to create private incidents by default.

  7. Add the following checklist items:

    1. Start test

    2. Edit test

    3. End test

  8. Click “Save Playbook”.

  9. Verify that upon saving, “TA PB 4” exists in the playbook list view in backstage.

 Pass

41

A playbook is initially only visible to playbook creator

  1. Log in as user-1.

  2. Launch playbook backstage.

  3. Create a new playbook “U1PB”.

  4. Verify that “Members” has only user-1 included.

  5. Switch to a normal channel view.

  6. Bring up the incident creation dialog.

  7. Verify that user-1 can see “U1PB” listed in the playbook dropdown.

  8. Log in as user-2.

  9. Launch playbook backstage.

  10. Verify that user-2 does not see “U1PB” in the Playbooks list.

  11. Switch to a normal channel view.

  12. Bring up the incident creation dialog.

  13. Verify that user-2 cannot see “U1PB” listed in the playbook dropdown.

 Pass

42

Private playbook cannot be deleted by non-participating member user of the team

Continue from the above test.

  1. Log in as user-1.

  2. Issue a DELETE request to http://localhost:8065/plugins/com.mattermost.plugin-incident-response/api/v1/playbooks/[playbook-id].

  3. Verify that the request cannot be completed.

  4. As team admin, verify that the playbook did not get deleted.

 Pass

Test Area - Limit playbook access to specific users

As a playbook creator, I can specify who else can see and edit the playbook so that sensitive content wouldn’t be unintentionally revealed.

Test ID

Test Case

Test Procedure

Result

Notes

43

Users who have access to a playbook can limit playbook access to specific users

  1. Log in as team admin.

  2. Create a playbook “TA 1”.

  3. Add checklist items “1”, “2” and “3”.

  4. Invite user-2 in the playbook.

  5. Click “Save Playbook”.

  6. Login as user-1.

  7. Launch playbook backstage.

  8. Verify that user-1 cannot find “TA 1” in the backstage.

  9. Switch to normal channel view.

  10. Bring up the incident creation dialog.

  11. Verify that user-1 cannot find “TA 1” in the playbook dropdown.

  12. Login as user-2.

  13. Launch playbook backstage.

  14. Verify that user-1 can find “TA 1” in the backstage.

  15. Switch to normal channel view.

  16. Bring up the incident creation dialog.

  17. Verify that user-1 can find “TA 1” in the playbook dropdown.

 Pass

44

Playbook Any playbook member can edit users who have access to playbook

Continue from the above test.

  1. As team admin, launch playbook backstage.

  2. Find “TA 1”.

  3. Click “Edit”.

  4. Invite user-1 to manage the playbook “TA 1”.

  5. Click “Save Playbook”.

  6. Verify that user-1 can now find “TA 1” in the backstage.

  7. Switch to normal channel view.

  8. Bring up the incident creation dialog.

  9. Verify that user-1 can now find “TA 1” in the playbook dropdown.

  10. Launch the playbook backstage.

  11. Find and click on “TA-1”.

  12. Remove user-2 from the members list of the playbook.

  13. As user-2, verify that “TA-1” is no longer available in the incident creation modal.

  14. As user-2, verify that “TA-1” is no longer available in the team’s playbook backstage.

  15. As user-1, add user-2 to the playbook again.

  16. As user-2, verify that “TA-1” is now available in the incident creation modal.

  17. As user-2, verify that “TA-1” is available in the playbook backstage.

 Pass

45

User removed by any member of a playbook can no longer access playbook

Continue from the above test.

  1. As user-1, launch playbook backstage.

  2. Find “TA 1”.

  3. Click “Edit”.

  4. Remove team admin from the playbook “TA 1”.

  5. Click “Save Playbook”.

  6. Verify that user-2 cannot find “TA 1” in the backstage.

  7. Switch to normal channel view.

  8. Bring up the incident creation dialog.

  9. Verify that team admin cannot find “TA 1” in the playbook dropdown.

 Pass

46

Playbook member can limit playbook access to specific users

  1. Log in as team admin.

  2. Create a playbook “Sa 1”.

  3. Add checklist items “a”, “b” and “c”.

  4. Invite user-2 in the playbook.

  5. Click “Save Playbook”.

  6. Login as user-1.

  7. Launch playbook backstage.

  8. Verify that user-1 cannot find “Sa 1” in the backstage.

  9. Switch to normal channel view.

  10. Bring up the incident creation dialog.

  11. Verify that user-1 cannot find “Sa 1” in the playbook dropdown.

  12. Login as user-2.

  13. Launch playbook backstage.

  14. Verify that user-2 can find “Sa 1” in the backstage.

  15. Switch to normal channel view.

  16. Bring up the incident creation dialog.

  17. Verify that user-2 can find “Sa 1” in the playbook dropdown.

 Pass

47

Playbook member can edit users who have access to playbook

Continue from the above test.

  1. As team admin, launch playbook backstage.

  2. Find “Sa 1”.

  3. Click “Edit”.

  4. Invite user-1 to manage the playbook “Sa 1”.

  5. Click “Save Playbook”.

  6. Verify that user-1 can now find “Sa 1” in the backstage.

  7. Switch to normal channel view.

  8. Bring up the incident creation dialog.

  9. Verify that user-1 can now find “Sa 1” in the playbook dropdown.

 Pass

48

User removed by system admin can no longer access playbook

Continue from the above test.

  1. As team admin, launch playbook backstage.

  2. Find “Sa 1”.

  3. Click “Edit”.

  4. Remove user-2 from the playbook “Sa 1”.

  5. Click “Save Playbook”.

  6. Verify that user-2 cannot find “Sa 1” in the backstage.

  7. Switch to normal channel view.

  8. Bring up the incident creation dialog.

  9. Verify that user-2 cannot find “Sa 1” in the playbook dropdown.

 Pass

49

Last playbook member cannot be removed from the playbook

  1. Log in as user-1.

  2. Create a playbook.

  3. Add user-2 as a member to the playbook.

  4. Log in as user-2.

  5. Verify that user-2 can access the playbook in the backstage.

  6. Go to the editing view of the playbook.

  7. Remove user-1 from the playbook.

  8. Save the playbook.

  9. Verify that the playbook saved successfully.

  10. Go back to the editing view of the playbook.

  11. Remove the user themself (user-2) as the member.

  12. Click “Save Playbook”.

  13. Verify taht the playbook cannot be saved.

 Pass

...