Page Comparison

Document version 0.0

Status: Test Development in Progress

...

Test Area - Attach a slash command to each step when editing playbook

Test Area - Create stages when editing a playbook
Test ID	Test Case	Test Steps	Result	Notes
14	Playbook creator can edit playbook	Log in as user-0. Launch playbook backstage (using any method). Click the “New Playbook” button. Enter “P1” for Playbook Name. Click “Save Playbook”. Switch to a normal channel. Bring up the incident creation modal (using any method). Verify that the playbook dropdown has “P1”. From the playbook backstage, find P1. Click P1 to edit the playbook. Change the playbook name to “PB-Private”. Click “Save Playbook”. Verify that the playbook in the backstage shows up as “PB-Private”. Switch to a normal channel. Bring up the incident creation modal. Verify that the playbook dropdown does not have “P1” but “PB-Private”.
15	Any playbook member can edit playbook to add a new member	Continue from the above test. From the playbook backstage, find PB-Private. Click on the playbook to open the “Edit Playbook” view In the “Members” field, add `@user-1`. Log in as user-1. Bring up the incident creation modal. Verify that user-1 can see “PB-Private” in the playbook dropdown. Launch playbook backstage. Find “PB-Private” and click to open the “Edit Playbook” view. In the “Members” field, add `@user-2`. Click “Save Playbook”. As user-2, verify that “PB-Private” is available in the playbook dropdown of the incident creation modal.
16	Any playbook member can edit playbook to remove an existing member	Continue from the above test. Log in as user-2. Launch playbook backstage. Find “PB-Private” and click to open the “Edit Playbook” view. From the “Members” field, remove `@user-1`. As user-1, verify that “PB-Private” is not available in the playbook dropdown of the incident creation modal.
17	A playbook member cannot edit playbook to remove all the members of the playbook	Continue from the above test. As user-2, launch playbook backstage. Find “PB-Private” and click to open the “Edit Playbook” view. From the “Members” field, remove all users including user-0 and user-2. Verify that the playbook cannot be saved. Add user-1 to the Members field. Save playbook.		The “Save Playbook” button should stay inactive unless there is at least one member
18	Any playbook member can edit playbook to change the private/public incident setting	Continue from the above test. As user-1, launch the playbook backstage. Find “PB-Private” and click to open the edit view. Toggle the “Create Public Incident” to enable the setting. Save playbook. Click the playbook again to view the edit mode. Verify that the “Create Public Incident” setting is enabled.
19	Any playbook member can edit playbook to add a new stage	Continue from the above test. As user-1, launch the playbook backstage. Find “PB-Private” and click to open the edit view. Click `+ Add new stage`. Change the stage name from “New stage” to “Stage 2”. Save playbook. Click the playbook again to view the edit mode. Verify that the playbook contains “Default Stage” and “Stage 2” for stages.
20	A playbook can have multiple stages with the same stage name	As user-1, launch the playbook backstage. Click “New Playbook”. Enter “PB-New” for Playbook Name. Enter “Stage 1” as the 1st stage name. Click “Add new stage”. Enter “Stage 1” as the 2nd stage name. ??		Valid?
21	A playbook member can edit playbook to add a new checklist item to a stage added by themself	22	Any playbook member can edit playbook to add a new checklist item to a stage added by a different member	23	A playbook member cannot edit playbook such that the playbook name field is empty	24	Any playbook member can reorder the checklist items within a stage	25	Any playbook member can reorder the checklist items within different stages	26	A playbook member cannot reorder the playbook such that the checklist items can be pulled outside a stage	27	Any playbook member can delete a checklist item	28	A playbook member can edit the playbook to delete all the checklist items such that all the stages can have no checklist item at all	29	Any playbook member can reorder the stages	30	A playbook member cannot reorder the playbook such that the stages can be nested	Verify that the stages cannot be stacked inside another stage	31	Any playbook member can delete a stage	32	A playbook member cannot delete the last remaining stage such that the playbook has no stages at all	33	Any playbook member can delete playbook	34	A playbook member can create playbook with a large number of stages	Verify playbook with a large number of stages in the edit page	35	A playbook member can create playbook with a large number of checklist items within the stages	Verify playbook with a large number of checklist items within the stages in the edit page	36	The components in the playbook edit page render normally	Verify the appearance of the components in the playbook edit page
Test Area - Set active workflow stage
Test ID	Test Case	Test Steps	Result	Notes
37	An active stage is annotated with `Active`label in the RHS	38	Non-active stages are not annotated with `Active` labels	39	Incident commander can set a different stage as `Active` in an ongoing incident	40	Workflow member can set a different stage as `Active` in an ongoing incident	41	Workflow member cannot set a different stage as `Active` in an incident that has been ended	42	Workflow member can set a different stage as `Active` in an incident that has been restarted	43	Plugin posts a message in the incident channel when changing an active stage	44	Workflow member views the active stage by default in incident RHS	45	When an active stage is changed by a different user, the stage selected in the RHS remains as is but without the `Active` annotation	46	In case of only one stage, the stage is set as active by default	47	In case of multiple stages, the first stage is set as active by default	48	Switching between the stages does not set the stage as active	49	Selecting a non-active stage from the RHS dropdown shows a `Make Active` option prompt	50	Selecting an active stage from the RHS dropdown does not show a `Make Active` option prompt	51	Workflow member can check items off of stages that are active	52	Workflow member can uncheck items of stages that are not active	53	Workflow member can check off items of an ongoing incident	54	Workflow member cannot checkoff items of an ended incident	55	Workflow member can checkoff items of a restarted incident
Test Area - View steps in stages during a workflow
Test ID	Test Case	Test Steps	Result	Notes
56	The `Stages` dropdown in the RHS has all the stages for that incident	57	Selecting a different the stage in the RHS shows the checklist corresponding to that stage	58	Saving a playbook with only one stage and leaving the stage name empty, will create an incident with stage “Default Stage”	59	Any workflow member can switch between the stages in the RHS in an ongoing incident	60	Any workflow member can switch between the stages in the RHS in an incident that has been ended	61	Any workflow member can switch between the stages in the RHS in an incident that has been restarted	62	Selecting a different stage will not change the default stage view upon page refresh	Refreshing the page or navigating back to the incident channel will always show the `Active` stage in the RHS and not the stage selected earlier	63	Selecting a different stage will not change the stage view for a different workflow member
Test Area - Assign step owners
Test ID	Test Case	Test Steps	Result	Notes
64	Checklist item in the RHS has a dropdown for assignee	65	The assignee dropdown shows the list of workflow members	66	The assignee dropdown does not show users that are not workflow members	67	By default, checklist item does not have any assignee	68	A checklist item can be checked off without an assignee	69	A checklist item can be unchecked without an assignee	70	Workflow member can select him/herself as the checklist assignee	71	Workflow member can select a different workflow member as the checklist assignee	72	Workflow member can change assignee from him/herself to ‘No Assignee’ for a checklist item	73	Workflow member can change another workflow member to ‘No Assignee’ for a checklist item	74	Plugin posts a message in incident channel when an assignee is selected	75	Plugin posts a message in incident channel when ‘No Assignee’ is selected	76	Workflow member cannot select multiple assignees	77	Workflow member cannot select a different assignee in an incident that is ended	78	Workflow member can select a different assignee in an incident that is restarted	79	Assignee does not change when a workflow member leaves the workflow channel	80	Assignee does not change when a workflow member is kicked from the workflow channel	81	The assignee dropdown list gets updated when a new member is added to the workflow channel	82	The assignee dropdown list gets updated when a workflow member leaves the channel	83	The assignee dropdown list gets updated when a workflow member is kicked from the channel	84	An assignee can checkoff a checklist item	85	An assignee can uncheck a checklist item	86	A non-assignee can checkoff a checklist item	87	A non-assignee can uncheck a checklist item	88	Anything in the backstage about assignee?	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Under “Stage 1”, click “Add new checklist item”. Enter “S1-C1” as the checklist item. Click “Save Playbook”. From the playbook list in the backstage find “PB-Private”. Click to view the edit view. Verify that “Stage 1” has “S1-C1” as the checklist item. Switch to a normal channel view. Start an incident with “PB-Private” to verify the checklist item as been added to “Stage 1”.
22	Any playbook member can edit playbook to add a new checklist item to a stage added by a different member	Continue from the above test. Log in as user-2. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Under “Stage 1”, click “Add new checklist item”. Enter “S1-U2” as the checklist item. Click “Save Playbook”. From the playbook list in the backstage find “PB-Private”. Click to view the edit view. Verify that “Stage 1” has “S1-U2” as the checklist item. Switch to a normal channel view. Start an incident with “PB-Private” to verify the checklist item has been added to “Stage 1”.
23	A playbook member cannot edit playbook such that the playbook name field is empty	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Delete the playbook name “PB-Private”. Verify that the “Save Playbook” button becomes deactivate. Click on the “Save Playbook” button. Verify that the playbook cannot be saved, and the user is still in the edit view. Enter a space in the playbook name field. Repeat steps 5-7.
24	Any playbook member can reorder the checklist items within a stage	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Under “Stage 1”, click the hamburger icon next to the checklist item “S1-C1” and drag it under “S1-U2”. Verify that “S1-C1” is successfully moved under “S1-U2”. Click the hamburger icon next to “S1-U2” and drag it above “S1-C1”. Verify that “S1-U2” is successfully moved under “S1-C1”. Switch to a normal channel view. Start an incident with “PB-Private” to verify the reordering was successful.
25	Any playbook member can reorder the checklist items within different stages	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Under “Stage 1”, click the hamburger icon next to the checklist item “S1-U2” and drag it under the 2nd stage “Stage 2”. Verify that “S1-U2” is successfully moved under “Stage 2”. Switch to a normal channel view. Start an incident with “PB-Private” to verify the reordering was successful.
26	A playbook member cannot reorder the playbook such that the checklist items can be pulled outside a stage	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Click the hamburger icon next to the checklist item “S1-C1” and drag it right above “Stage 1”. Verify that the checklist item does not move from its original spot under “Stage 1”. Drag and drop “S1-C1” outside stage area. Verify that the checklist item does not move from its original spot under “Stage 1”. Switch to a normal channel view. Start an incident with “PB-Private” to verify the order of the checklist items is normal.
27	Any playbook member can delete a checklist item	Continue from the above test. Log in as user-1. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Under “Stage 2” find “S1-U2” and hover the mouse over so that the “X” appears next to it. Click the “X” for “S1-U2”. Verify that the checklist item “S1-U2” is deleted. Click “Save Playbook”. Click “PB-Private” to go to edit view. Verify that the checklist item “S1-U2” is not available in any stage. Switch to a normal channel view. Start an incident with “PB-Private” to verify that “S1-U2” is not available in any stage.
28	A playbook member can edit the playbook to delete all the checklist items such that all the stages can have no checklist item at all	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Click the “X” next to all checklist items of all stages. Verify that all checklist items are deleted. Click “Save Playbook”. Switch to a normal channel view. Start an incident with “PB-Private” to verify that none of stages have any checklist items.
29	Any playbook member can reorder the stages	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Add a new stage “Stage 3”. Click the hamburger icon next to “Stage 1” and drag it below “Stage 3”. Verify that “Stage 1” is successfully moved under “Stage 3”. Click the hamburger icon next to “Stage 3” and drag it above “Stage 2”. Verify that “Stage 3” is successfully moved above “Stage 2”. Switch to a normal channel view. Start an incident with “PB-Private” to verify the stage reordering was successful.
30	A playbook member cannot reorder the playbook such that the stages can be nested	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Click the hamburger icon next to “Stage 1” and drag it under “Stage 3” to align it within the checklist items of “Stage 3”. Verify that “Stage 1” is not moved from its original position. Switch to a normal channel view. Start an incident with “PB-Private” to verify the stage reordering was not successful.		Verify that the stages cannot be stacked inside another stage
31	Any playbook member can delete a stage	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Click the “X” button next to “Stage 3”. Verify that “Stage 3” is deleted. Save playbook. Switch to a normal channel view. Start an incident with “PB-Private” to verify that the incident created does not have “Stage 3”.
32	A playbook member cannot delete the last remaining stage such that the playbook has no stages at all	Continue from the above test. Launch playbook backstage. Find “PB-Private” from the playbook list. Click to open edit view. Click the “X” button next to “Stage 1”. Click the “X” button next to “Stage 2”. Verify that the “Save Playbook” button becomes inactive. Click on the “Save Playbook”. Verify that the user is still in the edit view of “PB-Private” and the action was not saved.
33	Any playbook member can delete playbook	Continue from the above test. Log in as user-2. Launch playbook backstage. Find “PB-Private” from the playbook list. Under “Actions” for the playbook, click the ellipses. Click “Delete”. Verify that a “Confirm Playbook Deletion” modal appears. Click “Delete Playbook”. Verify that “PB-Private” is not present in the playbook list. Switch to a normal channel view. Open up an incident modal. Verify that “PB-Private” is not available in the playbook dropdown.
34	A playbook member can create playbook with a large number of stages	Log in as user-1. Launch playbook backstage. Click “New Playbook”. Enter “PB-N1” as the playbook name. Click “Add new stage” 20 or more times to create a long list of stages. Click “Save Playbook”. Switch to a normal channel. Start an incident using “PB-N1” as the playbook. Verify that the Stages dropdown in the incident RHS has all the stages added in step 5. Verify the Stage dropdown displays the stages properly with scrollbar.		Verify playbook with a large number of stages in the edit page
35	A playbook member can create playbook with a large number of checklist items within the stages	Continue from the above test. Launch playbook backstage. Find “PB-N1” and click to go to edit view. For the first stage, add at least 20 checklist items. Save playbook. Switch to a normal channel. Start an incident using “PB-N1” as the playbook. In the incident RHS, ensure that the first stage is selected from the dropdown. Verify that all the checklist items added in step 3 are available.		Verify playbook with a large number of checklist items within the stages in the edit page
36	The components in the playbook edit page render normally	Verify the appearance of the components in the playbook edit page.

Test Area - Set active workflow stage
Test ID	Test Case	Test Steps	Result	Notes
37	An active stage is annotated with `Active`label in the RHS
38	Non-active stages are not annotated with `Active` labels
39	Incident commander can set a different stage as `Active` in an ongoing incident
40	Workflow member can set a different stage as `Active` in an ongoing incident
41	Workflow member cannot set a different stage as `Active` in an incident that has been ended
42	Workflow member can set a different stage as `Active` in an incident that has been restarted
43	Plugin posts a message in the incident channel when changing an active stage
44	Workflow member views the active stage by default in incident RHS
45	When an active stage is changed by a different user, the stage selected in the RHS remains as is but without the `Active` annotation
46	In case of only one stage, the stage is set as active by default
47	In case of multiple stages, the first stage is set as active by default
48	Switching between the stages does not set the stage as active
49	Selecting a non-active stage from the RHS dropdown shows a `Make Active` option prompt
50	Selecting an active stage from the RHS dropdown does not show a `Make Active` option prompt
51	Workflow member can check items off of stages that are active
52	Workflow member can uncheck items of stages that are not active
53	Workflow member can check off items of an ongoing incident
54	Workflow member cannot checkoff items of an ended incident
55	Workflow member can checkoff items of a restarted incident

Test Area - View steps in stages during a workflow
Test ID	Test Case	Test Steps	Result	Notes
56	The `Stages` dropdown in the RHS has all the stages for that incident
57	Selecting a different the stage in the RHS shows the checklist corresponding to that stage
58	Saving a playbook with only one stage and leaving the stage name empty, will create an incident with stage “Default Stage”
59	Any workflow member can switch between the stages in the RHS in an ongoing incident
60	Any workflow member can switch between the stages in the RHS in an incident that has been ended
61	Any workflow member can switch between the stages in the RHS in an incident that has been restarted
62	Selecting a different stage will not change the default stage view upon page refresh			Refreshing the page or navigating back to the incident channel will always show the `Active` stage in the RHS and not the stage selected earlier
63	Selecting a different stage will not change the stage view for a different workflow member

Test Area - Assign step owners
Test ID	Test Case	Test Steps	Result	Notes
64	Checklist item in the RHS has a dropdown for assignee
65	The assignee dropdown shows the list of workflow members
66	The assignee dropdown does not show users that are not workflow members
67	By default, checklist item does not have any assignee
68	A checklist item can be checked off without an assignee
69	A checklist item can be unchecked without an assignee
70	Workflow member can select him/herself as the checklist assignee
71	Workflow member can select a different workflow member as the checklist assignee
72	Workflow member can change assignee from him/herself to ‘No Assignee’ for a checklist item
73	Workflow member can change another workflow member to ‘No Assignee’ for a checklist item
74	Plugin posts a message in incident channel when an assignee is selected
75	Plugin posts a message in incident channel when ‘No Assignee’ is selected
76	Workflow member cannot select multiple assignees
77	Workflow member cannot select a different assignee in an incident that is ended
78	Workflow member can select a different assignee in an incident that is restarted
79	Assignee does not change when a workflow member leaves the workflow channel
80	Assignee does not change when a workflow member is kicked from the workflow channel
81	The assignee dropdown list gets updated when a new member is added to the workflow channel
82	The assignee dropdown list gets updated when a workflow member leaves the channel
83	The assignee dropdown list gets updated when a workflow member is kicked from the channel
84	An assignee can checkoff a checklist item
85	An assignee can uncheck a checklist item
86	A non-assignee can checkoff a checklist item
87	A non-assignee can uncheck a checklist item
88	Anything in the backstage about assignee?

Test Area (v0.7) - Attach a slash command to each step when editing playbook
Test ID	Test Case	Test Steps	Result	Notes
89	A newly added checklist item in a playbook has a slash command field
90	A playbook can be edited to add a slash command to any checklist item
91	A checklist item configured with a slash command shows the slash command for that item in the RHS
92	A checklist item can only be configured with one slash command at most
93	Typing an invalid slash command

Test Area (v0.7) - Browse existing slash commands in playbook editing
Test ID	Test Case	Test Steps	Result	Notes
	Typing a `/` in the slash command field brings up a list of existing slash commands
	Typing a non-/ character in the slash command field?			Should this be allowed?
	Selecting from the slash command list auto completes the slash command
	Typing a non-existing slash command shows a “invalid slash command” error
	The slash command field can be left empty

Notes

Test Area (v0.7) - Execute the attached slash command with a button click
Test ID	Test Case	Test Steps	Result	Notes
	89	A playbook can be edited to add a slash command to any checklist item	90	A newly added checklist item in a playbook has a slash command field	91	checklist item configured without a slash command does not have a `Run` button next to it
	A checklist item configured with a slash command shows the slash command for that item in the RHS	92	valid slash command has a `Run` button next to it
	A checklist item can only be configured with one an invalid slash command at most	93	In case of two valid slash commands, the running the command from the checklist item will only run the first command successfully	This is in sync with the core feature behavior	94	Typing an invalid slash command	95	Running an invalid slash command from RHS checklist shows an ‘invalid slash command’ error	Like core feature?	96	A slash command can be run multiple times from the RHS checklist	97	A plugin post notifies when a slash command fails to run successfully	98	https://mattermost.atlassian.net/browse/MM-26733 : “any textbox” what is this?
Test Area - Browse existing slash commands in playbook editing
Test ID	Test Case	Test Steps	Result	Notes
99	Typing a `/` in the slash command field brings up a list of existing slash commands	100	Typing a non-/ character in the slash command field?	Should this be allowed?	101	Selecting from the slash command list auto completes the slash command	102	Typing a non-existing slash command shows a “invalid slash command” error	103	The slash command field can be left empty
Test Area - Execute the attached slash command with a button click
Test ID	Test Case	Test Steps	Result	does not have a `Run` button next to it
	A checklist item configured with a nonexistent slash command does not have a `Run`button next to it
	Clicking the `Run` button for a checklist slash command runs the slash command
	Plugin posts a message in the incident channel when a slash command `Run`is clicked			A message that the checklist item slash command was run (not the slash command system message)
	Running a slash command with system message attached to it post the slash command system message in the incident channel
	In case of two valid slash commands, the running the command from the checklist item will only run the first command successfully			This is in sync with the core feature behavior
	Running an invalid slash command from RHS checklist shows an ‘invalid slash command’ error			Like core feature?
	A slash command can be run multiple times from the RHS checklist
	A plugin post notifies when a slash command fails to run successfully

Test Area - Tests/bugs outside of requirements and user stories
Test ID	Test Case	Test Steps	Result	Notes

Version	Old Version 2	New Version 3
Changes made by	Prapti Shrestha (Deactivated)	Prapti Shrestha (Deactivated)
Saved on	Aug 13, 2020	Aug 15, 2020

Versions Compared

Key

Document version 0.0

Status: Test Development in Progress