Document version 0.1
Status: Test in Progress…
...
Test Area - Plugin Setup | ||||
---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes |
1 | Admin can install plugin in a non-EE instance |
| ||
2 | Admin can install plugin in an E10 instance |
| ||
3 | Admin can install plugin in an E20 instance |
| Pass | |
4 | Admin can enable plugin |
| Pass | |
5 | Plugin functionality is available to admin |
| Pass | |
6 | Plugin functionality is available to normal user |
| Pass | |
7 | Admin can disable plugin |
| Pass | |
8 | Admin can remove plugin |
| Pass | |
9 | Plugin upgrades normally from v0.4 to v0.6 |
| ||
10 | Plugin upgrades normally from v0.5 to v0.6 |
| ||
11 | Incidents started with the prior plugin versions are still active and available in v0.6 |
| ||
12 | Incidents ended with prior plugin versions are not available in v0.6 |
| ||
13 | Playbook started with the older plugin version is still available upon plugin upgrade |
|
Test Area - Create stages when editing a playbook | |||||||
---|---|---|---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes | |||
14 | Playbook creator can edit playbook |
| Pass | ||||
15 | Any playbook member can edit playbook to add a new member | Continue from the above test.
| Pass | ||||
16 | Any playbook member can edit playbook to remove an existing member | Continue from the above test.
| Pass | ||||
17 | A playbook member cannot edit playbook to remove all the members of the playbook | Continue from the above test.
| Pass | The “Save Playbook” button should stay inactive unless there is at least one member | |||
18 | Any playbook member can edit playbook to change the private/public incident setting | Continue from the above test.
| Pass | ||||
19 | Any playbook member can edit playbook to add a new stage | Continue from the above test.
| Pass | ||||
20 | A playbook can have multiple stages with the same stage name |
| Pass | Valid? Ian Tao (Unlicensed) confirm? | |||
21 | A playbook member can edit playbook to add a new checklist item to a stage added by themself | Continue from the above test.
| Pass | ||||
22 | Any playbook member can edit playbook to add a new checklist item to a stage added by a different member | Continue from the above test.
| Pass | ||||
23 | A playbook member cannot edit playbook such that the playbook name field is empty | Continue from the above test.
| Pass | ||||
24 | Any playbook member can reorder the checklist items within a stage | Continue from the above test.
| Pass | ||||
25 | Any playbook member can reorder the checklist items within different stages | Continue from the above test.
| Pass | Cannot be reordered under a different stage without a step: | |||
26 | A playbook member cannot reorder the playbook such that the checklist items can be pulled outside a stage | Continue from the above test.
| Pass | ||||
27 | Any playbook member can delete a checklist item | Continue from the above test.
| Pass | ||||
28 | A playbook member can edit the playbook to delete all the checklist items such that all the stages can have no checklist item at all | Continue from the above test.
| N/A | Invalid. | |||
29 | Any playbook member can reorder the stages | Continue from the above test.
| Pass | ||||
30 | A playbook member cannot reorder the playbook such that the stages can be nested | Continue from the above test.
| Pass | Verify that the stages cannot be stacked inside another stage | 31 | Any playbook member can delete a stage | Step 5: The dragged item gets dropped below (not inside) whatever stage item it was dragged into. |
31 | Any playbook member can delete a stage | Continue from the above test.
| Pass | ||||
32 | A playbook member cannot delete the last remaining stage such that the playbook has no stages at all | Continue from the above test.
| Fail | ||||
33 | Any playbook member can delete playbook | Continue from the above test.
| Pass | ||||
34 | A playbook member can create playbook with a large number of stages |
| Pass | Verify playbook with a large number of stages in the edit page | |||
35 | A playbook member can create playbook with a large number of checklist items within the stages | Continue from the above test.
| Pass | Verify playbook with a large number of checklist items within the stages in the edit page | |||
36 | The components in the playbook edit page render normally | Verify the appearance of the components in the playbook edit page. | Pass |
Test Area - Set active workflow stage | ||||
---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes |
37 | An active stage is annotated with |
| Pass | |
38 | Non-active stages are not annotated with | Continue with the above test.
| Pass | |
39 | Selecting a non-active stage from the RHS dropdown shows a | Continue with the above test.
| Pass | |
40 | Incident commander can set a different stage as | Continue with the above test.
| Pass | |
41 | Workflow member can set a different stage as | Continue with the above test.
| Pass | |
42 | Workflow member | Continue with the above test.
| Fail | Ian Tao (Unlicensed) confirm? Updated to “can set a different stage…” |
43 | Workflow member can set a different stage as | Continue with the above test.
| Pass | |
44 | Plugin posts a message in the incident channel when changing an active stage | Continue from the above test.
| Pass | |
45 | Workflow member views the active stage by default in incident RHS | Continue with the above test.
| Pass | |
46 | When an active stage is changed by a different user, the stage selected in the RHS remains as is but without the | Continue with the above test.
| Pass | |
47 | In case of only one stage, the stage is set as active by default |
| Pass | |
48 | In case of multiple stages, the first stage is set as active by default |
| Pass | |
49 | Switching between the stages does not set the stage as active | Continue from the above test.
| Pass | |
50 | Selecting an active stage from the RHS dropdown does not show a | Continue from the above test.
| Pass | |
51 | Workflow member can check items off of stages that are active | Continue from the above test.
| Pass | |
52 | Workflow member can uncheck items of stages that are not active | Continue from the above test.
| Pass | |
53 | Workflow member can check off items of an ongoing incident | Pass | Covered by the above test | |
54 | Workflow member | Continue from the above test.
| Pass | Valid? Ian Tao (Unlicensed) confirm? Updated to “can checkoff…” |
55 | Workflow member can checkoff items of a restarted incident | Continue from the above test.
| Pass |
Test Area - View steps in stages during a workflow | ||||
---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes |
56 | The | Pass | Covered by #34 | |
57 | Selecting a different the stage in the RHS shows the checklist corresponding to that stage |
| Pass | |
58 | Saving a playbook with only one stage and leaving the stage name empty, will create an incident with stage “Default Stage” |
| Pass | |
59 | Any workflow member can switch between the stages in the RHS in an ongoing incident | Continue from the above test.
| Pass | |
60 | Any workflow member can switch between the stages in the RHS in an incident that has been ended | Continue from the above test.
| Pass | |
61 | Any workflow member can switch between the stages in the RHS in an incident that has been restarted | Continue from the above test.
| Pass | |
62 | Selecting a different stage will not change the default active stage view upon page refresh | Continue from the above test.
| Pass | Refreshing the page or navigating back to the incident channel will always show the |
63 | Selecting a different stage will not change the stage view for a different workflow member | Continue from the above test.
| Pass | |
64 | Stage title should state the current and total number of stages as X/Y | Continue from the above test.
| Pass |
Test Area - Assign step owners | ||||
---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes |
65 | Checklist item in the RHS has a dropdown for assignee | Continue from the above tests.
| Pass | |
66 | By default, checklist item does not have any assignee | Continue from the above tests.
| Pass | |
67 | The assignee dropdown shows the list of workflow members | Continue from the above tests.
| Pass | |
68 | The assignee dropdown does not show users that are not workflow members | Continue from the above test.
| Pass | |
69 | A checklist item can be checked off without an assignee | Continue from the above test.
| Pass | |
70 | A checklist item can be unchecked without an assignee | Continue from the above test.
| Pass | |
71 | Workflow member can select him/herself as the checklist assignee | Continue from the above test.
| Pass | |
72 | Workflow member can select a different workflow member as the checklist assignee | Continue from the above test.
| Pass | |
73 | Workflow member can change assignee from him/herself to ‘No Assignee’ for a checklist item | Continue from the above test.
| Pass | |
74 | Workflow member can change another workflow member to ‘No Assignee’ for a checklist item | Continue from the above test.
| Pass | |
75 | Plugin posts a message in incident channel when an assignee is selected | Continue from the above tests.
| Pass | |
76 | Plugin posts a message in incident channel when ‘No Assignee’ is selected | Continue from the above tests.
| Pass | |
77 | Workflow member cannot select multiple assignees | Continue from the above tests.
| Pass | |
78 | Workflow member cannot select a different assignee in an incident that is ended | Continue from the above tests.
| Pass | |
79 | Workflow member can select a different assignee in an incident that is restarted | Continue for the above tests.
| Pass | |
80 | Assignee does not change when a workflow member leaves the workflow channel | Continue form the above tests.
| Pass | |
81 | The assignee dropdown list gets updated when a workflow member is kicked from the channel | Continue from the above tests.
| Pass | |
82 | Assignee does not change when a workflow member is kicked from the workflow channel | Continue from the above tests.
| Pass | |
83 | The assignee dropdown list gets updated when a workflow member leaves the channel | Continue from the above tests.
| Pass | |
84 | The assignee dropdown list gets updated when a new member is added to the workflow channel | Continue from the above tests.
| Pass | |
85 | An assignee can checkoff a checklist item | Continue from the above tests.
| Pass | |
86 | An assignee can uncheck a checklist item | Continue from the above tests.
| Pass | |
87 | A non-assignee can checkoff a checklist item | Continue from the above tests.
| Pass | |
88 | A non-assignee can uncheck a checklist item | Continue from the above tests.
| Pass | |
89 |
| N/A | At the moment you can’t assign an item in the backstage, but that seems useful. Ian Tao (Unlicensed) ? |
Test Area (v0.7) - Attach a slash command to each step when editing playbook | ||||
---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes |
90 | A newly added checklist item in a playbook has a slash command field |
| Pass | |
91 | A playbook can be edited to add a slash command to any checklist item | Continue from the above test.
| Pass | |
92 | A checklist item configured with a slash command shows the slash command for that item in the RHS | Continue from the above tests.
| Pass | |
93 | A checklist item can only be configured with one slash command at most |
| N/A | Valid? I think this should be allowed (there’s no technical reason it can’t). Ian Tao (Unlicensed) ? |
94 | Typing an invalid slash command | N/A | There’s nothing preventing the user from typing an invalid slash command. |
Test Area (v0.7) - Browse existing slash commands in playbook editing | ||||
---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes |
95 | Typing a |
| Pass | |
96 | Typing a non-/ character in the slash command field? | Should this be allowed?N/A | See #94 | |
97 | Selecting from the slash command list auto completes the slash command | Continue from the above tests.
| Pass | |
98 | Typing a non-existing slash command shows a “invalid slash command” error | N/A | Valid? We do not validate the slash commands at the moment. | |
99 | The slash command field can be left empty | Continue from the above tests.
| Pass |
Test Area (v0.7) - Execute the attached slash command with a button click The tests below might need to get adjusted (or added) to match Revised step execution: https://mattermost.atlassian.net/browse/MM-27012 | |||||
---|---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes | |
100 | A checklist item configured without a slash command does not have a |
| Pass | ||
101 | A checklist item configured with a valid slash command has a | Continue from the above test.
| Pass | ||
102 | A checklist item configured with an invalid slash command does not have a | Continue from the above test.
| N/A | This doesn’t happen currently. It would require us to spend mana on validating the slash command. | |
103 | A checklist item configured with a nonexistent slash command does not have a | Continue from the above test.
| N/A | See #102 | |
104 | Clicking the | Continue from the above test.
| Pass | ||
105 | Plugin posts a message in the incident channel when a slash command | Continue from the above test.
| N/A Ticket created | A message that the checklist item slash command was run (not the slash command system message) Ian Tao (Unlicensed) should we make a ticket for this feature? We currently don’t do it, but it makes sense. | |
106 | Running a slash command with system message attached to it post posts the slash command system message in the incident channel | Continue from the above test.
| Pass | ||
107 | In case of two valid slash commands, | the running the command from the checklist item will only run the first command successfully | Continue from the above test.
| Pass | This is in sync with the core feature behavior Seems correct to me (ChrisP) |
108 | Running an invalid slash command from RHS checklist shows an ‘invalid slash command’ error | Depends on the steps in #102. | Like core feature?N/A | ||
109 | A slash command can be run multiple times from the RHS checklist | Continue from the above tests.
| Pass | ||
110 | A plugin post notifies when a slash command fails to run successfully | Verify that when a slash command fails to execute by clicking the | Fail |
Test Area - Tests/bugs outside of requirements and user stories | ||||
---|---|---|---|---|
Test ID | Test Case | Test Steps | Result | Notes |
111 | Checklist item description |
| Pass | |
112 | Add pagination to playbook list |
| Pass | |
113 | Browse Playbook templates |
| Pass | |
114 | Channel link capability to the workflow markdown | A channel link in the checklist text (it has to be correct), such as
| Pass | |
115 | Test new playbook creation with Blank Playbook template | Sanity tests with Blank Playbook template. | Pass | |
116 | Test new playbook creation with Incident Response Playbook | Sanity tests with Incident Response Playbook template. | Pass | |
117 | Show workflow duration |
| Pass | |
118 | Return paginated response for all list endpoints |
| Pass | |
119 | Empty playbook landing |
| Pass | |
120 | Sortable names, stages, steps in playbook backstage |
| Pass | |
121 |
| Pass |
Other open bugs and issues:
...