...
...
...
...
...
...
...
...
...
...
...
...
...
...
Document version 1.
...
1
Status:
...
Tests Done.
Document Version | Description | Date |
---|---|---|
0.0 | Initial version | 05/11 |
0.1 | Test development done. In review. | 05/13 |
0.2 |
| 05/15 |
1.0 | Test in progress | 05/20 |
1.1 | Tests done | 05/1920 |
References
...
The following setup will be necessary in order to begin testing:
A Mattermost v5.23 test server: https://ir-test-523.test.mattermost.cloud/
Build Hash: ed1576d007a09b07574b18784b9eabb3d8c9db53
The test server is equipped with an E10 E20 license for the plugin to be installed.
Incident Plugin v0.4.0: plugin-incident-response-0.4.0-alpha.2
Channel Export Plugin v0.2
Test Server:
Test Server (upgrade tests):
Build Hash:
...
Upgrade test server: https://ir-522.test.mattermost.cloud/
Test Date: 05/20/2020
Tests
Test Area - Plugin Setup | ||||
Test ID | Test Case | Test Procedure | Result | Notes |
---|---|---|---|---|
1 | Admin can install plugin in a non-EE instance |
| Pass | |
2 | Admin can install plugin in an E10 instance |
| Pass | |
3 | Admin can install plugin in an E20 instance |
| Pass |
|
4 | Admin can enable plugin |
| Pass |
|
5 | Plugin functionality is available to admin |
| Pass |
|
6 | Plugin functionality is available to normal user |
| Pass |
|
7 | Admin can disable plugin |
| Pass |
|
8 | Admin can remove plugin |
| Pass |
|
9 | Plugin upgrades normally from v0.3 to v0.4 |
| Pass | |
10 | Incidents started with older plugin version are still active and available in v0.4 |
| Pass |
|
+11 | Incidents started with older plugin version are displayed correctly in the backstage incident list view | Continue from the above test.
| Pass | |
+12 | Incidents started with older plugin version info is correct in the backstage incident summary page | Continue from the above test.
| Pass | |
1113 | Incidents ended with plugin v0.3 are not available in v0.4 |
| Pass |
|
1214 | Playbook started with the older plugin version is still available upon plugin upgrade |
| Pass |
Login as an admin user
Enable the Incident Response plugin
Navigate to System Console > PLUGINS > Plugin Management.
In “Installed Plugins” find Incident Response and click “Enabled”.
Log out.
Test Area - Monitor As a user of a team, I can see a list of all past and current incidents within the team so that the information is not lost. | ||||||
---|---|---|---|---|---|---|
Test ID | Test Case | Test Procedure | Result | Notes | ||
1315 | Clicking playbook button on RHS opens Playbook backstage by default |
| 14 | Backstage shows a tab for incidents | Pass | |
16 | Clicking “Incidents & Playbooks Backstage” in the main menu opens Incidents backstage |
| Pass | |||
17 | Backstage shows a tab for incidents | Continue from the above test.
| Pass | |||
1518 | Clicking on “Incidents” tab on backstage LHS highlights the Incidents tab | Continue from the above test.
| Pass | |||
1619 | Clicking on “Incidents” tab on backstage LHS loads the list of all incidents | Continue from the above test.
| Pass | |||
1720 | Incidents list view displays the team name the incidents belong to | Continue from the above test.
| Pass | |||
1821 | Incident list in the backstage displays the following columns: name, status, start timestamp, end timestamp and the commander | Continue from the above test.
| Pass | |||
1922 | The incident list in the backstage shows all active incidents that belong to the current team only |
| Pass | |||
2023 | The incident list in the backstage shows all inactive incidents that belong to the current team only |
| Pass | |||
2124 | Active incidents show an “Ongoing” status in the | Continue from the above test.
| Pass | |||
2225 | Inactive incidents (ended incidents) show an “Ended” status in the | Continue from the above test.
| Pass | |||
2326 | The incident list in the backstage shows updated end timestamp for an incident | Continue from the above test.
| Pass | |||
27 | An active incident shows “--” in End Date column | Continue from the above test.
| Pass | |||
2528 | The incident list in the backstage shows updated commander for an incident |
| Pass | |||
2629 | The incident names in the “Name” column render as clickable links | Continue from the above test.
| Fail | |||
30 | Long incident name wraps in the “Name” column displaying the entire incident name |
| Fail | |||
As the incident plugin, the channel is automatically archived when the incident ends so that it reduces clutter. | 28||||||
31 | Channel is archived when ending an incident with slash command |
| N/A | Moved to 0.5 | ||
32 | Channel is archived when ending an incident from RHS |
| N/A | Moved to 0.5 | ||
33 | The incident channel shows up under archived channel once the incident is ended |
| 31 | N/A | Moved to 0.5 | |
34 | Incident channels do not appear in LHS when archived channel display is turned off |
| N/A | Moved to 0.5 |
Test Area - Coordinate As a user, I can export channel transcript from the incident detail page so that it can be saved for record | |||||||
---|---|---|---|---|---|---|---|
Test ID | Test Case | Test Procedure | Result | Notes | |||
3235 | When Channel Export plugin is not installed, the “Export Incident Channel” link is inactive |
| Pass | ||||
3336 | When Channel Export plugin is not installed, hovering over the “Export Incident Channel” link shows a message to “install the channel export plugin” | Continue from the above test.
| Pass | ||||
37 | When Channel Export plugin is installed in an instance with no EE license, the “Export Incident Channel” link is inactive |
| 35Pass | ||||
38 | When Channel Export plugin is installed in an instance with no EE license, hovering on the “Export Incident Channel” link shows a message that “E20 license is required” | Continue from the above test.
| 36 | Pass | |||
39 | When Channel Export plugin is installed in an instance with an E10 license, the “Export Incident” link is inactive |
| 37Pass | ||||
40 | When Channel Export plugin is installed in an instance with an E10 license, hovering on the “Export Incident Channel” link shows a message that “E20 license is required” | Continue from the above test.
| 38Pass | ||||
41 | When Channel export plugin is installed but not enabled, the “Export Incident Channel” link is inactive | When Channel
| 39 | Pass | |||
42 | When Channel Export plugin is installed but not enabled, hovering over the “Export Incident Channel” shows “enable the plugin” message | Continue from the above test.
| 40Fail | ||||
43 | When Channel Export plugin is installed and enabled, an active “Export Incident Channel” link and icon is available in the incident details view as an active link |
| Pass | 41||||
44 | When Channel Export plugin is installed and enabled, hovering over the “Export Incident Channel” link shows a message that the action will download the channel content | Continue from the above test.
| N/A | I think I added this on my own for consistency, but is this valid? | |||
4245 | A system admin can export the incident channel the admin is a member of by clicking “Export Incident Channel” link |
| Pass | The features of the Channel Export plugin were tested in Test Plan - Channel Export v0.0.1 | |||
4346 | A system admin can export the incident channel the admin is NOT a member of by clicking “Export Incident Channel” link |
| Fail | ||||
47 | An incident member can export the incident channel by clicking “Export Incident Channel” link |
| Pass | ||||
48 | The exported csv file captures the current status contents of the incident channel |
| Pass | This feature was tested in Test Plan - Channel Export v0.0.1 46 | |||
49 | “Export Incident Channel” button is inactive for non-incident user |
| N/A | 47||||
50 | A non-incident user sees a “no access” message upon hovering on the “Export Incident Channel” button | Continue from the above test.
| N/A | Don’t think this test is valid anymore since a non-member cannot see the summary at all in the backstage. Can someone confirm? | |||
4851 | Exporting an ended incident channel from backstage exports the |
| Pass | 49||||
52 | Non-member cannot export an ended incident channel |
| N/A | Same as #47. Don’t think this test is valid anymore since a non-member cannot see the summary at all in the backstage. Can someone please confirm? | |||
5053 | Verify the appearance of Once the Channel Export plugin is disabled, the “Export Incident Channel” link and icon appear in dark theme |
| 51 | Verify the appearance of the “Export Incident should deactivate again |
| Fail | |
54 | Verify the appearance of the “Export Incident Channel” link and icon appear in dark theme |
| Fail | ||||
55 | Verify the appearance of the “Export Incident Channel” link and icon appear in light theme |
| Pass | ||||
5256 | Verify the appearance of the “Export Incident Channel” link and icon appear in other themes |
| Fail | ||||
As the incident commander, I receive an incident summary and a link to more info after ending an incident so that I know where to go for post-mortem. | |||||||
Test Area - Measure As a user, I can see a summary of the following on incident detail page so that I can get the gist at a glance:
| ||||||
---|---|---|---|---|---|---|
Test ID | Test Case | Test Procedure | Result | Notes | ||
5357 | User cannot view incident summary if the user is not an incident member |
| Pass | Is this test case correct since we’re limiting a non-user’s access to the incident summary. | ||
5458 | From the incidents list in the backstage, user can navigate to incident summary upon clicking the incident name of an active incident if the user is an incident member |
| Pass | |||
5559 | From the incidents list in the backstage, user can navigate to incident summary upon clicking the incident name of an inactive incident |
| Pass | |||
5660 | Incident summary page displays the incident name as the page heading | Continue from the above test.
| Pass | |||
5761 | Incident summary page has a “<“ button that brings a user back to the incidents list view | Continue from the above test.
| Pass | |||
5862 | Incident summary page bears a link to the incident channel | Continue from the above test.
| Pass | |||
5963 | Hovering on the incident channel link shows a tooltip suggesting that it’s a shortcut to the incident channel | Continue from the above test.
| Pass | |||
6064 | Clicking on incident channel link of active incidents brings the user to the incident channel | Continue from the above test.
| Pass | 61|||
65 | Clicking on incident channel link of inactive incidents, when |
| N/A | |||
66 | Clicking on incident channel link of inactive incidents, when | N/A | ??? Should the link be inactive and in greyscale if false? | |||
6367 | Incident summary shows the current status of the incident |
| Pass | |||
6468 | Incident summary shows the current commander of the incident |
| Pass | |||
6569 | For active incidents, incident summary shows the duration of how long the incident has been active for |
| Pass | |||
6670 | For inactive incidents, incident summary shows the duration of how long the incident remained active for before the incident ended |
| Pass | |||
6771 | For active incidents, incident summary shows the current number of members in the incident channel | Continue from the above test.
| Pass | |||
72 | For an inactive incident, incident summary shows the number of | member members in the incident channel when the incident was ended | Continue from the above test.
| in the incident.69
| ??? This will need to be tested once the channel-archive-on-ending feature is finalized | |
73 | For active incident, incident summary shows the total number of messages in the incident channel |
| What counts as a message? Do bot messages count? In that case, some numbers are not showing up correctly. 70 Why is the first one 2 messages? Shouldn’t it be 1? | |||
74 | For inactive incident, incident summary shows the total number of messages in the archived channel of the inactive incident | Continue from the above test.
| This will need to be tested once the channel-archive-on-ending feature is finalized |
Test Area - Tests/bugs outside of user stories | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Test ID | Test Case | Test Procedure | Result | Ticket | Notes | ||||||
75 | Playbook name too longPlaybook name empty | ||||||||||
76 | Clicking checklist item timestamp should permalink jump to corresponding post |
| Pass | ||||||||
77 | Add a message indicating no checklist items in Incidents RHS | Moved to 0.5 | |||||||||
78 | Display incident checklist timeline graph | Ticket is still open | |||||||||
79 | Support incident list pagination |
| Moved to 0.5 | ||||||||
80 | Support incident list sorting on End Date |
| the incident backstage to view the incident list.Verify that incidents are listed with the newest one on top by default
| in the header
| last one that was ended
| descending
| descending
| .Click “End Date” again
| Moved to 0.5 | ||
81 | Support incident list sorting on Start Date | Continue from the test for “Support incident list sorting on End Date”
| end
| ‘Ended’
| ended, followed by the ‘Ongoing’ incidents (ascending order of creation?)
| correctly.
| Moved to 0.5 | ||||
82 | Omit playbooks table header if empty | Ticket is still open | |||||||||
83 | Reduce in-channel checklist notifications | Pass | Continue from the test for “Support incident list sorting on End Date” Click “Start Date” in the header. Verify that the incident list is rearranged such that the first incident to get created is now at the top of the list, followed by other incidents (ongoing or ended) in ascending order of the time the incidents were created. | Support incident list sorting on Start Date | |||||||
84 | Desktop: Channel Export does not download file | Can still repro | |||||||||
85 | Checklist items can be created with empty string | Pass | |||||||||
86 | Filter incidents to active team | Pass | https://mattermost.atlassian.net/browse/MM-24581Omit playbooks table header if empty-25024 | From ticket desc: The incidents list in the backstage currently provides no `teamId` to the server, resulting in the admin seeing incidents across all teams, but non-admins seeing nothing. | |||||||
87 | Incident list on RHS is empty on Safari | Pass | https://mattermost.atlassian.net/browse/MM-24721Reduce in-channel checklist notifications24743 | ||||||||
88 | Playbook can be created with an empty string char | Pass | https://mattermost.atlassian.net/browse/MM-24731Desktop: Channel Export does not download file24907 | ||||||||
89 | IR: Playbook with empty name is created |
Observed: Playbook is created with an empty name. | Pass | https://mattermost.atlassian.net/browse/MM-25257Checklist items can be created with empty string24904 | |||||||
90 |
| Filter incidents to active team |
Duplicate of https://mattermost.atlassian.net/browse/MM-25024 | From ticket desc: The incidents list in the backstage currently provides no `teamId` to the server, resulting in the admin seeing incidents across all teams, but non-admins seeing nothing. | Incident list on RHS is empty on Safari25257 | ||||||
91 | Long incident name pushes the components of the header outside the normal view | https://mattermost.atlassian.net/browse/MM-24743Playbook can be created with an empty string char25400 | |||||||||
92 | Clicking checklist item timestamp when viewing the "item checked" post in incident channel reloads the page | IR: Playbook with empty name is created |
Observed: Playbook is created with an empty name. | ||||||||
93 |
...
IR (v0.4.0) Bugs:
Jira Legacy | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|