Creating multiple incidents at the same time (concurrently) causes all_headers to be overwritten

Description

I discovered this while load-testing the incident response plugin. This behaviour is documented however it is more pronounced while load-testing and haven't been observed in production yet.

The root cause of the issue is that the plugin does not update `all_headers` atomically because of which it sometimes overwrites the previous value that may be updated concurrently. This is a known limitation of the KVStore.Set API, which is why `KV.Set` with KVOptions was introduced.

This issue may become irrelevant if plugin switches to using database directly.

Code in question: https://github.com/mattermost/mattermost-plugin-incident-response/blob/f1ac32ac1bb7d2df1172f9617aaa0544d9b644c0/server/pluginkvstore/incident_store.go#L216

Steps to reproduce:
1. Clear out plugin db
2. Use load-testing tool to start stress testing the plugin
3. Observe

QA Test Steps

None

Mana

None

Assignee

Unassigned

QA Assignee

None

Reporter

Ali Farooq

Epic Link

None

Fix versions

Mattermost Team

Workflows

Sprint

None

QA Testing Areas

None

GitHub Issue

None

Components

Severity

None
Configure