Technical Specification - Granular Data Retention

Summary

Adds the ability to configure custom (granular) deletion policies for posts for specific teams and/or channels. When the daily retention job runs it selectively deletes posts by age based on the configuration.

The granular policies completely override the global setting.

The feature requires some new APIs, some new tables, and an edit to the existing data retention job logic.

API

Global Policy (existing API)

GET /api/v4/data_retention/policy retrieves the global retention settings.

{
    "message_deletion_enabled": false,
    "file_deletion_enabled": false,
    "message_retention_cutoff": 0,
    "file_retention_cutoff": 0
}

The JSON object reflects the fields in the DataRetentionSettings config object for:

• EnableMessageDeletion

• EnableFileDeletion

• MessageRetentionDays

• FileRetentionDays

Updates to the global retention policy continues to be managed via the config APIs.

Retention Policies

POST /api/v4/retention_policies create a new retention policy.

Request:

{
    "display_name": "foo",
    "post_duration": 4
}

201 response:

{
    "id": "m8zoumpj9pn9zexospoxi5dzoc",
    "display_name": "foo",
    "post_duration": 4
}

PUT /api/v4/retention_policies/:policy_id/patch patches a retention policy.

Request:

{
    "display_name": "foo2",
    "post_duration": 365
}

200 Response:

{
    "id": "m8zoumpj9pn9zexospoxi5dzoc",
    "display_name": "foo2",
    "post_duration": 365
}

GET /api/v4/retention_policies/:policy_id gets a retention policy by id.

{
    "id": "m8zoumpj9pn9zexospoxi5dzoc",
    "display_name": "foo2",
    "post_duration": 365,
    "teams": [
        {
            "id": "z7rxbxbfb7yxdydxzi8pestath",
            "display_name": "My Team 1"
        }
    ],
    "channels": [
        {
            "id": "z7rxbxbfb7yxdydxzi8pestath",
            "display_name": "Off Topic",
            "team_display_name": "My Team 1"
        }
    ]
}

DELETE /api/v4/retention_policies/:policy_id deletes a retention policy.

Deleting a RetentionPolicies record also deletes all of the associated RetentionPoliciesChannels and RetentionPoliciesTeams records.

GET /api/v4/retention_policies lists all retention policies, including associated teams and channels.

{
    "policies": [
        {
            "display_name": "foo",
            "post_duration": 2,
            "teams": [
                {
                    "id": "z7rxbxbfb7yxdydxzi8pestath",
                    "display_name": "My Team 1"
                }
            ],
            "channels": [
                {
                    "id": "z7rxbxbfb7yxdydxzi8pestath",
                    "display_name": "Off Topic",
                    "team_display_name": "My Team 1"
                }
            ]
        }
    ],
    "total_count": 1
}

Retention Policies Teams

POST /api/v4/retention_policies/:policy_id/teams associates a team to a retention policy.

Request:

{
    "team_id": "z7rxbxbfb7yxdydxzi8pestath"
}

DELETE /api/v4/retention_policies/:policy_id/teams/:team_id removes a team from a retention policy.

Retention Policies Channels

POST /api/v4/retention_policies/:policy_id/channels associates a channel to a retention policy.

Request:

{
    "channel_id": "z7rxbxbfb7yxdydxzi8pestath"
}

DELETE /api/v4/retention_policies/:policy_id/channels/:channel_id deletes a channel from a retention policy.

Database

RetentionPolicies table

RetentionPoliciesChannels table

TBD: Does ChannelId need to be indexed?

RetentionPoliciesTeams table

TBD: Does TeamId need to be indexed?

Model

• Rename DataRetentionPolicy to GlobalDataRetentionPolicy

• Add RetentionPolicy representing a record in the RetentionPolicies table.

• Add RetentionPolicyChannel representing a record in the RetentionPoliciesChannels table.

• Add RetentionPolicyTeam representing a record in the RetentionPoliciesTeams table.

Enterprise

Changes to various methods on DataRetentionWorker in data_retention/worker.go are required.

We need a query to determine the Posts records to be deleted. Something like this (untested):

-- untested
SELECT
    Posts.Id
FROM
    Posts
    JOIN Channels ON Channels.Id = Posts.ChannelId
    JOIN Teams ON Teams.Id = Channels.TeamId
WHERE
    (
        Posts.CreateAt > @data_retention_channel1_timestamp
        AND ChannelId = @data_retion_channel1_channel_id
    )
    OR (
        Posts.CreateAt > @data_retention_channel2_timestamp
        AND ChannelId = @data_retion_channel2_channel_id
    )
    OR (
        Posts.CreateAt > @data_retention_team1_timestamp
        AND Teams.Id = @data_retion_team1_team_id
        AND ChannelId NOT IN (@data_retention_channels_channel_ids)
    )
    OR (
        Posts.CreateAt > @data_retention_team2_timestamp
        AND Teams.Id = @data_retion_team2_team_id
        AND ChannelId NOT IN (@data_retention_channels_channel_ids)
    )
    OR (
        Posts.CreateAt > @global_retention_timestamp
        AND ChannelId NOT IN (@data_retention_channels_channel_ids)
        AND Teams.Id NOT IN (@data_retention_teams_team_ids)
    );

The post ids returned from that query can be used to delete:

• Reactions

• Preferences (of Category ‘flagged_post’)

• Threads

• ThreadMemberships

• LinkMetadata TBD: these aren’t currently purged via data retention, but they probably should be.

FileInfo (and their associated files on disk) and ChannelMemberHistory continue to be deleted system-wide with no new granularity.

TBD: Why is ChannelMemberHistory currently deleted as part of message retention days settings?

Mobile

No changes. Existing file-deleted and post-deleted UX covers all.