CleanUpLabels has a hardcoded set of labels to remove from. Automerge is one of them. However, this label is already removed when the PR is merged. Either we stop removing it on merge, or not remove it on cleanup. The previous is preferred because it will cleanup labels even when a PR is merged manually.
No testing required.