Feat: Use built-in priority classes for critical EKS-A components and limit the consumption of these classes for normal workload by default

[Cajga]

What would you like to be added:
By default, Kubernetes ships with two priority classes which is also true for EKS-A:

$ kubectl get priorityclasses.scheduling.k8s.io -A
NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            29d
system-node-critical      2000001000   false            29d

These should be used for critical components/add-ons of the cluster to make sure that these components are not evicted first in case of a resource pressure.

Currently, EKS-A does not use these priority classes for all of it's infra components:

$ kubectl get pods -n kube-system kube-vip-kls107 -o yaml|grep priority
  priority: 0
$ kubectl get pods -n eksa-system eksa-controller-manager-6bb5cb4fb4-lx8q5 -o yaml|grep -i priori
  preemptionPolicy: PreemptLowerPriority
  priority: 0
$ kubectl get pods -n eksa-system tink-controller-5d876c7b44-z55h8 -o yaml|grep -i priori
  preemptionPolicy: PreemptLowerPriority
  priority: 0
$ kubectl get pods -n eksa-system kube-vip-pm5cx -o yaml|grep -i priori
  preemptionPolicy: PreemptLowerPriority
  priority: 0

IMO, it would be a stability improvement to make sure that all critical EKS-A components are using one of these priority classes.

On top of this, with the current config, any workload on EKS-A could start to use these built-in priority classes which could result into a situation where EKS-A components are not able to run anymore as K8s scheduler would favor normal workloads in case of a resource pressure. Kubernetes provides a way to prevents this through AdmissionConfiguration. It would be advisable to implement this limitation for normal workload for the built-in classes.

Why is this needed:
To improve cluster stability of the cluster in case of resource pressure.