My Profile Photo

Sheogorath's Blog

Kubernetes native delete protection using Validation Admission Policies

on

In Kubernetes 1.30 Validation Admission Policies became stable and generally available. Validation Admission Policies are a lightweight, built-in policy engine for the Kubernetes API server, that allows to use the “Common Expression Language” (CEL) to express policies for single objects and warn, log or deny them.

To play around with this new feature, I decided to implement a simple feature, that would have saved me one a lot of work, when I needed to recover a lot of longhorn volumes after accidentally deleting the longhorn-system namespace on my cluster. But that’s what backups are for, so don’t forget these.

Anyway, creating an active policy is a two step process. First you create the ValidatingAdmissionPolicy, and then a ValidatingAdmissionPolicyBinding. The same way, you might know from Roles and RoleBindings, just with policies.

Creating the Validation Admission Policy

As a first part, there is the general header and definition:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "deletion-protection"
spec:

Now it starts with the matchConstraints, which is a matcher for what objects/API and operations this policy is applied to. Since this Policy is supposed to allow a generic deletion protection, it matches on all objects and the delete operation:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "deletion-protection"
spec:
  matchConstraints:
    resourceRules:
      - apiGroups:   ["*"]
        apiVersions: ["*"]
        operations:  ["DELETE"]
        resources:   ["*"]

Now we can pre-filter the objects, this helps to keep the performance impact minimal. If the all matchCondition are evaluating as true, the API call will be subject to the policy, otherwise it’ll pass right through.

For the deletion-protection policy, we want to check for an annotation with the key forbid-deletion. Only if this key is present, we need to go further with the evaluation:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "deletion-protection"
spec:
  matchConstraints:
    resourceRules:
      - apiGroups:   ["*"]
        apiVersions: ["*"]
        operations:  ["DELETE"]
        resources:   ["*"]
  matchConditions:
    - name: 'protectedFromDeletion'
      expression: "has(oldObject.metadata.annotations) && 'forbid-deletion' in oldObject.metadata.annotations"

To make the decision whether this API call should succeed of fail, one adds validations, here we evaluate, that the content of the forbid-deletion annotation is not true. Since a call has to pass all validations in order to be considered valid, a check that forbid-deletion is true on the object makes the validation fail:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "deletion-protection"
spec:
  matchConstraints:
    resourceRules:
      - apiGroups:   ["*"]
        apiVersions: ["*"]
        operations:  ["DELETE"]
        resources:   ["*"]
  matchConditions:
    - name: 'protectedFromDeletion'
      expression: "has(oldObject.metadata.annotations) && 'forbid-deletion' in oldObject.metadata.annotations"
  validations:
    - expression: "!(oldObject.metadata.annotations['forbid-deletion'] == 'true')"
      messageExpression: "'Object is protected by deletion-protection ValidatingAdmissionPolicy, set annoation forbid-deletion to false'"
      reason: Forbidden

For good measure adding a failurePolicy of Fail prevents requests from accidentally slipping through:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "deletion-protection"
spec:
  matchConstraints:
    resourceRules:
      - apiGroups:   ["*"]
        apiVersions: ["*"]
        operations:  ["DELETE"]
        resources:   ["*"]
  matchConditions:
    - name: 'protectedFromDeletion'
      expression: "has(oldObject.metadata.annotations) && 'forbid-deletion' in oldObject.metadata.annotations"
  validations:
    - expression: "!(oldObject.metadata.annotations['forbid-deletion'] == 'true')"
      messageExpression: "'Object is protected by deletion-protection ValidatingAdmissionPolicy, set annoation forbid-deletion to false'"
      reason: Forbidden
  failurePolicy: Fail

With the ValidationAdmissionPolicy done, the first step is complete. These policies are best developed on the CEL playground, which provides you with the ability to test your policy, as well as checking the evaluation cost, which shows the performance impact from the policy.

Creating the Validation Admission Policy Binding

After writing the policy, it still needs to be bound in order to be actually active. The binding also allows you to define parameter and most importantly, the effect a failed policy evaluation has on the API request.

It also starts with metadata:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "deletion-protection"
spec:

In the spec section defines the policyName with the name from the policy above and the validationActions, which allows defines whether the request should print a warning, be written to the audit log or denied.

Finally there is a matchResources selector, which can help to further narrow down what API requests are affected by the policy evaluation, this can also be used to safe-guard namespaces like kube-system from policies.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "deletion-protection"
spec:
  policyName: "deletion-protection"
  validationActions: [Deny]
  matchResources:
    namespaceSelector: {}

Conclusion

This lightweight, built-in solution for policies, can probably help a lot of simpler policy needs, like the ones I have myself, without introducing a full policy engine like OPA or Kyverno.

The limitation in scope, e.g. the absence of mutation features, as well as the restriction to single objects keep the built-in policies minimalistic and prevent overuse. The use of CEL originates form their introduction as part of CRDs in recent version of Kubernetes, which Validation Admission Policies now make generally available.

Note: This article only scratches the surface of what this ValidationAdmissionPolicies can do, and to fully understand it, you’ll need to look Kubernetes documentation linked below.

References