Consul K8s on OpenShift: Service Mesh SCC Requirements

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

It is common for users running an OpenShift cluster will run into issues with third-party applications within their cluster due to the application containers not accounting for SCC UID namespace restrictions in OpenShift. You can read more regarding this in this Red Hat blog. 

This KB article aims to provide users a temporary workaround to allow the Consul K8s Control Plane service-mesh connect-injected pods and/or containers permission to operate and come online by providing the least amount of permissions required.

• Users should aim at upgrading their OpenShift consul-k8s-control-plane and consul-dataplane images to versions ≥ 1.4.4 or ≥ 1.5.1 as soon as possible to be able to leverage the restricted-v2 SCC with their service-mesh versus the custom SCC example provided here.
• consul-k8s PR #4152 is backported and merged into consul-k8s-control-plane ≥ 1.4.4 and ≥ 1.5.1 and will account for managing the UIDs appropriately on OpenShift with Consul.

Applicability

This article pertains only to Consul on Kubernetes OpenShift platforms. Other platforms such as EKS, AKS, and GKE are not subject to SCC restrictions and Consul K8s should continue to operate as expected.

SCC and Service Mesh

Consul K8s Control Plane defaults to the below Security Contexts upon a connect-inject pod deployment.

# consul-connect-inject-init
       securityContext:
        capabilities:
          add:
            - NET_BIND_SERVICE
          drop:
            - ALL
        privileged: false
        runAsUser: 5996
        runAsGroup: 5996
        runAsNonRoot: true
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false

# consul-dataplane
      securityContext:
        capabilities:
          add:
            - NET_BIND_SERVICE
          drop:
            - ALL
        runAsUser: 5995
        runAsGroup: 5995
        runAsNonRoot: true
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false

OpenShift will not permit these container deployments with hard-coded UIDs and results in failed pod deployments.

• Goal: Allow UIDs 5995 and 5996 for pod injection and admission using the least amount of permissions.
• Blockers: OpenShift defaults to using the restricted-v2 SCC when none are specified which means:
  • Container users must run within the OpenShift project's specific UID range

runAsUser:
 type: MustRunAsRange

NET_BIND_SERVICE capability is permitted, but not enabled by default for each container (this is determined by the application developer)

allowedCapabilities:
- NET_BIND_SERVICE
defaultAddCapabilities: null

Configuring Consul Service Mesh SCC

For the applicable versions of Consul K8s Control Plane that are affected by this particular issue, the following procedure outlines how to template, create, and apply a custom SCC for the OpenShift cluster to be able to run Dataplane sidecars while the cluster operating at or below the versions listed in the above Applicability table. 

Procedure

1. Copy the SCC permissions configured by the built-in nonroot-v2 SCC from your OpenShift cluster to a local yaml file (the filename for this article is consul-service-mesh-v1.yaml).

   $ oc get scc nonroot-v2 -o yaml > consul-service-mesh-v1.yaml 

2. Update consul-service-mesh-v1.yaml to include the following permissions:

   defaultAddCapabilities:
    - NET_BIND_SERVICE
   runAsUser:
    type: MustRunAsNonRoot

3. Update the consul-service-mesh-v1.yaml name and description. 

   metadata:
    name: consul-service-mesh-v1
   annotations:
    kubernetes.io/description: consul-service-mesh-v1 enables traffic-redirection capabilities
   for users 5995 and 5996 of the consul-dataplane and consul-connect-inject-init containers for any
   application sidecar (transparent proxy enablement). The permissions are simply the replication of 
   nonroot-v2 permissions with the addition of defaulting the NET_BIND_SERVICE capability.

4. Run the following against your OpenShift cluster to create the consul-service-mesh-v1 SCC:

   $ oc apply consul-service-mesh-v1.yaml

5. Configure the newly created SCC to bind to the service accounts pertaining to the connect-injected applications.
   • Follow the official OpenShift documentation for Managing security context constraints to determine what method is best for your use-case. 

SCC Usage

ClusterRole for SCC

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: 'system:openshift:scc:consul-service-mesh-v1'
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: 'true'
rules:
  - verbs:
    - use
    apiGroups:
      - security.openshift.io
    resources:
      - securitycontextconstraints
    resourceNames:
      - consul-service-mesh-v1

ClusterRoleBinding for SCC

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul-scc-binding
subjects:
  - kind: ServiceAccount
    name: <your_service_account>
    namespace: <your_namespace>
  - kind: ServiceAccount
    name: <another_service_account>
    namespace: <your_second_namespace>
roleRef:
  kind: ClusterRole
  name: system:openshift:scc:consul-service-mesh-v1
  apiGroup: rbac.authorization.k8s.io

Applying the SCC to all service accounts in a particular namespace

$ oc adm policy add-scc-to-group consul-service-mesh-v1 system:serviceaccounts:<your_namespace>

Appendices

A.1 OpenShift nonroot-v2 and consul-service-mesh-v1 SCCs

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: nonroot-v2
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    kubernetes.io/description: nonroot provides all features of the restricted SCC
      but allows users to run with any non-root UID.  The user must specify the UID
      or it must be specified on the by the manifest of the container runtime. On
      top of the legacy 'nonroot' SCC, it also requires to drop ALL capabilities and
      does not allow privilege escalation binaries. It will also default the seccomp
      profile to runtime/default if unset, otherwise this seccomp profile is required.
priority: null
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities:
  - NET_BIND_SERVICE
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
groups: []
readOnlyRootFilesystem: false
requiredDropCapabilities:
- ALL
runAsUser:
  type: MustRunAsNonRoot
seLinuxContext:
  type: MustRunAs
seccompProfiles:
  - runtime/default
supplementalGroups:
  type: RunAsAny
users: []
volumes:
- configMap
- csi
- downwardAPI
- emptyDir
- ephemeral
- persistentVolumeClaim
- projected
- secret

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: consul-service-mesh-v1
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    kubernetes.io/description: consul-service-mesh-v1 enables traffic-redirection capabilities
      for users 5995 and 5996 of the consul-dataplane and consul-connect-inject-init containers for any
      application sidecar (transparent proxy enablement). The permissions are simply the replication of 
      nonroot-v2 permissions with the addition of defaulting the NET_BIND_SERVICE capability.
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities:
- NET_BIND_SERVICE
defaultAddCapabilities:
- NET_BIND_SERVICE
fsGroup:
  type: MustRunAs
groups: []
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities:
- ALL
runAsUser:
  type: MustRunAsNonRoot
seLinuxContext:
  type: MustRunAs
seccompProfiles:
- runtime/default
supplementalGroups:
  type: RunAsAny
users: []
volumes:
- configMap
- csi
- downwardAPI
- emptyDir
- ephemeral
- persistentVolumeClaim
- projected
- secret