Multi-tenancy the GitOps way

This document will guide you to manage Tenant resources the GitOps way with Flux configured with the multi-tenancy lockdown.

The proposed approach consists on making Flux to reconcile Tenant resources as Tenant Owners, while still providing Namespace as a Service to Tenants.

This means that Tenants can operate and declare multiple Namespaces in their own Git repositories while not escaping the policies enforced by Capsule.

Quickstart

Install

In order to make it work you can install the FluxCD addon via Helm:

helm install -n capsule-system capsule-addon-fluxcd \
  oci://ghcr.io/projectcapsule/charts/capsule-addon-fluxcd

Configure Tenants

The audience for this part is the platform administrator user persona.

In order to make Flux controllers reconcile Tenant resources impersonating a Tenant Owner, a Tenant Owner as Service Account is required.

To be recognized by the addon that will automate the required configurations, the ServiceAccount needs the capsule.addon.fluxcd/enabled=true annotation.

Assuming a configured oil Tenant, the following Tenant Owner ServiceAccount must be declared:

---
apiVersion: v1
kind: Namespace
metadata:
  name: oil-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitops-reconciler
  namespace: oil-system
  annotations:
    capsule.addon.fluxcd/enabled: "true"

set it as a valid oil Tenant owner, and made Capsule recognize its Group:

---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: oil
spec:
  additionalRoleBindings:
  - clusterRoleName: cluster-admin
    subjects:
    - name: gitops-reconciler
      kind: ServiceAccount
      namespace: oil-system
  owners:
  - name: system:serviceaccount:oil-system:gitops-reconciler
    kind: ServiceAccount
---
apiVersion: capsule.clastix.io/v1beta2
kind: CapsuleConfiguration
metadata:
  name: default
spec:
  userGroups:
  - capsule.clastix.io
  - system:serviceaccounts:oil-system

The addon will automate:

RBAC configuration for the Tenant owner ServiceAccount
Tenant owner ServiceAccount token generation
Tenant owner kubeconfig needed to send Flux reconciliation requests through the Capsule proxy
Tenant kubeconfig distribution across all Tenant Namespaces.

The last automation is needed so that the kubeconfig can be set on Kustomizations/HelmReleases across all Tenant's Namespaces.

More details on this are available in the deep-dive section.

How to use

The audience for this part is the platform administrator user persona.

Consider a Tenant named oil that has a dedicated Git repository that contains oil's configurations.

You as a platform administrator want to provide to the oil Tenant a Namespace-as-a-Service with a GitOps experience, allowing the tenant to version the configurations in a Git repository.

You, as Tenant owner, can configure Flux reconciliation resources to be applied as Tenant owner:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: oil-apps
  namespace: oil-system
spec:
  serviceAccountName: gitops-reconciler
  kubeConfig:
    secretRef:
      name: gitops-reconciler-kubeconfig
      key: kubeconfig
  sourceRef:
    kind: GitRepository
    name: oil
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: oil
  namespace: oil-system
spec:
  url: https://github.com/oil/oil-apps

Let's analyze the setup field by field:

the GitRepository and the Kustomization are in a Tenant system Namespace
the Kustomization refers to a ServiceAccount to be impersonated when reconciling the resources the Kustomization refers to: this ServiceAccount is an oil Tenant owner
the Kustomization refers also to a kubeConfig to be used when reconciling the resources the Kustomization refers to: this is needed to make requests through the Capsule proxy in order to operate on cluster-wide resources as a Tenant

The oil tenant can also declare new Namespaces thanks to the segregation provided by Capsule.

Note: it can be avoided to explicitly set the service account name when it's set as default Service Account name at Flux's kustomize-controller level via the default-service-account flag.

More information are available in the addon repository.

Deep dive

Flux and multi-tenancy

Flux v2 released a set of features that further increased security for multi-tenancy scenarios.

These features enable you to:

disable cross-Namespace reference of Source CRs from Reconciliation CRs and Notification CRs. This way, especially for tenants, they can't access resources outside their space. This can be achieved with --no-cross-namespace-refs=true option of kustomize, helm, notification, image-reflector, image-automation controllers.
set a default ServiceAccount impersonation for Reconciliation CRs. This is supposed to be an unprivileged SA that reconciles just the tenant's desired state. This will be enforced when is not otherwise specified explicitly in Reconciliation CR spec. This can be enforced with the --default-service-account=<name> option of helm and kustomize controllers.

For this responsibility we identify a Tenant GitOps Reconciler identity, which is a ServiceAccount and it's also the tenant owner (more on tenants and owners later on, with Capsule).
disallow remote bases for Kustomizations. Actually, this is not strictly required, but it decreases the risk of referencing Kustomizations which aren't part of the controlled GitOps pipelines. In a multi-tenant scenario this is important too. They can be disabled with --no-remote-bases=true option of the kustomize controller.

Where required, to ensure privileged Reconciliation resources have the needed privileges to be reconciled, we can explicitly set a privileged ServiceAccounts.

In any case, is required that the ServiceAccount is in the same Namespace of the Kustomization, so unprivileged spaces should not have privileged ServiceAccounts available.

For example, for the root Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller # It has cluster-admin permissions
  path: ./clusters/staging
  sourceRef:
    kind: GitRepository
    name: flux-system

In example, the cluster admin is supposed to apply this Kustomization, during the cluster bootstrap that i.e. will reconcile also Flux itself. All the remaining Reconciliation resources can be children of this Kustomization.

Namespace-as-a-Service

Tenants could have his own set of Namespaces to operate on but it should be prepared by higher-level roles, like platform admins: the declarations would be part of the platform space. They would be responsible of tenants administration, and each change (e.g. new tenant Namespace) should be a request that would pass through approval.

What if we would like to provide tenants the ability to manage also their own space the GitOps-way? Enter Capsule.

Manual setup

Legenda:

Privileged space: group of Namespaces which are not part of any Tenant.

Privileged identity: identity that won't pass through Capsule tenant access control.

Unprivileged space: group of Namespaces which are part of a Tenant.

Unprivileged identity: identity that would pass through Capsule tenant access control.

Tenant GitOps Reconciler: a machine Tenant Owner expected to reconcile Tenant desired state.

Capsule

Capsule provides a Custom Resource Tenant and ability to set its owners through spec.owners as references to:

User
Group
ServiceAccount

Tenant and Tenant Owner

We would like to let a machine reconcile Tenant's states, we'll need a ServiceAccount as a Tenant Owner:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitops-reconciler
  namespace: my-tenant
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: my-tenant
spec:
  owners:
  - name: system:serviceaccount:my-tenant:gitops-reconciler # the Tenant GitOps Reconciler

From now on, we'll refer to it as the Tenant GitOps Reconciler.

Tenant Groups

We also need to state that Capsule should enforce tenant access control for requests coming from tenants, and we can do that by specifying one of the Groups bound by default by Kubernetes to the Tenant GitOps Reconciler ServiceAccount in the CapsuleConfiguration:

apiVersion: capsule.clastix.io/v1beta2
kind: CapsuleConfiguration
metadata:
  name: default
spec:
  userGroups:
  - system:serviceaccounts:my-tenant

Other privileged requests, e.g. for reconciliation coming from the Flux privileged ServiceAccounts like flux-system/kustomize-controller will bypass Capsule.

Flux

Flux enables to specify with which identity Reconciliation resources are reconciled, through:

ServiceAccount impersonation
kubeconfig

ServiceAccount

As by default Flux reconciles those resources with Flux cluster-admin Service Accounts, we set at controller-level the default ServiceAccount impersonation to the unprivileged Tenant GitOps Reconciler:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-controllers.yaml
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/0
        value: --default-service-account=gitops-reconciler # the Tenant GitOps Reconciler
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"

This way tenants can't make Flux apply their Reconciliation resources with Flux's privileged Service Accounts, by not specifying a spec.ServiceAccountName on them.

At the same time at resource-level in privileged space we still can specify a privileged ServiceAccount, and its reconciliation requests won't pass through Capsule validation:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller
  path: ./clusters/staging
  sourceRef:
    kind: GitRepository
    name: flux-system

Kubeconfig

We also need to specify on Tenant's Reconciliation resources, the Secret with kubeconfig configured to use the Capsule Proxy as the API server in order to provide the Tenant GitOps Reconciler the ability to list cluster-level resources. The kubeconfig would specify also as the token the Tenant GitOps Reconciler SA token,

For example:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: my-app
  namespace: my-tenant
spec:
  kubeConfig:
    secretRef:
      name: gitops-reconciler-kubeconfig
      key: kubeconfig 
  sourceRef:
    kind: GitRepository
    name: my-tenant
  path: ./staging

We'll see how to prepare the related Secret (i.e. gitops-reconciler-kubeconfig) later on.

Each request made with this kubeconfig will be done impersonating the user of the default impersonation SA, that is the same of the token specified in the kubeconfig. To deepen on this please go to #Insights.

The recipe

How to setup Tenants GitOps-ready

Given that Capsule and Capsule Proxy are installed, and Flux v2 configured with multi-tenancy lockdown features, of which the patch below:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-components.yaml
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/0
        value: --no-cross-namespace-refs=true      
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-remote-bases=true      
    target:
      kind: Deployment
      name: "kustomize-controller"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/0
        value: --default-service-account=gitops-reconciler # The Tenant GitOps Reconciler
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"
  - patch: |
      - op: add
        path: /spec/serviceAccountName
        value: kustomize-controller      
    target:
      kind: Kustomization
      name: "flux-system"

this is the required set of resources to setup a Tenant:

Namespace: the Tenant GitOps Reconciler "home". This is not part of the Tenant to avoid a chicken & egg problem:
```
apiVersion: v1
kind: Namespace
metadata:
  name: my-tenant
```

ServiceAccount of the Tenant GitOps Reconciler, in the above Namespace:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitops-reconciler
  namespace: my-tenant

Tenant resource with the above Tenant GitOps Reconciler's SA as Tenant Owner, with:

Additional binding to cluster-admin ClusterRole for the Tenant's Namespaces and Namespace of the Tenant GitOps Reconciler' ServiceAccount. By default Capsule binds only admin ClusterRole, which has no privileges over Custom Resources, but cluster-admin has. This is needed to operate on Flux CRs:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: my-tenant
spec:
  additionalRoleBindings:
  - clusterRoleName: cluster-admin
    subjects:
    - name: gitops-reconciler
      kind: ServiceAccount
      namespace: my-tenant
  owners:
  - name: system:serviceaccount:my-tenant:gitops-reconciler
    kind: ServiceAccount

Additional binding to cluster-admin ClusterRole for home Namespace of the Tenant GitOps Reconciler' ServiceAccount, so that the Tenant GitOps Reconciler can create Flux CRs on the tenant home Namespace and use Reconciliation resource's spec.targetNamespace to place resources to Tenant Namespaces:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitops-reconciler
  namespace: my-tenant
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: gitops-reconciler
  namespace: my-tenant

Additional Group in the CapsuleConfiguration to make Tenant GitOps Reconciler requests pass through Capsule admission (group system:serviceaccount:<tenant-gitops-reconciler-home-namespace>):
```
apiVersion: capsule.clastix.io/v1alpha1
kind: CapsuleConfiguration
metadata:
  name: default
spec:
  userGroups:
  - system:serviceaccounts:my-tenant
```

Additional ClusterRole with related ClusterRoleBinding that allows the Tenant GitOps Reconciler to impersonate his own User (e.g. system:serviceaccount:my-tenant:gitops-reconciler):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-tenant-gitops-reconciler-impersonator
rules:
- apiGroups: [""]
  resources: ["users"]
  verbs: ["impersonate"]
  resourceNames: ["system:serviceaccount:my-tenant:gitops-reconciler"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-tenant-gitops-reconciler-impersonate
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: my-tenant-gitops-reconciler-impersonator
subjects:
- name: gitops-reconciler
  kind: ServiceAccount
  namespace: my-tenant

Secret with kubeconfig for the Tenant GitOps Reconciler with Capsule Proxy as kubeconfig.server and the SA token as kubeconfig.token.

This is supported only with Service Account static tokens.

Flux Source and Reconciliation resources that refer to Tenant desired state. This typically points to a specific path inside a dedicated Git repository, where tenant's root configuration reside:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: my-tenant
  namespace: my-tenant
spec:
  url: https://github.com/my-tenant/all.git # Git repository URL
  ref:
    branch: main # Git reference
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: my-tenant
  namespace: my-tenant
spec:
  kubeConfig:
    secretRef:
      name: gitops-reconciler-kubeconfig
      key: kubeconfig
  sourceRef:
    kind: GitRepository
    name: my-tenant
  path: config # Path to config from GitRepository Source

This Kustomization can in turn refer to further Kustomization resources creating a tenant configuration hierarchy.

Generate the Capsule Proxy kubeconfig Secret

You need to create a Secret in the Tenant GitOps Reconciler home Namespace, containing the kubeconfig that specifies:

server: Capsule Proxy Service URL with related CA certificate for TLS
token: the token of the Tenant GitOps Reconciler

With required privileges over the target Namespace to create Secret, you can generate it with the proxy-kubeconfig-generator utility:

$ go install github.com/maxgio92/proxy-kubeconfig-generator@latest
$ proxy-kubeconfig-generator \
  --kubeconfig-secret-key kubeconfig \
  --namespace my-tenant \
  --server 'https://capsule-proxy.capsule-system.svc:9001' \
  --server-tls-secret-namespace capsule-system \
  --server-tls-secret-name capsule-proxy \
  --serviceaccount gitops-reconciler

How a Tenant can declare his state

Considering the example above, a Tenant my-tenant could place in his own repository (i.e. https://github.com/my-tenant/all), on branch main at path /config further Reconciliation resources, like:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: my-apps
  namespace: my-tenant
spec:
  kubeConfig:
    secretRef:
      name: gitops-reconciler-kubeconfig
      key: kubeconfig 
  sourceRef:
    kind: GitRepository
    name: my-tenant
  path: config/apps

that refer to the same Source but different path (i.e. config/apps) that could contain his applications' manifests.

The same is valid for a HelmReleases, that instead will refer to an HelmRepository Source.

The reconciliation requests will pass through Capsule Proxy as Tenant GitOps Reconciler with impersonation. Then, as the identity group of the requests matches the Capsule groups they will be validated by Capsule, and finally the RBAC will provide boundaries to Tenant GitOps Reconciler privileges.

If the spec.kubeConfig is not specified the Flux privileged ServiceAccount will impersonate the default unprivileged Tenant GitOps Reconciler ServiceAccount as configured with --default-service-account option of kustomize and helm controllers, but it list requests on cluster-level resources like Namespaces will fail.

Full setup

To have a glimpse on a full setup you can follow the flux2-capsule-multi-tenancy repository. For simplicity, the system and tenants declarations are on the same repository but on dedicated git branches.

It's a fork of flux2-multi-tenancy but with the integration we saw with Capsule.

Insights

Why ServiceAccount that impersonates its own User

As stated just above, you'd be wondering why a user would make a request impersonating himself (i.e. the Tenant GitOps Reconciler ServiceAccount User).

This is because we need to make tenant reconciliation requests through Capsule Proxy and we want to protect from risk of privilege escalation done through bypass of impersonation.

Threats

Bypass unprivileged impersonation

The reason why we can't set impersonation to be optional is because, as each tenant is allowed to not specify neither the kubeconfig nor the impersonation SA for the Reconciliation resource, and because in any case that kubeconfig could contain whatever privileged credentials, Flux would otherwise use the privileged ServiceAccount, to reconcile tenant resources.

That way, a tenant would be capable of managing the GitOps way the cluster as he was a cluster admin.

Furthermore, let's see if there are other vulnerabilities we are able to protect from.

Impersonate privileged SA

Then, what if a tenant tries to escalate by using one of the Flux controllers privileged ServiceAccounts?

As spec.ServiceAccountName for Reconciliation resource cannot cross-namespace reference Service Accounts, tenants are able to let Flux apply his own resources only with ServiceAccounts that reside in his own Namespaces. Which is, Namespace of the ServiceAccount and Namespace of the Reconciliation resource must match.

He could neither create the Reconciliation resource where a privileged ServiceAccount is present (like flux-system), as the Namespace has to be owned by the Tenant. Capsule would block those Reconciliation resource creation requests.

Create and impersonate privileged SA

Then, what if a tenant tries to escalate by creating a privileged ServiceAccount inside on of his own Namespaces?

A tenant could create a ServiceAccount in an owned Namespace, but he can't neither bind at cluster-level nor at a non-owned Namespace-level a ClusterRole, as that wouldn't be permitted by Capsule admission controllers.

Now let's go on with the practical part.

Change ownership of privileged Namespaces (e.g. flux-system)

He could try to use privileged ServiceAccount by changing ownership of a privileged Namespace so that he could create Reconciliation resource there and using the privileged SA. This is not permitted as he can't patch Namespaces which have not been created by him. Capsule request validation would not pass.

For other protections against threats in this multi-tenancy scenario please see the Capsule Multi-Tenancy Benchmark.

Capsule

Tenants and Rancher Projects

Managed Kubernetes

Multi-tenancy the GitOps way

Quickstart

Install

Configure Tenants

How to use

Deep dive

Flux and multi-tenancy

Namespace-as-a-Service

Manual setup

Capsule

Tenant and Tenant Owner

Tenant Groups

Flux

ServiceAccount

Kubeconfig

The recipe

How to setup Tenants GitOps-ready

Generate the Capsule Proxy kubeconfig Secret

How a Tenant can declare his state

Full setup

Insights

Why ServiceAccount that impersonates its own User

Threats

Bypass unprivileged impersonation

Impersonate privileged SA

Create and impersonate privileged SA

Change ownership of privileged Namespaces (e.g. flux-system)

References