Kubernetes

Palette supports various configuration options for the EKS Kubernetes pack. These are explained in the following sections.

Configure Managed Control Plane Settings

The following sections relate to the customizable options within the managedControlPlane.* section.

Disable IAM OIDC Identity Provider

Use this parameter to disable creation of the Identity and Access Management (IAM) OpenID Connect (OIDC) identity provider.

Parameter	Description	Default
`disableAssociateOIDCProvider`	When set to `true`, this parameter prevents the creation of an IAM OIDC identity provider for the EKS cluster. Once the cluster is created, changing this field has no effect.	`false`

Example

disableAssociateOIDCProvider: true

Configure Logging

Use the following parameters to control the logging for the managed control plane services.

tip

You can use the Control Plane Logging preset to enable or disable all of these parameters.

Parameter	Description	Default
`logging.apiServer`	Toggles logging for the Kubernetes API server.	`false`
`logging.audit`	Toggles Kubernetes API audit logging.	`false`
`logging.authenticator`	Toggles cluster authenticator logging.	`false`
`logging.controllerManager`	Toggles logging for the Kubernetes controller manager.	`false`
`logging.scheduler`	Toggles logging for the Kubernetes scheduler.	`false`

Example

managedControlPlane:
  logging:
    apiServer: true
    audit: true
    authenticator: true
    controllerManager: true
    scheduler: false

Configure IAM Roles for Service Accounts

Use the following parameters to assign IAM roles to Kubernetes service accounts using IAM Roles for Service Accounts (IRSA).

Parameter	Description
`irsaRoles.name`	The name of the IAM role to be created for IRSA. This typically includes the cluster name for uniqueness.
`irsaRoles.policies`	A list of AWS Managed Policy Amazon Resource Names (ARNs) or custom policy ARNs that should be attached to the role.
`irsaRoles.serviceAccount`	Optional object specifying the service account `name` and `namespace` that the role should trust. If omitted, it defaults to the audience `sts.amazonaws.com`.

Example

irsaRoles:
  - name: "{{.spectro.system.cluster.name}}-irsa-cni"
    policies:
      - arn:aws:iam::123456789012:policy/MyCustomCNIForEKS
    serviceAccount:
      name: aws-node
      namespace: kube-system

  - name: "mycluster-irsa-csi"
    policies:
      - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

Configure Security Group Overrides

Use the following parameters to override the default security groups for the Amazon EKS components.

Parameter	Description
`securityGroupOverrides.controlplane`	The security group ID for the EKS control plane instances. If not provided, one is automatically created.
`securityGroupOverrides.bastion`	The security group ID for the bastion host, if configured.
`securityGroupOverrides.node`	The primary security group for worker nodes, either self-managed or managed.
`securityGroupOverrides.node-eks-additional`	Additional security group for worker nodes, if needed.
`securityGroupOverrides.apiserver-lb`	The security group for the API server load balancer, which is the public endpoint to your cluster.
`securityGroupOverrides.lb`	The security group for any other load balancers used by the cluster.

Example

securityGroupOverrides:
  controlplane: "sg-11111111"
  bastion: "sg-22222222"
  node: "sg-33333333"
  node-eks-additional: "sg-44444444"
  apiserver-lb: "sg-55555555"
  lb: "sg-66666666"

Configure OIDC Identity Provider

Use these parameters to configure an external OIDC identity provider for Amazon EKS, such as Okta or Auth0.

Parameter	Description
`oidcIdentityProvider.identityProviderConfigName`	The display name for the OIDC provider config in EKS.
`oidcIdentityProvider.issuerUrl`	The base URL of the external OIDC identity provider.
`oidcIdentityProvider.clientId`	The OIDC client application identifier.
`oidcIdentityProvider.usernameClaim`	Specifies which JWT claim is mapped to the Kubernetes username.
`oidcIdentityProvider.usernamePrefix`	An optional string prepended to the username. This helps avoid collision with existing names.
`oidcIdentityProvider.groupsClaim`	Specifies which JWT claim is mapped to the Kubernetes groups.
`oidcIdentityProvider.groupsPrefix`	An optional string prepended to group names.
`oidcIdentityProvider.requiredClaims`	A map of key-value pairs that must be present in the token. The token is only accepted if all these claim requirements are met.

Example

oidcIdentityProvider:
  identityProviderConfigName: "eks-oidc"
  issuerUrl: "https://tenant.okta.com"
  clientId: "my-oidc-client-id"
  usernameClaim: "email"
  usernamePrefix: "-"
  groupsClaim: "groups"
  groupsPrefix: "-"
  requiredClaims:
    email_verified: "true"

Configure Additional Control Plane Policies

Use this parameter to attach extra policies in ARN format to the Amazon EKS control plane role.

Parameter	Description
`roleAdditionalPolicies`	A list of AWS IAM Policy ARNs to attach to the control plane's IAM role in addition to the default AWS-managed ones.

Example

roleAdditionalPolicies:
  - "arn:aws:iam::123456789012:policy/MyCustomPolicy"
  - "arn:aws:iam::123456789012:policy/AnotherCustomPolicy"

Map IAM Identities to Kubernetes RBAC Groups

Use the following parameters to map IAM users and roles to Kubernetes Role-based Access Control (RBAC) groups, enabling fine-grained access control.

Parameter	Description
`iamAuthenticatorConfig.mapRoles`	A list that maps one or more IAM roles to Kubernetes usernames and groups.
`iamAuthenticatorConfig.mapUsers`	A list that maps one or more IAM users to Kubernetes usernames and groups.

Example

iamAuthenticatorConfig:
  mapRoles:
    - rolearn: arn:aws:iam::000000000000:role/KubernetesNode
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers:
    - userarn: arn:aws:iam::000000000000:user/Alice
      username: alice
      groups:
        - system:masters

Configure Managed Machine Pool Settings

The following sections relate to the customizable options within the managedMachinePool.* section.

Configure Custom IAM Role and Policies for Worker Nodes

Parameter	Description
`roleName`	A custom IAM role name for the worker node group. Must be a self-managed role with EKS worker policies attached. The role name cannot start with `ng-role_` as this is reserved for roles generated by Spectro Cloud.
`roleAdditionalPolicies`	A list of additional policy ARNs to attach to the worker node group's IAM role in addition to the required EKS worker policies.

Example

managedMachinePool:
  roleName: "eks-workers-mycluster"
  roleAdditionalPolicies:
    - "arn:aws:iam::123456789012:policy/MyExtraPolicy"
    - "arn:aws:iam::123456789012:policy/AnotherPolicy"

Configure Client Configuration Settings

The following sections relate to the customizable options within the clientConfig.* section.

Configure OIDC-Based Authentication in Kubeconfig

Use the following parameters to configure OIDC-based authentication when generating a kubeconfig for CLI access and other client tools.

Parameter	Description
`oidc-issuer-url`	The OIDC issuer URL used in the generated kubeconfig.
`oidc-client-id`	The OIDC client ID. This should match the Amazon EKS OIDC identity provider client ID.
`oidc-client-secret`	The OIDC client secret. This is only required if the provider requires `client_secret` authentication.
`oidc-extra-scope`	The additional OAuth scopes to request. These are specific categories of user data, such as email or user profile.

Example

clientConfig:
  oidc-issuer-url: "https://tenant.okta.com"
  oidc-client-id: "my-oidc-client-id"
  oidc-client-secret: "my-oidc-client-secret"
  oidc-extra-scope: "profile,email"

Node Customization

The following sections relate to the customizable options within the nodeCustomization.* section. These options only apply when you have selected a custom Amazon Machine Image (AMI) Type for worker node pools.

You can configure both Amazon Linux 2 (AL2) and Amazon Linux 2023 (AL2023) worker nodes using the nodeCustomization section. The configuration options differ between the two AMI types, as described in the following sections.

info

The nodeCustomization section is not included by default in the Kubernetes pack manifest. You can add it to your manifest in the top-level of the YAML file. Ensure it is not nested under any other section. Examples are provided in the following sections for the Amazon Linux 2 and Amazon Linux 2023 AMI types.

Day-2 Usage Considerations

Any Day-2 additions or changes to the nodeCustomization section will invoke a worker node pool repave.

Configure Amazon Linux 2 AMI Nodes

If you have selected an AL2 AMI for one or more of your worker node groups, you can provide additional configuration and command overrides as described in the following table.

Parameter	Description
`preKubeadmCommands`	A list of shell commands to execute before the `kubeadm` command is executed. These commands are issued on each worker node before they are bootstrapped to the cluster.
`postKubeadmCommands`	A list of shell commands to execute after the `kubeadm` command is executed. These commands are issued on each worker node after they are bootstrapped to the cluster. This can be useful for additional setup or configuration tasks that need to be performed after the node joins the cluster.

Example

## EKS settings
managedControlPlane:
---
managedMachinePool:
---
nodeCustomization:
  al2:
    preKubeadmCommands:
      - |
        echo "[pre-kubeadm] ===== Setting sysctl params =====" | tee --append /var/log/kube_bootstrap.log
        sysctl --write net.bridge.bridge-nf-call-iptables=1
        sysctl --write net.ipv4.ip_forward=1
    postKubeadmCommands:
      - |
        echo "[post-kubeadm] ===== Pulling troubleshooting tools =====" | tee --append /var/log/kube_bootstrap.log
        crictl pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
        echo "[post-kubeadm] Done." | tee --append /var/log/kube_bootstrap.log

Configure Amazon Linux 2023 AMI Nodes

If you have selected an AL2023 AMI for one or more of your worker node groups, you can provide additional configuration and command overrides as described in the following table.

Parameter	Type	Description
`userDataScript`	String (block scalar)	A custom multi-line Bash script that is executed as the root user on every worker node after all `nodeadm` operations are complete. You can use it to execute any extra bootstrap logic, such as install additional packages, pre-pull images, configure HTTP/HTTPS proxies, or inject Certificate Authority (CA) certificates into the system trust store.

Example

## EKS settings
managedControlPlane:
---
managedMachinePool:
---
nodeCustomization:
  al2023:
    userDataScript: |
      #!/bin/bash
      set -euo pipefail

      # Install additional packages
      yum install --assumeyes htop jq iptables-services

      # Pre-cache commonly used container images
      nohup docker pull public.ecr.aws/eks-distro/kubernetes/pause:3.2 &

      # Configure HTTP/HTTPS proxy if needed
      cat > /etc/profile.d/http-proxy.sh << 'EOF'
      export HTTP_PROXY="http://proxy.example.com:3128"
      export HTTPS_PROXY="http://proxy.example.com:3128"
      export NO_PROXY="localhost,127.0.0.1,10.11.12.13,.internal"
      EOF

      # Inject a custom CA certificate
      cat > /etc/pki/ca-trust/source/anchors/myorganization-root-ca.crt <<'EOF_CERT'
      -----BEGIN CERTIFICATE-----
      ...PEM-encoded root/intermediate CA...
      -----END CERTIFICATE-----
      EOF_CERT

      # Make sure file permissions are correct
      chmod 0644 /etc/pki/ca-trust/source/anchors/myorganization-root-ca.crt

      # Rebuild the consolidated CA bundle
      update-ca-trust extract

      echo "[user-data] Custom CA installed and system trust store updated."

Node Customization Layout for AL2 and AL2023 Combined

## EKS settings
managedControlPlane:
...
managedMachinePool:
...
nodeCustomization:
  al2:
    preKubeadmCommands:
      - |
        ...
    postKubeadmCommands:
      - |
        ...
  al2023:
    userDataScript: |
    ...

Required IAM Permissions for Configuration

You must ensure the AWS IAM user or role performing these actions has sufficient privileges. The following table is an overview of the permissions required for each configuration section.

Configuration Section	IAM Permissions Required
Disable IAM OIDC Identity Provider	`iam:CreateOpenIDConnectProvider` `iam:DeleteOpenIDConnectProvider` `iam:GetOpenIDConnectProvider` `iam:TagOpenIDConnectProvider`
Configure Logging	`logs:CreateLogGroup` `logs:CreateLogStream` `logs:PutLogEvents` `eks:UpdateClusterConfig`
Configure IAM Roles for Service Accounts	`iam:CreateRole` `iam:AttachRolePolicy` `iam:PassRole` `iam:GetRole` `iam:ListRoles`
Configure Security Group Overrides	`ec2:DescribeSecurityGroups` `ec2:CreateSecurityGroup` `ec2:AuthorizeSecurityGroupIngress` `ec2:AuthorizeSecurityGroupEgress` `ec2:DeleteSecurityGroup`
Configure OIDC Identity Provider	`eks:AssociateIdentityProviderConfig` `eks:DisassociateIdentityProviderConfig` `eks:DescribeIdentityProviderConfig` `iam:CreateOpenIDConnectProvider` `iam:DeleteOpenIDConnectProvider` `iam:TagOpenIDConnectProvider`
Configure Additional Control Plane Policies	`iam:AttachRolePolicy` `iam:DetachRolePolicy` `iam:PassRole`
Map IAM Identities to Kubernetes RBAC Groups	Depending on the cluster authentication mode, EKS API mode may require `eks:DescribeCluster` and `eks:UpdateClusterConfig`.
Configure Custom IAM Role and Policies for Worker Nodes	`iam:CreateRole` `iam:AttachRolePolicy` `iam:PassRole` `iam:GetRole` `iam:ListRoles`
Configure OIDC-Based Authentication in Kubeconfig	`eks:DescribeCluster`

Core IAM Policies Coverage

If using the Core IAM Policies, including the Controllers EKS Policy, you may need the following additional IAM permissions.

Configuration Section	Missing Permissions
Configure Logging	`logs:CreateLogGroup` `logs:CreateLogStream` `logs:PutLogEvents`
Configure IAM Roles for Service Accounts	`iam:ListRoles`
Configure Security Group Overrides	`ec2:AuthorizeSecurityGroupEgress`
Configure Custom IAM Role and Policies for Worker Nodes	`iam:ListRoles`

Minimum Permissions Coverage

If using Minimum Permissions (dynamic or static), including the Controllers EKS Policy, you may need the following additional IAM permissions.

Configuration Section	Remaining Missing Permissions
Disable IAM OIDC Identity Provider	`iam:GetOpenIDConnectProvider`
Configure Logging	`logs:CreateLogGroup` `logs:CreateLogStream` `logs:PutLogEvents`
Configure IAM Roles for Service Accounts	`iam:ListRoles`
Configure Security Group Overrides	`ec2:AuthorizeSecurityGroupEgress`
Configure Custom IAM Role and Policies for Worker Nodes	`iam:ListRoles`

Terraform

You can retrieve details about the Kubernetes pack for AWS EKS by using the following Terraform code.

data "spectrocloud_registry" "public_registry" {
  name = "Public Repo"
}

data "spectrocloud_pack_simple" "k8s" {
  name    = "kubernetes-eks"
  version = "1.29"
  type = "helm"
  registry_uid = data.spectrocloud_registry.public_registry.id
}

Configure Managed Control Plane Settings​

Disable IAM OIDC Identity Provider​

Configure Logging​

Configure IAM Roles for Service Accounts​

Configure Security Group Overrides​

Configure OIDC Identity Provider​

Configure Additional Control Plane Policies​

Map IAM Identities to Kubernetes RBAC Groups​

Configure Managed Machine Pool Settings​

Configure Custom IAM Role and Policies for Worker Nodes​

Configure Client Configuration Settings​

Configure OIDC-Based Authentication in Kubeconfig​

Node Customization​

Day-2 Usage Considerations​

Configure Amazon Linux 2 AMI Nodes​

Configure Amazon Linux 2023 AMI Nodes​

Node Customization Layout for AL2 and AL2023 Combined​

Required IAM Permissions for Configuration​

Core IAM Policies Coverage​

Minimum Permissions Coverage​

Terraform​

Configure Managed Control Plane Settings

Disable IAM OIDC Identity Provider

Configure Logging

Configure IAM Roles for Service Accounts

Configure Security Group Overrides

Configure OIDC Identity Provider

Configure Additional Control Plane Policies

Map IAM Identities to Kubernetes RBAC Groups

Configure Managed Machine Pool Settings

Configure Custom IAM Role and Policies for Worker Nodes

Configure Client Configuration Settings

Configure OIDC-Based Authentication in Kubeconfig

Node Customization

Day-2 Usage Considerations

Configure Amazon Linux 2 AMI Nodes

Configure Amazon Linux 2023 AMI Nodes

Node Customization Layout for AL2 and AL2023 Combined

Required IAM Permissions for Configuration

Core IAM Policies Coverage

Minimum Permissions Coverage

Terraform