This document proposes a unified Kubernetes-based strategy for running application stacks across multiple environments: EKS (production and final validation), local development, CI, and on‑prem / isolated (VMware / ESXi) setups.

The core principle is:

All application and Helm-level artifacts remain identical across environments. Differences are isolated strictly into a platform layer (base charts, defaults, and cluster configuration).

This approach minimizes environment drift, enables fast integration testing, supports proper autoscaling validation, and avoids coupling application delivery to specific infrastructure implementations.

We need a solution that:

• Runs the same application stack across:

  • AWS EKS (production)

  • Local developer machines

  • CI pipelines (ephemeral environments)

  • On‑prem / isolated environments (VMware / ESXi, single or few VMs)

• Supports integration and system tests that provision and tear down full stacks quickly

• Allows proper autoscaling validation (HPA and cluster autoscaling)

• Avoids leaking infrastructure or cloud-specific assumptions into application code or Helm charts

• Keeps the number of supported Kubernetes distributions to a minimum

This proposal explicitly does not aim to:

• Perfectly reproduce AWS‑specific behavior outside AWS

• Make local or on‑prem environments identical to EKS at the infrastructure level

• Support unlimited Kubernetes distributions

• Hide all behavioral differences between cloud and non‑cloud environments

Behavioral parity is achieved only on EKS. Other environments provide functional validation.

• Kubernetes APIs (not VMs, not cloud providers) define the system boundary

• Application charts depend only on:

  • Core Kubernetes APIs

  • A small, explicit set of platform capabilities

Application charts must not embed platform or vendor specifics.

To minimize complexity while covering all requirements:

1. Amazon EKS (mandatory)

2. k3s (non‑EKS environments)

• EKS provides authoritative behavior for networking, autoscaling, IAM, and storage

• k3s provides:

  • Upstream‑compatible Kubernetes APIs

  • Multi‑node clusters on VMs

  • Functional autoscaling (HPA + cluster autoscaler)

  • Operation on local machines, CI, and VMware / ESXi

This combination covers all environments with only two Kubernetes substrates.

Interpretation rule:

Each Kubernetes cluster must provide the following capabilities:

• A default StorageClass suitable for standard PVC usage

• A default IngressClass

• A working metrics pipeline (metrics‑server / Prometheus)

• Support for HPA

• Support for cluster autoscaling (where applicable)

• Standard DNS and service discovery

Implementation details may differ per environment but must be invisible to applications.

The following cannot be made equivalent outside AWS:

• AWS VPC CNI semantics (ENI limits, IP exhaustion behavior)

• ALB/NLB controller behavior

• IRSA and IAM failure modes

• EBS/EFS performance and attachment semantics

• Spot interruption and AWS capacity constraints

Tests relying on these behaviors must run on EKS.

LocalStack may be used as an AWS API emulator in local/CI environments:

• Emulates AWS services (S3, SQS, SNS, etc.)

• Runs as a workload inside Kubernetes

• Reduces AWS cost and dependency for non‑prod tests

LocalStack does not replace Kubernetes or EKS.

1. Provision Kubernetes cluster (k3s or EKS)

2. Deploy platform layer

3. Deploy application stack (same charts/values)

4. Run tests

5. Collect results

6. Tear down cluster

Local / CI → k3s multi‑node → EKS

Confidence increases at each stage; artifacts remain identical.

This proposal establishes a clear, minimal, and scalable Kubernetes strategy:

• Kubernetes is the abstraction boundary

• EKS is the reference and final validator

• k3s covers all non‑AWS environments

• Application artifacts remain identical everywhere

• Platform differences are explicit, documented, and controlled

This balances speed, correctness, and operational sanity without over‑engineering or false guarantees.

• Approve substrate selection (EKS + k3s)

• Define platform capability contract as policy

• Document base chart responsibilities

• Classify tests by required environment (k3s vs EKS)!