Automated k3s Cluster Deployment on vSphere with Terraform Cloud, GitLab CI/CD, and HashiCorp Vault

This architecture represents an automated deployment pipeline for provisioning a Kubernetes (k3s) cluster using Terraform Cloud, GitLab CI/CD, Packer, and HashiCorp Vault for secure configuration management. The process spans from code version control to the final deployment of a Kubernetes cluster on a local vSphere environment.

Here’s a breakdown of each step in the architecture:

1. Push Git Tag from Desktop:
   • A developer pushes a git tag from their local environment to the GitLab repository to initiate the deployment pipeline.
2. Version Control System (VCS) Connection:
   • The GitLab repository is connected to Terraform Cloud as a VCS, allowing Terraform Cloud to detect changes in the repository and trigger workflows automatically.
3. GitLab Repo for Terraform k3s Build:
   • The project is organized into multiple repositories. This specific repository contains the Terraform code necessary to configure and deploy the k3s Kubernetes cluster.
4. Trigger CI/CD Pipeline:
   • The push event triggers the GitLab CI/CD pipeline, which automates the entire build and deployment process.
5. Terraform Private Module Registry:
   • Terraform Cloud pulls any necessary Terraform modules from a private module registry. This allows for reusability and consistency across different deployments of k3s VMs.
6. Cloud Agent for Local Provisioning:
   • A Terraform Cloud Agent is used to provision resources locally on vSphere. This agent bridges the gap between Terraform Cloud and on-premises infrastructure, making it possible to provision infrastructure in a local environment.
7. Packer Template for VM Creation:
   • Packer is used to create VM templates, which are stored for use in provisioning. Packer generates a template that defines the configuration of each VM, ensuring a standardized setup for all cluster nodes.
8. Provisioning Master and Worker Nodes:
   • Using the Packer-generated VM template, the Terraform Cloud Agent provisions the k3s cluster, creating both master nodes and worker nodes. These nodes form the foundation of the Kubernetes cluster.
9. Configure k3s Cluster:
   • Once the VMs are provisioned, Terraform configures them to function as a k3s cluster. This involves setting up networking, connecting nodes, and ensuring the Kubernetes control plane is properly established.
10. Securely Store Kubernetes Config with HashiCorp Vault:

• The kubeconfig file, which contains access credentials and configuration details for the k3s cluster, is stored securely in HashiCorp Vault. This ensures that sensitive information is managed securely and can be accessed by authorized personnel when needed.

---

Key Components in the Architecture

• GitLab CI/CD: Manages the automated build and deployment pipeline, ensuring each code change is automatically reflected in the infrastructure.
• Terraform Cloud: Centralizes the Infrastructure as Code (IaC) management and allows version-controlled Terraform configurations.
• Terraform Cloud Agent: Provides connectivity for provisioning infrastructure locally on vSphere, despite using Terraform Cloud.
• Packer: Standardizes VM configurations by creating reusable templates, which are then deployed by Terraform.
• k3s Kubernetes: A lightweight Kubernetes distribution, ideal for smaller clusters or resource-limited environments.
• HashiCorp Vault: Manages and securely stores sensitive configuration data, such as kubeconfig files.

Summary

This architecture offers a fully automated pipeline for deploying a k3s Kubernetes cluster on a local vSphere infrastructure, while leveraging Terraform Cloud for IaC and HashiCorp Vault for secure configuration management. The use of GitLab CI/CD and Packer enhances consistency, repeatability, and security across deployments. This setup is especially useful for on-premises environments needing modern CI/CD and IaC practices for Kubernetes provisioning.