Why you may want to run your Kubernetes clusters on bare metal, plus key considerations and best practices for doing so.
Hrittik Roy Software Engineer
While virtualized cloud infrastructure works great for the majority of workloads out there, bare metal servers are a better fit for a substantial number of use cases. The reasons are many, from the potential for host resource contention among VMs to limited hardware and software configurability offered by traditional virtualized cloud services.
When running on bare metal servers, your application interacts directly with the host, without a hypervisor layer in between. This benefits performance, cost and control and security of your cluster.
In this article, we’ll explain the advantages of running your Kubernetes clusters on bare metal servers, some key decisions you'll need to make if you take this route and best practices for deploying your container orchestrator on bare metal.
More about infrastructure and tooling for Kubernetes:
The decision whether to run Kubernetes on bare metal versus VMs greatly depends on your goals and requirements, as each deployment option has its pros and cons. Virtual machine bootstrapping or managed Kubernetes clustering is great, but bare metal has the upper hand in some situations:
While the performance overhead of a hypervisor is generally minimal, bare metal can make enough of a difference in performance and latency for some specialized teams.
Resource sharing among virtual machines can lead to issues for certain critical tasks that require full infrastructure capacity utilization. When running on dedicated hardware, you don't have to worry about other users' workloads interfering with yours.
Hypervisors can virtualize most—but not all—types of hardware. A server may be too old or too new to be virtualized, for example, or it might have specialized features a hypervisor does not support. Use cases like baseband processing require low-latency kernels, so virtualizing them isn’t efficient or feasible.
With bare metal, you can allow access to different kernels and hardware to increase flexibility. You can fully leverage the unique capabilities of the underlying hardware components (network cards, for example).
Bare metal means one host per system, limiting the attack surface. With single tenancy, you don’t have to worry about hypervisor or virtual machine escape vulnerability attacks.
Another big advantage in this area is the ability to fully secure your workloads by encrypting your nodes and using security hardening frameworks like SELinux and AppArmor.
You also don't have to rely on the cloud provider for kernel updates. You have complete control, so you can ensure that you’re up to date with security patches.
There are fewer abstractions at play with bare metal, including in the networking layer, where virtualization may impact performance. Specs and libraries like the Container Network Interface make setting up networking on bare metal easier.
You can secure the system with high precision by allocating a subnet lease to each host. You can control how data is routed between hosts using tools like flannel and Calico.
Last but not least, the ease of troubleshooting and fault isolation on bare metal simply saves you a lot of time.
Using a bare metal configuration that’s optimal for your application means better resource utilization and more predictability in the cost of scaling infrastructure as your needs change.
A 2020 study by Ericsson estimated that moving away from virtualized systems can result in a total cost of ownership reduction of up to 30 percent.
You also save on licensing fees and support contracts, as you don’t have to pay for managing and maintaining the hypervisor layer. These costs can be significant, especially for organizations that use a lot of servers.
Those are the advantages of running K8s on bare metal. Here are some disadvantages of this approach that can make it difficult for the less experienced teams.
Using managed Kubernetes services is very easy. Click a few buttons or make a few API calls, and you have a ready Kubernetes cluster. Not the case with bare metal!
You have to update and provision a node, then configure different tools and processes and finally join your worker nodes together. Deploying networking, configuring DNS and configuring storage options are all your responsibility.
When you need to scale capacity, there are no buttons you can click to scale the cluster. You have to provision nodes and then join them to your cluster. There are, of course, solutions that abstract away the complexity of these tasks; they’re just not as simple as managed services or VM-based cluster deployments.
Patching and updating your OS is also up to you. You have to stay on top of patches as they come out. The same goes for certificate rotation and backup management, which can all be difficult to keep up with on your own.
The blank slate that bare metal gives you can leave you confused about how exactly to follow best practices and meet IT requirements. You have more flexibility as to what OS, interfaces and hardening solutions you can use, but you have to assess and deploy them so that they’re compatible. You have to plan for version compatibility and account for breaking changes.
Managing, maintaining and configuring bare metal requires dedicated team resources, which not every organization can or wants to have on board.
Once you’ve decided to run your workloads on bare metal, you have several important choices to make in these areas: hardware, software, networking, cluster management and monitoring. Your choices will affect not only your experience standing up and running the infrastructure but also your ability to scale and manage your cluster down the line, as requirements change.
If you’ve gone the bare metal route, chances are you want to get the most out of your hardware. You have to select the right hardware configuration for your workload. Some workloads run best on a few compute-intensive and/or GPU-accelerated nodes, while others are best suited for clusters of many lower-spec machines.
You have to balance performance requirements with your desired hardware utilization rate and, of course, your budget.
The operating system you choose will play a major role in security and software consistency across your bare metal Kubernetes cluster. Choosing an OS that can be used across the whole cluster is helpful, but, depending on your requirements, you might need to run more than one. The key is to use something that’s popular, can be easily secured, has a big and active community and, if required, dedicated support. CentOS, Ubuntu and Debian are some of the popular choices.
Once you’ve selected your OS layer, you’ll need to maintain a fleet of other installations. Runtimes, monitoring tools and cluster management tools must all be kept up to date to maintain a good security posture.
The options for networking solutions are many, including overlay networks (like flannel) and CNI-based solutions like Calico and Cilium, to name a few. These can help you bootstrap your cluster and enhance security. If you need to expose your Kubernetes services outside of the cluster, you can configure a load balancing solution, like MetalLB, to distribute traffic.
With all the networking that you manage, securing the network requires special attention. Misconfigured firewall rules, network policies and access controls can leave your cluster compromised.
You can configure and manage your Kubernetes cluster with tools like kubectl. Another important technology to have in your toolbox is kubeadm, used for setting up a cluster.
Using these management tools to interact with the Kubernetes API, you can add and remove nodes, upgrade Kubernetes versions and manage workloads over the cluster's lifecycle.
Monitoring is a superpower! Monitoring and analytics tools like Prometheus, Grafana and Elasticsearch show you how healthy your cluster is.
Building and operating a bare metal cluster requires you to manage events, metrics and logs in order to optimize your workloads and resolve issues as they surface. In most cases you can automate those tasks.
Once you’ve selected the basic components for your bare metal cluster, follow these best practices for getting the most out of it:
Backups are complex with bare metal. There are no snapshots like the ones that enable VM backups. With one whole bare metal server per node, however, a failure can kill a lot of containers. So, backing up your workloads and having a recovery plan are requirements you cannot afford to skip.
Following principles like GitOps, to keep a single source of truth with all the manifests to restore your cluster, comes in handy here. With additional backup tools, you can make sure your cluster is restored and operational when it has to be.
Planning for the future is essential. If your demand increases, you have to be sure to have enough compute, storage and memory to handle the load, and if it decreases, it’s not great to keep underutilized hardware on your business balance sheet. A bare-metal-as-a-service offering (like Equinix’s) helps in both cases. You can remotely rent dedicated bare metal servers as needed, paying for as much capacity as you use at any given moment, rather than buying, deploying and managing hardware on your own.
The most important consideration when planning to scale your resources is interoperability of your cluster components with new hardware you add in the future. Manually test and configure the hardware to join your cluster as nodes, making sure that nothing breaks operations.
Remember to account for Day 2 operations, where it’s best to keep things as simple and easy to manage as possible.
Establish processes to regularly check for hardware failures and replace any failing hardware as soon as possible. Have a process for updating your operating systems with the latest patches or versions.
Lean on your monitoring solutions to create automated alerts and be prepared to take action when they come in.
From configuring encryption to managing firewalls, making sure your environment is secure is entirely your responsibility when running on bare metal. Bootstrapping tools like kubeadm can help you launch a cluster with a good security posture, but it’s up to you to maintain that posture, test and fix vulnerabilities.
The best practice here is to secure network configurations with proper access controls, like TLS-protected interfaces. Use security audit and review tools to find and fix issues related to your cluster and infrastructure regularly.
Once your team has determined that bare metal is the high-performance Kubernetes infrastructure it needs, there are a few important next steps.
First, deploying your Kubernetes infrastructure requires some careful planning. This includes choosing the right hardware, setting up the network and storage infrastructure and configuring Kubernetes itself. You have to make your security and monitoring choices at this stage as well. Making the right decisions here requires some expertise, which is available from many third-party vendors if not available in house.
Once your Kubernetes infrastructure is up and running, it will be time to start deploying applications. Kubernetes itself has a powerful set of tools for managing and scaling containerized applications, and there are many great tools others have built. It can take some time to learn them, however. Your team will need to have grokked key Kubernetes concepts, such as pods, services and deployments, and get comfortable configuring and managing these resources.
Finally, don’t overlook the importance of constant monitoring of your Kubernetes infrastructure. This is how you continuously optimize your infrastructure, by tracking performance metrics and identifying bottlenecks to resolve, scale it as needed and maintain a good security posture. Here, again, it’s important to have the right level of expertise on hand to ensure your infrastructure runs optimally, so, if you don’t have it in house, seek a partner who can help.
