This is the multi-page printable view of this section. Click here to print.
Welcome to the Sidero documentation.
If you’re interested in this project and would like to help in engineering efforts, or have general usage questions, we are happy to have you! We hold a weekly meeting that all audiences are welcome to attend.
You can subscribe to this meeting by joining the community forum above.
Sidero (“Iron” in Greek) is a project created by the Sidero Labs team. Sidero Metal provides lightweight, composable tools that can be used to create bare-metal Talos Linux + Kubernetes clusters. These tools are built around the Cluster API project.
Because of the design of Cluster API, there is inherently a “chicken and egg” problem: you need an existing Kubernetes cluster in order to provision the management plane, that can then provision more clusters. The initial management plane cluster that runs the Sidero Metal provider does not need to be based on Talos Linux - although it is recommended for security and stability reasons. The Getting Started guide will walk you through installing Sidero Metal either on an existing cluster, or by quickly creating a docker based cluster used to bootstrap the process.
Sidero Metal is currently made up of two components:
Sidero Metal also needs these co-requisites in order to be useful:
All components mentioned above can be installed using Cluster API’s clusterctl tool.
See the Getting Started for more details.
As of Cluster API version 0.3.9, Sidero is included as a default infrastructure provider in clusterctl.
To install Sidero and the other Talos providers, simply issue:
clusterctl init -b talos -c talos -i sidero
Sidero supports several variables to configure the installation, these variables can be set either as environment
variables or as variables in the clusterctl configuration:
SIDERO_CONTROLLER_MANAGER_HOST_NETWORK (false): run sidero-controller-manager on host networkSIDERO_CONTROLLER_MANAGER_API_ENDPOINT (empty): specifies the IP address controller manager can be reached on, defaults to the node IPSIDERO_CONTROLLER_MANAGER_API_PORT (8081): specifies the port controller manager can be reached onSIDERO_CONTROLLER_MANAGER_CONTAINER_API_PORT (8081): specifies the controller manager internal container portSIDERO_CONTROLLER_MANAGER_EXTRA_AGENT_KERNEL_ARGS (empty): specifies additional Linux kernel arguments for the Sidero agent (for example, different console settings)SIDERO_CONTROLLER_MANAGER_AUTO_ACCEPT_SERVERS (false): automatically accept discovered servers, by default .spec.accepted should be changed to true to accept the serverSIDERO_CONTROLLER_MANAGER_AUTO_BMC_SETUP (true): automatically attempt to configure the BMC with a sidero user that will be used for all IPMI tasks.SIDERO_CONTROLLER_MANAGER_INSECURE_WIPE (true): wipe only the first megabyte of each disk on the server, otherwise wipe the full diskSIDERO_CONTROLLER_MANAGER_SERVER_REBOOT_TIMEOUT (20m): timeout for the server reboot (how long it might take for the server to be rebooted before Sidero retries an IPMI reboot operation)SIDERO_CONTROLLER_MANAGER_IPMI_PXE_METHOD (uefi): IPMI boot from PXE method: uefi for UEFI boot or bios for BIOS bootSIDERO_CONTROLLER_MANAGER_BOOT_FROM_DISK_METHOD (ipxe-exit): configures the way Sidero forces server to boot from disk when server hits iPXE server after initial install: ipxe-exit returns iPXE script with exit command, http-404 returns HTTP 404 Not Found error, ipxe-sanboot uses iPXE sanboot command to boot from the first hard diskSidero provides two endpoints which should be made available to the infrastructure:
SIDERO_CONTROLLER_MANAGER_API_ENDPOINT and SIDERO_CONTROLLER_MANAGER_API_PORT)These endpoints could be exposed to the infrastructure using different strategies:
sidero-controller-manager on the host network.Note: If you want to run
sidero-controller-manageron the host network using port different from8081you should set bothSIDERO_CONTROLLER_MANAGER_API_PORTandSIDERO_CONTROLLER_MANAGER_CONTAINER_API_PORTto the same value.
The overarching architecture of Sidero centers around a “management plane”. This plane is expected to serve as a single interface upon which administrators can create, scale, upgrade, and delete Kubernetes clusters. At a high level view, the management plane + created clusters should look something like:
Sidero, the Talos bootstrap/controlplane providers, and Cluster API each provide several custom resources (CRDs) to Kubernetes. These CRDs are crucial to understanding the connections between each provider and in troubleshooting problems. It may also help to look at the cluster template to get an idea of the relationships between these.
It’s worth defining the most basic resources that CAPI provides first, as they are related to several subsequent resources below.
ClusterCluster is the highest level CAPI resource.
It allows users to specify things like network layout of the cluster, as well as contains references to the infrastructure and control plane resources that will be used to create the cluster.
MachinesMachine represents an infrastructure component hosting a Kubernetes node.
Allows for specification of things like Kubernetes version, as well as contains reference to the infrastructure resource that relates to this machine.
MachineDeploymentsMachineDeployments are similar to a Deployment and their relationship to Pods in Kubernetes primitives.
A MachineDeployment allows for specification of a number of Machine replicas with a given specification.
TalosConfigsThe TalosConfig resource allows a user to specify the type (init, controlplane, join) for a given machine.
The bootstrap provider will then generate a Talos machine configuration for that machine.
This resource also provides the ability to pass a full, pre-generated machine configuration.
Finally, users have the ability to pass configPatches, which are applied to edit a generate machine configuration with user-defined settings.
The TalosConfig corresponds to the bootstrap sections of Machines, MachineDeployments, and the controlPlaneConfig section of TalosControlPlanes.
TalosConfigTemplatesTalosConfigTemplates are similar to the TalosConfig above, but used when specifying a bootstrap reference in a MachineDeployment.
TalosControlPlanesThe control plane provider presents a single CRD, the TalosControlPlane.
This resource is similar to MachineDeployments, but is targeted exclusively for the Kubernetes control plane nodes.
The TalosControlPlane allows for specification of the number of replicas, version of Kubernetes for the control plane nodes, references to the infrastructure resource to use (infrastructureTemplate section), as well as the configuration of the bootstrap data via the controlPlaneConfig section.
This resource is referred to by the CAPI Cluster resource via the controlPlaneRef section.
MetalClustersA MetalCluster is Sidero’s view of the cluster resource.
This resource allows users to define the control plane endpoint that corresponds to the Kubernetes API server.
This resource corresponds to the infrastructureRef section of Cluster API’s Cluster resource.
MetalMachinesA MetalMachine is Sidero’s view of a machine.
Allows for reference of a single server or a server class from which a physical server will be picked to bootstrap.
MetalMachineTemplatesA MetalMachineTemplate is similar to a MetalMachine above, but serves as a template that is reused for resources like MachineDeployments or TalosControlPlanes that allocate multiple Machines at once.
ServerBindingsServerBindings represent a one-to-one mapping between a Server resource and a MetalMachine resource.
A ServerBinding is used internally to keep track of servers that are allocated to a Kubernetes cluster and used to make decisions on cleaning and returning servers to a ServerClass upon deallocation.
EnvironmentsThese define a desired deployment environment for Talos, including things like which kernel to use, kernel args to pass, and the initrd to use.
Sidero allows you to define a default environment, as well as other environments that may be specific to a subset of nodes.
Users can override the environment at the ServerClass or Server level, if you have requirements for different kernels or kernel parameters.
See the Environments section of our Configuration docs for examples and more detail.
ServersThese represent physical machines as resources in the management plane.
These Servers are created when the physical machine PXE boots and completes a “discovery” process in which it registers with the management plane and provides SMBIOS information such as the CPU manufacturer and version, and memory information.
See the Servers section of our Configuration docs for examples and more detail.
ServerClassesServerClasses are a grouping of the Servers mentioned above, grouped to create classes of servers based on Memory, CPU or other attributes.
These can be used to compose a bank of Servers that are eligible for provisioning.
See the ServerClasses section of our Configuration docs for examples and more detail.
While the controller does not present unique CRDs within Kubernetes, it’s important to understand the metadata resources that are returned to physical servers during the boot process.
The Sidero controller manager server may be familiar to you if you have used cloud environments previously.
Using Talos machine configurations created by the Talos Cluster API bootstrap provider, along with patches specified by editing Server/ServerClass resources or TalosConfig/TalosControlPlane resources, metadata is returned to servers who query the controller manager at boot time.
See the Metadata section of our Configuration docs for examples and more detail.
Most of the time, Sidero does very little, so it needs very few resources. However, since it is in charge of any number of workload clusters, it should be built with redundancy. It is also common, if the cluster is single-purpose, to combine the controlplane and worker node roles. Virtual machines are also perfectly well-suited for this role.
Minimum suggested dimensions:
This tutorial will walk you through a complete Sidero setup and the formation, scaling, and destruction of a workload cluster.
To complete this tutorial, you will need a few things:
talosctl, kubectl, and clusterctl can be run.ClusterAPI or CAPI is the common system for managing Kubernetes clusters in a declarative fashion.
Management Cluster is the cluster on which Sidero itself runs. It is generally a special-purpose Kubernetes cluster whose sole responsibility is maintaining the CRD database of Sidero and providing the services necessary to manage your workload Kubernetes clusters.
Sidero is the ClusterAPI-powered system which manages baremetal infrastructure for Kubernetes.
Talos is the Kubernetes-focused Linux operating system built by the same people who bring to you Sidero. It is a very small, entirely API-driven OS which is meant to provide a reliable and self-maintaining base on which Kubernetes clusters may run. More information about Talos can be found at https://talos.dev.
Workload Cluster is a cluster, managed by Sidero, on which your Kubernetes workloads may be run. The workload clusters are where you run your own applications and infrastructure. Sidero creates them from your available resources, maintains them over time as your needs and resources change, and removes them whenever it is told to do so.
You will need three CLI tools installed on your workstation in order to interact with Sidero:
kubectlclusterctltalosctlkubectlSince kubectl is the standard Kubernetes control tool, many distributions
already exist for it.
Feel free to check your own package manager to see if it is available natively.
Otherwise, you may install it directly from the main distribution point. The main article for this can be found here.
sudo curl -Lo /usr/local/bin/kubectl \
"https://dl.k8s.io/release/$(\
curl -L -s https://dl.k8s.io/release/stable.txt\
)/bin/linux/amd64/kubectl"
sudo chmod +x /usr/local/bin/kubectl
clusterctlThe clusterctl tool is the standard control tool for ClusterAPI (CAPI).
It is less common, so it is also less likely to be in package managers.
The main article for installing clusterctl can be found
here.
sudo curl -Lo /usr/local/bin/clusterctl \
"https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.4.7/clusterctl-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64"
sudo chmod +x /usr/local/bin/clusterctl
Note: This version of Sidero is only compatible with CAPI v1alpha4, so versions of
clusterctlabove v0.4.x will not work. Please use the latest v0.4.x version ofclusterctlfrom the release page.
talosctlThe talosctl tool is used to interact with the Talos (our Kubernetes-focused
operating system) API.
The latest version can be found on our
Releases page.
sudo curl -Lo /usr/local/bin/talosctl \
"https://github.com/talos-systems/talos/releases/latest/download/talosctl-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64"
chmod +x /usr/local/bin/talosctl
In order to run Sidero, you first need a Kubernetes “cluster”. There is nothing special about this cluster. It can be, for example:
Two important things are needed in this cluster:
v1.18 or laterFor the purposes of this tutorial, we will create this cluster in Docker on a workstation, perhaps a laptop.
If you already have a suitable Kubernetes cluster, feel free to skip this step.
The talosctl CLI tool has built-in support for spinning up Talos in docker containers.
Let’s use this to our advantage as an easy Kubernetes cluster to start from.
Issue the following to create a single-node Docker-based Kubernetes cluster:
export HOST_IP="192.168.1.150"
talosctl cluster create \
--name sidero-demo \
-p 69:69/udp,8081:8081/tcp \
--workers 0 \
--config-patch '[{"op": "add", "path": "/cluster/allowSchedulingOnMasters", "value": true}]' \
--endpoint $HOST_IP
The 192.168.1.150 IP address should be changed to the IP address of your Docker
host.
This is not the Docker bridge IP but the standard IP address of the
workstation.
Note that there are two ports mentioned in the command above. The first (69) is for TFTP. The second (8081) is for the web server (which serves netboot artifacts and configuration).
Exposing them here allows us to access the services that will get deployed on this node.
In turn, we will be running our Sidero services with hostNetwork: true,
so the Docker host will forward these to the Docker container,
which will in turn be running in the same namespace as the Sidero Kubernetes components.
A full separate management cluster will likely approach this differently,
with a load balancer or a means of sharing an IP address across multiple nodes (such as with MetalLB).
Finally, the --config-patch is optional,
but since we are running a single-node cluster in this Tutorial,
adding this will allow Sidero to run on the controlplane.
Otherwise, you would need to add worker nodes to this management plane cluster to be
able to run the Sidero components on it.
Once the cluster create command is complete, you can retrieve the kubeconfig for it using the Talos API:
talosctl kubeconfig
Note: by default, Talos will merge the kubeconfig for this cluster into your standard kubeconfig under the context name matching the cluster name your created above. If this name conflicts, it will be given a
-1, a-2or so on, so it is generally safe to run. However, if you would prefer to not modify your standard kubeconfig, you can supply a directory name as the third parameter, which will cause a new kubeconfig to be created there instead. Remember that if you choose to not use the standard location, your should set yourKUBECONFIGenvironment variable or pass the--kubeconfigoption to tell thekubectlclient the name of thekubeconfigfile.
In order to network boot Talos, we need to set up our DHCP server to supply the network boot parameters to our servers. For maximum flexibility, Sidero makes use of iPXE to be able to reference artifacts via HTTP. Some modern servers support direct UEFI HTTP boot, but most existing servers still rely on the old, slow TFTP-based PXE boot first. Therefore, we need to tell our DHCP server to find the iPXE binary on a TFTP server.
Conveniently, Sidero comes with a TFTP server which will serve the appropriate files. We need only set up our DHCP server to point to it.
The tricky bit is that at different phases, we need to serve different assets, but they all use the same DHCP metadata key.
In fact, for each architecture, we have as many as four different client types:
If you are lucky and all of the machines in a given DHCP zone can use the same network boot client mechanism, your DHCP server only needs to provide two options:
Server-Name (option 66) with the IP of the Sidero TFTP serviceBootfile-Name (option 67) with the appropriate value for the boot client type:
undionly.kpxeipxe.efihttp://sidero-server-url/tftp/ipxe.efihttp://sidero-server-url/boot.ipxeIn the ISC DHCP server, these options look like:
next-server 172.16.199.50;
filename "ipxe.efi";
Any given server will usually use only one of those, but if you have a mix of machines, you may need a combination of them. In this case, you would need a way to provide different images for different client or machine types.
Both ISC DHCP server and dnsmasq provide ways to supply such conditional responses. In this tutorial, we are working with ISC DHCP.
For modularity, we are breaking the conditional statements into a separate file
and using the include statement to load them into the main dhcpd.conf file.
In our example below, 172.16.199.50 is the IP address of our Sidero service.
ipxe-metal.conf:
allow bootp;
allow booting;
# IP address for PXE-based TFTP methods
next-server 172.16.199.50;
# Configuration for iPXE clients
class "ipxeclient" {
match if exists user-class and (option user-class = "iPXE");
filename "http://172.16.199.50/boot.ipxe";
}
# Configuration for legacy BIOS-based PXE boot
class "biosclients" {
match if not exists user-class and substring (option vendor-class-identifier, 15, 5) = "00000";
filename "undionly.kpxe";
}
# Configuration for UEFI-based PXE boot
class "pxeclients" {
match if not exists user-class and substring (option vendor-class-identifier, 0, 9) = "PXEClient";
filename "ipxe.efi";
}
# Configuration for UEFI-based HTTP boot
class "httpclients" {
match if not exists user-class and substring (option vendor-class-identifier, 0, 10) = "HTTPClient";
option vendor-class-identifier "HTTPClient";
filename "http://172.16.199.50/tftp/ipxe.efi";
}
Once this file is created, we can include it from our main dhcpd.conf inside a
subnet section.
shared-network sidero {
subnet 172.16.199.0 netmask 255.255.255.0 {
option domain-name-servers 8.8.8.8, 1.1.1.1;
option routers 172.16.199.1;
include "/config/ipxe-metal.conf";
}
}
Since we use a number of Ubiquiti EdgeRouter devices especially in our home test
networks, it is worth mentioning the curious syntax gymnastics we must go
through there.
Essentially, the quotes around the path need to be entered as HTML entities:
".
Ubiquiti EdgeRouter configuration statement:
set service dhcp-server shared-network-name sidero \
subnet 172.16.199.1 \
subnet-parameters "include "/config/ipxe-metal.conf";"
Also note the fact that there are two semicolons at the end of the line.
The first is part of the HTML-encoded " (") and the second is the actual terminating semicolon.
Getting the netboot environment is tricky and debugging it is difficult. Once running, it will generally stay running; the problem is nearly always one of a missing or incorrect configuration, since the process involves several different components.
We are working toward integrating as much as possible into Sidero, to provide as much intelligence and automation as can be had, but until then, you will likely need to figure out how to begin hunting down problems.
See the Sidero Troubleshooting guide for more assistance.
Sidero is included as a default infrastructure provider in clusterctl, so the
installation of both Sidero and the Cluster API (CAPI) components is as simple
as using the clusterctl tool.
Note: Because Cluster API upgrades are stateless, it is important to keep all Sidero configuration for reuse during upgrades.
Sidero has a number of configuration options which should be supplied at install
time, kept, and reused for upgrades.
These can also be specified in the clusterctl configuration file
($HOME/.cluster-api/clusterctl.yaml).
You can reference the clusterctl
docs
for more information on this.
For our purposes, we will use environment variables for our configuration options.
export SIDERO_CONTROLLER_MANAGER_HOST_NETWORK=true
export SIDERO_CONTROLLER_MANAGER_API_ENDPOINT=192.168.1.150
clusterctl init -b talos -c talos -i sidero
First, we are telling Sidero to use hostNetwork: true so that it binds its
ports directly to the host, rather than being available only from inside the
cluster.
There are many ways of exposing the services, but this is the simplest
path for the single-node management cluster.
When you scale the management cluster, you will need to use an alternative
method, such as an external load balancer or something like
MetalLB.
The 192.168.1.150 IP address is the IP address or DNS hostname as seen from the workload
clusters.
In our case, this should be the main IP address of your Docker
workstation.
If you built your cluster as specified in the [Prerequisite: Kubernetes] section in this tutorial, your services are already exposed and you can skip this section.
There are two external Services which Sidero serves and which must be made reachable by the servers which it will be driving.
For most servers, TFTP (port 69/udp) will be needed. This is used for PXE booting, both BIOS and UEFI. Being a primitive UDP protocol, many load balancers do not support TFTP. Instead, solutions such as MetalLB may be used to expose TFTP over a known IP address. For servers which support UEFI HTTP Network Boot, TFTP need not be used.
The kernel, initrd, and all configuration assets are served from the HTTP service (port 8081/tcp). It is needed for all servers, but since it is HTTP-based, it can be easily proxied, load balanced, or run through an ingress controller.
The main thing to keep in mind is that the services MUST match the IP or
hostname specified by the SIDERO_CONTROLLER_MANAGER_API_ENDPOINT environment
variable (or configuration parameter) when you installed Sidero.
It is a good idea to verify that the services are exposed as you think they should be.
$ curl -I http://192.168.1.150:8081/tftp/ipxe.efi
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 1020416
Content-Type: application/octet-stream
At this point, any servers on the same network as Sidero should network boot from Sidero.
To register a server with Sidero, simply turn it on and Sidero will do the rest.
Once the registration is complete, you should see the servers registered with kubectl get servers:
$ kubectl get servers -o wide
NAME HOSTNAME ACCEPTED ALLOCATED CLEAN
00000000-0000-0000-0000-d05099d33360 192.168.1.201 false false false
Note in the output above that the newly registered servers are not accepted.
In order for a server to be eligible for consideration, it must be marked as accepted.
Before a Server is accepted, no write action will be performed against it.
This default is for safety (don’t accidentally delete something just because it
was plugged in) and security (make sure you know the machine before it is given
credentials to communicate).
Note: if you are running in a safe environment, you can configure Sidero to automatically accept new machines.
For more information on server acceptance, see the server docs.
By default, Sidero comes with a single ServerClass any which matches any
(accepted) server.
This is sufficient for this demo, but you may wish to have
more flexibility by defining your own ServerClasses.
ServerClasses allow you to group machines which are sufficiently similar to
allow for unnamed allocation.
This is analogous to cloud providers using such classes as m3.large or
c2.small, but the names are free-form and only need to make sense to you.
For more information on ServerClasses, see the ServerClass docs.
In baremetal systems, there are commonly certain small features and configurations which are unique to the hardware. In many cases, such small variations may not require special configurations, but others do.
If hardware-specific differences do mandate configuration changes, we need a way to keep those changes local to the hardware specification so that at the higher level, a Server is just a Server (or a server in a ServerClass is just a Server like all the others in that Class).
The most common variations seem to be the installation disk and the console serial port.
Some machines have NVMe drives, which show up as something like /dev/nvme0n1.
Others may be SATA or SCSI, which show up as something like /dev/sda.
Some machines use /dev/ttyS0 for the serial console; others /dev/ttyS1.
Configuration patches can be applied to either Servers or ServerClasses, and those patches will be applied to the final machine configuration for those nodes without having to know anything about those nodes at the allocation level.
For examples of install disk patching, see the Installation Disk doc.
For more information about patching in general, see the Patching Guide.
Once created and accepted, you should see the servers that make up your ServerClasses appear as “available”:
$ kubectl get serverclass
NAME AVAILABLE IN USE
any ["00000000-0000-0000-0000-d05099d33360"] []
We are now ready to generate the configuration manifest templates for our first workload cluster.
There are several configuration parameters that should be set in order for the templating to work properly:
CONTROL_PLANE_ENDPOINT: The endpoint used for the Kubernetes API server (e.g. https://1.2.3.4:6443).
This is the equivalent of the endpoint you would specify in talosctl gen config.
There are a variety of ways to configure a control plane endpoint.
Some common ways for an HA setup are to use DNS, a load balancer, or BGP.
A simpler method is to use the IP of a single node.
This has the disadvantage of being a single point of failure, but it can be a simple way to get running.CONTROL_PLANE_SERVERCLASS: The server class to use for control plane nodes.WORKER_SERVERCLASS: The server class to use for worker nodes.KUBERNETES_VERSION: The version of Kubernetes to deploy (e.g. v1.21.1).CONTROL_PLANE_PORT: The port used for the Kubernetes API server (port 6443)For instance:
export CONTROL_PLANE_SERVERCLASS=any
export WORKER_SERVERCLASS=any
export TALOS_VERSION=v0.13.3
export KUBERNETES_VERSION=v1.22.2
export CONTROL_PLANE_PORT=6443
export CONTROL_PLANE_ENDPOINT=1.2.3.4
clusterctl generate cluster cluster-0 -i sidero > cluster-0.yaml
Take a look at this new cluster-0.yaml manifest and make any changes as you
see fit.
Feel free to adjust the replicas field of the TalosControlPlane and MachineDeployment objects to match the number of machines you want in your controlplane and worker sets, respecively.
MachineDeployment (worker) count is allowed to be 0.
Of course, these may also be scaled up or down after they have been created, as well.
When you are satisfied with your configuration, go ahead and apply it to Sidero:
kubectl apply -f cluster-0.yaml
At this point, Sidero will allocate Servers according to the requests in the cluster manifest. Once allocated, each of those machines will be installed with Talos, given their configuration, and form a cluster.
You can watch the progress of the Servers being selected:
watch kubectl --context=sidero-demo \
get servers,machines,clusters
First, you should see the Cluster created in the Provisioning phase.
Once the Cluster is Provisioned, a Machine will be created in the
Provisioning phase.
During the Provisioning phase, a Server will become allocated, the hardware
will be powered up, Talos will be installed onto it, and it will be rebooted
into Talos.
Depending on the hardware involved, this may take several minutes.
Eventually, the Machine should reach the Running phase.
The initial controlplane Machine will always be started first. Any additional nodes will be started after that and will join the cluster when they are ready.
In order to interact with the new machines (outside of Kubernetes), you will
need to obtain the talosctl client configuration, or talosconfig.
You can do this by retrieving the resource of the same type from the Sidero
management cluster:
kubectl --context=sidero-demo \
get talosconfig \
-l cluster.x-k8s.io/cluster-name=cluster-0 \
-o jsonpath='{.items[0].status.talosConfig}' \
> cluster-0-talosconfig.yaml
With the talosconfig obtained, the workload cluster’s kubeconfig can be retrieved in the normal Talos way:
talosctl --talosconfig cluster-0.yaml kubeconfig
Now, you should have two cluster available: you management cluster
(sidero-demo) and your workload cluster (cluster-0).
kubectl --context=sidero-demo get nodes
kubectl --context=cluster-0 get nodes
If you have more machines available, you can scale both the controlplane
(TalosControlPlane) and the workers (MachineDeployment) for any cluster
after it has been deployed.
This is done just like normal Kubernetes Deployments.
kubectl scale taloscontrolplane cluster-0-cp --replicas=3
Having the Sidero cluster running inside a Docker container is not the most robust place for it, but it did make for an expedient start.
Conveniently, you can create a Kubernetes cluster in Sidero and then pivot the management plane over to it.
Start by creating a workload cluster as you have already done.
In this example, this new cluster is called management.
After the new cluster is available, install Sidero onto it as we did before, making sure to set all the environment variables or configuration parameters for the new management cluster first.
export SIDERO_CONTROLLER_MANAGER_API_ENDPOINT=sidero.mydomain.com
clusterctl init \
--kubeconfig-context=management
-i sidero -b talos -c talos
Now, you can move the database from sidero-demo to management:
clusterctl move \
--kubeconfig-context=sidero-demo \
--to-kubeconfig-context=management
If you created your sidero-demo cluster using Docker as described in this
tutorial, you can now remove it:
talosctl cluster destroy --name sidero-demo
The first thing to do in troubleshooting problems with the Sidero installation and operation is to figure out where in the process that failure is occurring.
Keep in mind the general flow of the pieces. For instance:
The worst place to fail is also, unfortunately, the most common. This is the firmware phase, where the network card’s built-in firmware attempts to initiate the PXE boot process. This is the worst place because the firmware is completely opaque, with very little logging, and what logging does appear frequently is wiped from the console faster than you can read it.
If you fail here, the problem will most likely be with your DHCP configuration, though it could also be in the Sidero TFTP service configuration.
The easiest to validate is to use a tftp client to validate that the Sidero
TFTP service is available at the IP you are advertising via DHCP.
$ atftp 172.16.199.50
tftp> get ipxe.efi
TFTP is an old, slow protocol with very little feedback or checking.
Your only real way of telling if this fails is by timeout.
Over a local network, this get command should take a few seconds.
If it takes longer than 30 seconds, it is probably not working.
Success is also not usually indicated: you just get a prompt returned, and the file should show up in your current directory.
If you are failing to connect to TFTP, the problem is most likely with your Sidero Service exposure: how are you exposing the TFTP service in your management cluster to the outside world? This normally involves either setting host networking on the Deployment or installing and using something like MetalLB.
Environments are a custom resource provided by the Metal Controller Manager. An environment is a codified description of what should be returned by the PXE server when a physical server attempts to PXE boot.
Especially important in the environment types are the kernel args. From here, one can tweak the IP to the metadata server as well as various other kernel options that Talos and/or the Linux kernel supports.
Environments can be supplied to a given server either at the Server or the ServerClass level. The hierarchy from most to least respected is:
.spec.environmentRef provided at Server level.spec.environmentRef provided at ServerClass level"default" Environment created automatically and modified by an administratorA sample environment definition looks like this:
apiVersion: metal.sidero.dev/v1alpha1
kind: Environment
metadata:
name: default
spec:
kernel:
url: "https://github.com/talos-systems/talos/releases/download/v0.13.3/vmlinuz-amd64"
sha512: ""
args:
- console=tty0
- console=ttyS1,115200n8
- consoleblank=0
- earlyprintk=ttyS1,115200n8
- ima_appraise=fix
- ima_hash=sha512
- ima_template=ima-ng
- init_on_alloc=1
- initrd=initramfs.xz
- nvme_core.io_timeout=4294967295
- printk.devkmsg=on
- pti=on
- random.trust_cpu=on
- slab_nomerge=
- talos.platform=metal
initrd:
url: "https://github.com/talos-systems/talos/releases/download/v0.13.3/initramfs-amd64.xz"
sha512: ""
Example of overriding "default" Environment at the Server level:
apiVersion: metal.sidero.dev/v1alpha1
kind: Server
...
spec:
environmentRef:
namespace: default
name: boot
...
Example of overriding "default" Environment at the ServerClass level:
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
...
spec:
environmentRef:
namespace: default
name: boot
...
Servers are the basic resource of bare metal in the Metal Controller Manager. These are created by PXE booting the servers and allowing them to send a registration request to the management plane.
An example server may look like the following:
apiVersion: metal.sidero.dev/v1alpha1
kind: Server
metadata:
name: 00000000-0000-0000-0000-d05099d333e0
labels:
common-label: "true"
zone: east
environment: test
spec:
accepted: false
configPatches:
- op: replace
path: /cluster/network/cni
value:
name: custom
urls:
- http://192.168.1.199/assets/cilium.yaml
cpu:
manufacturer: Intel(R) Corporation
version: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz
system:
manufacturer: Dell Inc.
An installation disk is required by Talos on bare metal.
This can be specified in a configPatch:
apiVersion: metal.sidero.dev/v1alpha1
kind: Server
...
spec:
accepted: false
configPatches:
- op: replace
path: /machine/install/disk
value: /dev/sda
The install disk patch can also be set on the ServerClass:
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
...
spec:
configPatches:
- op: replace
path: /machine/install/disk
value: /dev/sda
In order for a server to be eligible for consideration, it must be accepted.
This is an important separation point which all Servers must pass.
Before a Server is accepted, no write action will be performed against it.
Thus, it is safe for a computer to be added to a network on which Sidero is operating.
Sidero will never write to or wipe any disk on a computer which is not marked as accepted.
This can be tedious for systems in which all attached computers should be considered to be under the control of Sidero.
Thus, you may also choose to automatically accept any machine into Sidero on its discovery.
Please keep in mind that this means that any newly-connected computer WILL BE WIPED automatically.
You can enable auto-acceptance by passing the --auto-accept-servers=true flag to sidero-controller-manager.
Once accepted, a server will be reset (all disks wiped) and then made available to Sidero.
You should never change an accepted Server to be not accepted while it is in use.
Because servers which are not accepted will not be modified, if a server which
was accepted is changed to not accepted, the disk will not be wiped upon
its exit.
Sidero can use IPMI information to control Server power state, reboot servers and set boot order.
IPMI information will be, by default, setup automatically if possible as part of the acceptance process.
In this design, a “sidero” user will be added to the IPMI user list and a randomly generated password will be issued.
This information is then squirreled away in a Kubernetes secret in the sidero-system namespace, with a name format of <server-uuid>-bmc.
Users wishing to turn off this feature can pass the --auto-bmc-setup=false flag to sidero-controller-manager
IPMI connection information can also be set manually in the Server spec after initial registration:
apiVersion: metal.sidero.dev/v1alpha1
kind: Server
...
spec:
bmc:
endpoint: 10.0.0.25
user: admin
pass: password
If IPMI information is set, server boot order might be set to boot from disk, then network, Sidero will switch servers to PXE boot once that is required.
Without IPMI info, Sidero can still register servers, wipe them and provision clusters, but Sidero won’t be able to reboot servers once they are removed from the cluster. If IPMI info is not set, servers should be configured to boot first from network, then from disk.
Sidero can also fetch IPMI credentials via the Secret reference:
apiVersion: metal.sidero.dev/v1alpha1
kind: Server
...
spec:
bmc:
endpoint: 10.0.0.25
userFrom:
secretKeyRef:
name: ipmi-credentials
key: username
passFrom:
secretKeyRef:
name: ipmi-credentials
key: password
As the Server resource is not namespaced, Secret should be created in the default namespace.
Server classes are a way to group distinct server resources.
The qualifiers and selector keys allow the administrator to specify criteria upon which to group these servers.
If both of these keys are missing, the server class matches all servers that it is watching.
If both of these keys define requirements, these requirements are combined (logical AND).
selectorselector groups server resources by their labels.
The Kubernetes documentation has more information on how to use this field.
qualifiersThere are currently two keys: cpu, systemInformation.
Each of these keys accepts a list of entries.
The top level keys are a “logical AND”, while the lists under each key are a “logical OR”.
Qualifiers that are not specified are not evaluated.
An example:
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
name: serverclass-sample
spec:
selector:
matchLabels:
common-label: "true"
matchExpressions:
- key: zone
operator: In
values:
- central
- east
- key: environment
operator: NotIn
values:
- prod
qualifiers:
cpu:
- manufacturer: "Intel(R) Corporation"
version: "Intel(R) Atom(TM) CPU C3558 @ 2.20GHz"
- manufacturer: Advanced Micro Devices, Inc.
version: AMD Ryzen 7 2700X Eight-Core Processor
systemInformation:
- manufacturer: Dell Inc.
Servers would only be added to the above class if they:
matchLabelsmatchExpressionsAdditionally, Sidero automatically creates and maintains a server class called "any" that includes all (accepted) servers.
Attempts to add qualifiers to it will be reverted.
configPatchesServer configs of servers matching a server class can be updated by using the configPatches section of the custom resource.
See patching for more information on how this works.
An example of settings the default install disk for all servers matching a server class:
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
...
spec:
configPatches:
- op: replace
path: /machine/install/disk
value: /dev/sda
The Sidero controller manager manages the Machine metadata. In terms of Talos (the OS on which the Kubernetes cluster is formed), this is the “machine config”, which is used during the automated installation.
The configuration of each machine is constructed from a number of sources:
TalosControlPlane custom resource for control plane nodes.TalosConfigTemplate custom resource.ServerClass which was used to select the Server into the Cluster.Server-specific patches.An example usage of setting a virtual IP for the control plane nodes and adding extra node-labels to nodes is shown below:
Note: because of the way JSON patches work the interface setting also needs to be set in
TalosControlPlanewhen defining a Virtual IP. This experience is not ideal, but will be addressed in a future release.
TalosControlPlane custom resource:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
name: workload-cluster
namespace: default
spec:
controlPlaneConfig:
controlplane:
configPatches:
- op: add
path: /machine/network
value:
interfaces:
- interface: eth0
dhcp: true
vip:
ip: 172.16.200.52
generateType: controlplane
talosVersion: v0.13
init:
configPatches:
- op: add
path: /machine/network
value:
interfaces:
- interface: eth0
dhcp: true
vip:
ip: 172.16.200.52
generateType: init
talosVersion: v0.13
infrastructureTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: MetalMachineTemplate
name: workload-cluster
replicas: 3
version: v1.23.0
TalosConfigTemplate custom resource:
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: TalosConfigTemplate
metadata:
name: workload-cluster
namespace: default
spec:
template:
spec:
generateType: join
talosVersion: v0.13
configPatches:
- op: add
path: /machine/kubelet
value:
extraArgs:
node-labels:
talos.dev/part-of: cluster/workload-cluster
and finally in the control plane ServerClass custom resource we augment the network information for other interfaces:
---
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
name: cp.small.x86
spec:
configPatches:
- op: replace
path: /machine/install/disk
value: /dev/nvme0n1
- op: add
path: /machine/install/extraKernelArgs
value:
- console=tty0
- console=ttyS1,115200n8
- op: add
path: /machine/network/interfaces/-
value:
interface: eth1
dhcp: true
qualifiers:
cpu:
- version: Intel(R) Xeon(R) E-2124G CPU @ 3.40GHz
systemInformation:
- manufacturer: Supermicro
selector:
matchLabels:
metal.sidero.dev/serverclass: cp.small.x86
the workload ServerClass defines the complete networking config
---
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
name: general.medium.x86
spec:
configPatches:
- op: replace
path: /machine/install/disk
value: /dev/nvme1n1
- op: add
path: /machine/install/extraKernelArgs
value:
- console=tty0
- console=ttyS1,115200n8
- op: add
path: /machine/network
value:
interfaces:
- interface: eth0
dhcp: true
- interface: eth1
dhcp: true
qualifiers:
cpu:
- version: Intel(R) Xeon(R) E-2136 CPU @ 3.30GHz
systemInformation:
- manufacturer: Supermicro
selector:
matchLabels:
metal.sidero.dev/serverclass: general.medium.x86
The base template is constructed from the Talos bootstrap provider, using data from the associated TalosControlPlane and TalosConfigTemplate manifest.
Then, any configuration patches are applied from the ServerClass and Server.
These patches take the form of an RFC 6902 JSON (or YAML) patch. An example of the use of this patch method can be found in Patching Guide.
Also note that while a Server can be a member of any number of ServerClasses, only the ServerClass which is used to select the Server into the Cluster will be used for the generation of the configuration of the Machine.
In this way, Servers may have a number of different configuration patch sets based on which Cluster they are in at any given time.
Imagine a scenario in which you have shown up to a datacenter with only a laptop and your task is to transition a rack of bare metal machines into an HA management plane and multiple Kubernetes clusters created by that management plane. In this guide, we will go through how to create a bootstrap cluster using a Docker-based Talos cluster, provision the management plane, and pivot over to it. Guides around post-pivoting setup and subsequent cluster creation should also be found in the “Guides” section of the sidebar.
Because of the design of Cluster API, there is inherently a “chicken and egg” problem with needing a Kubernetes cluster in order to provision the management plane. Talos Systems and the Cluster API community have created tools to help make this transition easier.
First, you need to install the latest talosctl by running the following script:
curl -Lo /usr/local/bin/talosctl https://github.com/talos-systems/talos/releases/latest/download/talosctl-$(uname -s | tr "[:upper:]" "[:lower:]")-amd64
chmod +x /usr/local/bin/talosctl
You can read more about Talos and talosctl at talos.dev.
Next, there are two big prerequisites involved with bootstrapping Sidero: routing and DHCP setup.
From the routing side, the laptop from which you are bootstrapping must be accessible by the bare metal machines that we will be booting. In the datacenter scenario described above, the easiest way to achieve this is probably to hook the laptop onto the server rack’s subnet by plugging it into the top-of-rack switch. This is needed for TFTP, PXE booting, and for the ability to register machines with the bootstrap plane.
DHCP configuration is needed to tell the metal servers what their “next server” is when PXE booting. The configuration of this is different for each environment and each DHCP server, thus it’s impossible to give an easy guide. However, here is an example of the configuration for an Ubiquti EdgeRouter that uses vyatta-dhcpd as the DHCP service:
This block shows the subnet setup, as well as the extra “subnet-parameters” that tell the DHCP server to include the ipxe-metal.conf file.
These commands are run under the
configureoption in EdgeRouter
$ show service dhcp-server shared-network-name MetalDHCP
authoritative enable
subnet 192.168.254.0/24 {
default-router 192.168.254.1
dns-server 192.168.1.200
lease 86400
start 192.168.254.2 {
stop 192.168.254.252
}
subnet-parameters "include "/config/ipxe-metal.conf";"
}
Here is the ipxe-metal.conf file.
$ cat /config/ipxe-metal.conf
allow bootp;
allow booting;
next-server 192.168.1.150;
filename "ipxe.efi"; # use "undionly.kpxe" for BIOS netboot or "ipxe.efi" for UEFI netboot
host talos-mgmt-0 {
fixed-address 192.168.254.2;
hardware ethernet d0:50:99:d3:33:60;
}
If you want to boot multiple architectures, you can use the DHCP Option 93 to specify the architecture.
First we need to define option 93 in the DHCP server configuration.
set service dhcp-server global-parameters "option system-arch code 93 = unsigned integer 16;"
Now we can specify condition based on option 93 in ipxe-metal.conf file
$ cat /config/ipxe-metal.conf
allow bootp;
allow booting;
next-server 192.168.1.150;
if option system-arch = 00:0b {
filename "ipxe-arm64.efi";
} else {
filename "ipxe.efi";
}
host talos-mgmt-0 {
fixed-address 192.168.254.2;
hardware ethernet d0:50:99:d3:33:60;
}
Notice that it sets a static address for the management node that I’ll be booting, in addition to providing the “next server” info.
This “next server” IP address will match references to PUBLIC_IP found below in this guide.
The talosctl CLI tool has built-in support for spinning up Talos in docker containers.
Let’s use this to our advantage as an easy Kubernetes cluster to start from.
Set an environment variable called PUBLIC_IP which is the “public” IP of your machine.
Note that “public” is a bit of a misnomer.
We’re really looking for the IP of your machine, not the IP of the node on the docker bridge (ex: 192.168.1.150).
export PUBLIC_IP="192.168.1.150"
We can now create our Docker cluster. Issue the following to create a single-node cluster:
talosctl cluster create \
--kubernetes-version 1.22.2 \
-p 69:69/udp,8081:8081/tcp \
--workers 0 \
--endpoint $PUBLIC_IP
Note that there are several ports mentioned in the command above. These allow us to access the services that will get deployed on this node.
Once the cluster create command is complete, issue talosctl kubeconfig /desired/path to fetch the kubeconfig for this cluster.
You should then set your KUBECONFIG environment variable to the path of this file.
Because this is a single node cluster, we need to remove the “NoSchedule” taint on the node to make sure non-controlplane components can be scheduled.
kubectl taint node talos-default-master-1 node-role.kubernetes.io/master:NoSchedule-
As of Cluster API version 0.3.9, Sidero is included as a default infrastructure provider in clusterctl.
To install Sidero and the other Talos providers, simply issue:
SIDERO_CONTROLLER_MANAGER_HOST_NETWORK=true \
SIDERO_CONTROLLER_MANAGER_API_ENDPOINT=$PUBLIC_IP \
clusterctl init -b talos -c talos -i sidero
We will now want to ensure that the Sidero services that got created are publicly accessible across our subnet. These variables above will allow the metal machines to speak to these services later.
At this point, any servers on the same network as Sidero should PXE boot using the Sidero PXE service.
To register a server with Sidero, simply turn it on and Sidero will do the rest.
Once the registration is complete, you should see the servers registered with kubectl get servers:
$ kubectl get servers -o wide
NAME HOSTNAME ACCEPTED ALLOCATED CLEAN
00000000-0000-0000-0000-d05099d33360 192.168.254.2 false false false
Sidero can use IPMI information to control Server power state, reboot servers and set boot order. IPMI information will be, by default, setup automatically if possible as part of the acceptance process. See IPMI for more information.
IPMI connection information can also be set manually in the Server spec after initial registration:
kubectl patch server 00000000-0000-0000-0000-d05099d33360 --type='json' -p='[{"op": "add", "path": "/spec/bmc", "value": {"endpoint": "192.168.88.9", "user": "ADMIN", "pass":"ADMIN"}}]'
If IPMI info is not set, servers should be configured to boot first from network, then from disk.
Note that for bare-metal setup, you would need to specify an installation disk. See Installation Disk for details on how to do this. You should configure this before accepting the server.
Note in the output above that the newly registered servers are not accepted.
In order for a server to be eligible for consideration, it must be marked as accepted.
Before a Server is accepted, no write action will be performed against it.
Servers can be accepted by issuing a patch command like:
kubectl patch server 00000000-0000-0000-0000-d05099d33360 --type='json' -p='[{"op": "replace", "path": "/spec/accepted", "value": true}]'
For more information on server acceptance, see the server docs.
We are now ready to template out our management plane. Using clusterctl, we can create a cluster manifest with:
clusterctl generate cluster management-plane -i sidero > management-plane.yaml
Note that there are several variables that should be set in order for the templating to work properly:
CONTROL_PLANE_ENDPOINT and CONTROL_PLANE_PORT: The endpoint (IP address or hostname) and the port used for the Kubernetes API server
(e.g. for https://1.2.3.4:6443: CONTROL_PLANE_ENDPOINT=1.2.3.4 and CONTROL_PLANE_PORT=6443).
This is the equivalent of the endpoint you would specify in talosctl gen config.
There are a variety of ways to configure a control plane endpoint.
Some common ways for an HA setup are to use DNS, a load balancer, or BGP.
A simpler method is to use the IP of a single node.
This has the disadvantage of being a single point of failure, but it can be a simple way to get running.CONTROL_PLANE_SERVERCLASS: The server class to use for control plane nodes.WORKER_SERVERCLASS: The server class to use for worker nodes.KUBERNETES_VERSION: The version of Kubernetes to deploy (e.g. v1.22.2).CONTROL_PLANE_PORT: The port used for the Kubernetes API server (port 6443)TALOS_VERSION: This should correspond to the minor version of Talos that you will be deploying (e.g. v0.13).
This value is used in determining the fields present in the machine configuration that gets generated for Talos nodes.For instance:
export CONTROL_PLANE_SERVERCLASS=any
export WORKER_SERVERCLASS=any
export TALOS_VERSION=v0.13
export KUBERNETES_VERSION=v1.22.2
export CONTROL_PLANE_PORT=6443
export CONTROL_PLANE_ENDPOINT=1.2.3.4
clusterctl generate cluster management-plane -i sidero > management-plane.yaml
In addition, you can specify the replicas for control-plane & worker nodes in management-plane.yaml manifest for TalosControlPlane and MachineDeployment objects.
Also, they can be scaled if needed (after applying the management-plane.yaml manifest):
kubectl get taloscontrolplane
kubectl get machinedeployment
kubectl scale taloscontrolplane management-plane-cp --replicas=3
Now that we have the manifest, we can simply apply it:
kubectl apply -f management-plane.yaml
NOTE: The templated manifest above is meant to act as a starting point. If customizations are needed to ensure proper setup of your Talos cluster, they should be added before applying.
Once the management plane is setup, you can fetch the talosconfig by using the cluster label. Be sure to update the cluster name and issue the following command:
kubectl get talosconfig \
-l cluster.x-k8s.io/cluster-name=<CLUSTER NAME> \
-o yaml -o jsonpath='{.items[0].status.talosConfig}' > management-plane-talosconfig.yaml
With the talosconfig in hand, the management plane’s kubeconfig can be fetched with talosctl --talosconfig management-plane-talosconfig.yaml kubeconfig
Once we have the kubeconfig for the management cluster, we now have the ability to pivot the cluster from our bootstrap. Using clusterctl, issue:
clusterctl init --kubeconfig=/path/to/management-plane/kubeconfig -i sidero -b talos -c talos
Followed by:
clusterctl move --to-kubeconfig=/path/to/management-plane/kubeconfig
Upon completion of this command, we can now tear down our bootstrap cluster with talosctl cluster destroy and begin using our management plane as our point of creation for all future clusters!
This guide will provide some very basic detail about how you can also build a Sidero management plane using the Talos ISO image instead of following the Docker-based process that we detail in our Getting Started tutorials.
Using the ISO is a perfectly valid way to build a Talos cluster, but this approach is not recommended for Sidero as it avoids the “pivot” step detailed here. Skipping this step means that the management plane does not become “self-hosted”, in that it cannot be upgraded and scaled using the Sidero processes we follow for workload clusters. For folks who are willing to take care of their management plane in other ways, however, this approach will work fine.
The rough outline of this process is very short and sweet, as it relies on other documentation:
For each management plane node, boot the ISO and install Talos using the “apply-config” process mentioned in our Talos Getting Started docs. These docs go into heavy detail on using the ISO, so they will not be recreated here.
With a Kubernetes cluster now in hand (and with access to it via talosctl and kubectl), you can simply pickup the Getting Started tutorial at the “Install Sidero” section here.
Keep in mind, however, that you will be unable to do the “pivoting” section of the tutorial, so just skip that step when you reach the end of the tutorial.
Note: It may also be of interest to view the prerequisite guides on CLI and DHCP setup, as they will still apply to this method.
This guide will detail the process for removing a server from Sidero. The process is fairly simple with a few pieces of information.
For the given server, take note of any serverclasses that are configured to match the server.
Take note of any clusters that make use of aforementioned serverclasses.
For each matching cluster, edit the cluster resource with kubectl edit cluster and set .spec.paused to true.
Doing this ensures that no new machines will get created for these servers during the decommissioning process.
If the server is already part of a cluster (kubectl get serverbindings should provide this info), you can now delete the machine that corresponds with this server via kubectl delete machine <machine_name>.
With the machine deleted, Sidero will reboot the machine and wipe its disks.
Once the disk wiping is complete and the server is turned off, you can finally delete the server from Sidero with kubectl delete server <server_name> and repurpose the server for something else.
Finally, unpause any clusters that were edited in step 3 by setting .spec.paused to false.
This guide will detail the steps needed to provision your first bare metal Talos cluster after completing the bootstrap and pivot steps detailed in the previous guide. There will be two main steps in this guide: reconfiguring the Sidero components now that they have been pivoted and the actual cluster creation.
In this guide, we will convert the services to use host networking. This is also necessary because some protocols like TFTP don’t allow for port configuration. Along with some nodeSelectors and a scale up of the metal controller manager deployment, creating the services this way allows for the creation of DNS names that point to all management plane nodes and provide an HA experience if desired. It should also be noted, however, that there are many options for achieving this functionality. Users can look into projects like MetalLB or KubeRouter with BGP and ECMP if they desire something else.
Metal Controller Manager:
## Use host networking
kubectl patch deploy -n sidero-system sidero-controller-manager --type='json' -p='[{"op": "add", "path": "/spec/template/spec/hostNetwork", "value": true}]'
Sidero by default appends talos.config kernel argument with based on the flags --api-endpoint and --api-port to the sidero-controller-manager:
talos.config=http://$API_ENDPOINT:$API_PORT/configdata?uuid=.
If this default value doesn’t apply, edit the environment with kubectl edit environment default and add the talos.config kernel arg with the IP of one of the management plane nodes (or the DNS entry you created).
The DHCP options configured in the previous guide should now be updated to point to your new management plane IP or to the DNS name if it was created.
A revised ipxe-metal.conf file looks like:
allow bootp;
allow booting;
next-server 192.168.254.2;
if exists user-class and option user-class = "iPXE" {
filename "http://192.168.254.2:8081/boot.ipxe";
} else {
if substring (option vendor-class-identifier, 15, 5) = "00000" {
# BIOS
if substring (option vendor-class-identifier, 0, 10) = "HTTPClient" {
option vendor-class-identifier "HTTPClient";
filename "http://192.168.254.2:8081/tftp/undionly.kpxe";
} else {
filename "undionly.kpxe";
}
} else {
# UEFI
if substring (option vendor-class-identifier, 0, 10) = "HTTPClient" {
option vendor-class-identifier "HTTPClient";
filename "http://192.168.254.2:8081/tftp/ipxe.efi";
} else {
filename "ipxe.efi";
}
}
}
host talos-mgmt-0 {
fixed-address 192.168.254.2;
hardware ethernet d0:50:99:d3:33:60;
}
There are multiple ways to boot the via iPXE:
http://192.168.254.2:8081/boot.ipxe.ipxe.efi or undionly.kpxe can be used (these images contain embedded iPXE scripts).At this point, any servers on the same network as Sidero should PXE boot using the Sidero PXE service.
To register a server with Sidero, simply turn it on and Sidero will do the rest.
Once the registration is complete, you should see the servers registered with kubectl get servers:
$ kubectl get servers -o wide
NAME HOSTNAME ACCEPTED ALLOCATED CLEAN
00000000-0000-0000-0000-d05099d33360 192.168.254.2 false false false
Note in the output above that the newly registered servers are not accepted.
In order for a server to be eligible for consideration, it must be marked as accepted.
Before a Server is accepted, no write action will be performed against it.
Servers can be accepted by issuing a patch command like:
kubectl patch server 00000000-0000-0000-0000-d05099d33360 --type='json' -p='[{"op": "replace", "path": "/spec/accepted", "value": true}]'
For more information on server acceptance, see the server docs.
The cluster creation process should be identical to what was detailed in the previous guide. Using clusterctl, we can create a cluster manifest with:
clusterctl generate cluster workload-cluster -i sidero > workload-cluster.yaml
Note that there are several variables that should be set in order for the templating to work properly:
CONTROL_PLANE_ENDPOINT and CONTROL_PLANE_PORT: The endpoint (IP address or hostname) and the port used for the Kubernetes API server
(e.g. for https://1.2.3.4:6443: CONTROL_PLANE_ENDPOINT=1.2.3.4 and CONTROL_PLANE_PORT=6443).
This is the equivalent of the endpoint you would specify in talosctl gen config.
There are a variety of ways to configure a control plane endpoint.
Some common ways for an HA setup are to use DNS, a load balancer, or BGP.
A simpler method is to use the IP of a single node.
This has the disadvantage of being a single point of failure, but it can be a simple way to get running.CONTROL_PLANE_SERVERCLASS: The server class to use for control plane nodes.WORKER_SERVERCLASS: The server class to use for worker nodes.KUBERNETES_VERSION: The version of Kubernetes to deploy (e.g. v1.19.4).TALOS_VERSION: This should correspond to the minor version of Talos that you will be deploying (e.g. v0.10).
This value is used in determining the fields present in the machine configuration that gets generated for Talos nodes.
Note that the default is currently v0.13.Now that we have the manifest, we can simply apply it:
kubectl apply -f workload-cluster.yaml
NOTE: The templated manifest above is meant to act as a starting point. If customizations are needed to ensure proper setup of your Talos cluster, they should be added before applying.
Once the workload cluster is setup, you can fetch the talosconfig with a command like:
kubectl get talosconfig -o yaml workload-cluster-cp-xxx -o jsonpath='{.status.talosConfig}' > workload-cluster-talosconfig.yaml
Then the workload cluster’s kubeconfig can be fetched with talosctl --talosconfig workload-cluster-talosconfig.yaml kubeconfig /desired/path.
Server resources can be updated by using the configPatches section of the custom resource.
Any field of the Talos machine config
can be overridden on a per-machine basis using this method.
The format of these patches is based on JSON 6902 that you may be used to in tools like kustomize.
Any patches specified in the server resource are processed by the Sidero controller before it returns a Talos machine config for a given server at boot time.
A set of patches may look like this:
apiVersion: metal.sidero.dev/v1alpha1
kind: Server
metadata:
name: 00000000-0000-0000-0000-d05099d33360
spec:
configPatches:
- op: replace
path: /machine/install
value:
disk: /dev/sda
- op: replace
path: /cluster/network/cni
value:
name: "custom"
urls:
- "http://192.168.1.199/assets/cilium.yaml"
While developing config patches it is usually convenient to test generated config with patches before actual server is provisioned with the config.
This can be achieved by querying the metadata server endpoint directly:
$ curl http://$PUBLIC_IP:8081/configdata?uuid=$SERVER_UUID
version: v1alpha1
...
Replace $PUBLIC_IP with the Sidero IP address and $SERVER_UUID with the name of the Server to test
against.
If metadata endpoint returns an error on applying JSON patches, make sure config subtree being patched exists in the config.
If it doesn’t exist, create it with the op: add above the op: replace patch.
Config patches might be combined from multiple sources (Server, ServerClass, TalosControlPlane, TalosConfigTemplate), which is explained in details
in Metadata section.
graph TD;
Start(Start);
End(End);
%% Decisions
IsOn{Is server is powered on?};
IsRegistered{Is server is registered?};
IsAccepted{Is server is accepted?};
IsClean{Is server is clean?};
IsAllocated{Is server is allocated?};
%% Actions
DoPowerOn[Power server on];
DoPowerOff[Power server off];
DoBootAgentEnvironment[Boot agent];
DoBootEnvironment[Boot environment];
DoRegister[Register server];
DoWipe[Wipe server];
%% Chart
Start-->IsOn;
IsOn--Yes-->End;
IsOn--No-->DoPowerOn;
DoPowerOn--->IsRegistered;
IsRegistered--Yes--->IsAccepted;
IsRegistered--No--->DoBootAgentEnvironment-->DoRegister;
DoRegister-->IsRegistered;
IsAccepted--Yes--->IsAllocated;
IsAccepted--No--->End;
IsAllocated--Yes--->DoBootEnvironment;
IsAllocated--No--->IsClean;
IsClean--No--->DoWipe-->DoPowerOff;
IsClean--Yes--->DoPowerOff;
DoBootEnvironment-->End;
DoPowerOff-->End;
graph TD;
Start(Start);
End(End);
%% Decisions
IsInstalled{Is installed};
%% Actions
DoInstall[Install];
DoReboot[Reboot];
%% Chart
Start-->IsInstalled;
IsInstalled--Yes-->End;
IsInstalled--No-->DoInstall;
DoInstall-->DoReboot;
DoReboot-->IsInstalled;
Upgrading a running workload cluster or management plane is the same process as describe in the Talos documentation.
To upgrade the Talos OS, see here.
In order to upgrade Kubernetes itself, see here.
It is important, however, to take special consideration for upgrades of the Talos v0.8.x series to v0.9.x. Because of the move from self-hosted control plane to static pods, some certificate information has changed that needs to be manually updated. The steps are as follows:
Upgrade a single control plane node to the v0.9.x series using the upgrade instructions above. upgrade
After upgrade, carry out a talosctl convert-k8s to move from the self-hosted control plane to static pods.
Targeting the upgraded node, issue talosctl read -n <node-ip> /system/state/config.yaml and copy out the cluster.aggregatorCA and cluster.serviceAccount sections.
In the management cluster, issue kubectl edit secret <cluster-name>-talos.
While in editing view, copy the data.certs field and decode it with echo '<certs-content>' | base64 -d
Note: It may also be a good idea to copy the secret in its entirety as a backup. This can be done with a simple
kubectl get secret <cluster-name>-talos -o yaml.
admin:
crt: xxx
key: xxx
etcd:
crt: xxx
key: xxx
k8s:
crt: xxx
key: xxx
k8saggregator:
crt: xxx
key: xxx
k8sserviceaccount:
key: xxx
os:
crt: xxx
key: xxx
Re-encode the data with cat <saved-file> | base64 | tr -d '\n'
With the secret still open for editing, update the data.certs field to contain the new base64 data.
Edit the cluster’s TalosControlPlane resource with kubectl edit tcp <name-of-control-plane>.
Update the spec.controlPlaneConfig.[controlplane,init].talosVersion fields to be v0.9.
Edit any TalosConfigTemplate resources and update spec.template.spec.talosVersion to be the same value.
At this point, any new controlplane or worker machines should receive the newer machine config format and join the cluster successfully. You can also proceed to upgrade existing nodes.
This guide will explain on how to use Sidero to manage Raspberrypi-4’s as servers. This guide goes hand in hand with the bootstrapping guide.
From the bootstrapping guide, reach “Install Sidero” and come back to this guide. Once you finish with this guide, you will need to go back to the bootstrapping guide and continue with “Register the servers”.
The rest of this guide goes with the assumption that you’ve a cluster setup with Sidero and ready to accept servers. This guide will explain the changes that needs to be made to be able to accept RPI4 as server.
To be able to boot talos on the Pi4 via network, we need to undergo a 2-step boot process. The Pi4 has an EEPROM which contains code to boot up the Pi. This EEPROM expects a specific boot folder structure as explained on this page. We will use the EEPROM to boot into UEFI, which we will then use to PXE and iPXE boot into sidero & talos.
NOTE: If you’ve updated the EEPROM with the image that was referenced on the talos docs,
you can either flash it with the one mentioned below, or visit the EEPROM config docs
and change the boot order of EEPROM to 0xf21.
Which means try booting from SD first, then try network.
To enable the EEPROM on the Pi to support network booting, we must update it to
the latest version.
Visit the release page and grab the
latest rpi-boot-eeprom-recovery-*-network.zip (as of time of writing,
v2021.0v.29-138a1 was used).
Put this on a SD card and plug it into the Pi.
The
Pi’s status light will flash rapidly after a few seconds, this indicates that
the EEPROM has been updated.
This operation needs to be done once per Pi.
Power on the Pi without an SD card in it and hook it up to a monitor, you will be greeted with the boot screen. On this screen you will find some information about the Pi. For this guide, we are only interested in the serial number. The first line under the Pi logo will be something like the following:
board: xxxxxx <serial> <MAC address>
Write down the 8 character serial.
Clone the talos-systems/pkg repo.
Create a new folder called raspberrypi4-uefi and raspberrypi4-uefi/serials.
Create a file raspberrypi4-uefi/pkg.yaml containing the following:
name: raspberrypi4-uefi
variant: alpine
install:
- unzip
steps:
# {{ if eq .ARCH "aarch64" }} This in fact is YAML comment, but Go templating instruction is evaluated by bldr restricting build to arm64 only
- sources:
- url: https://github.com/pftf/RPi4/releases/download/v1.26/RPi4_UEFI_Firmware_v1.26.zip # <-- update version NR accordingly.
destination: RPi4_UEFI_Firmware.zip
sha256: d6db87484dd98dfbeb64eef203944623130cec8cb71e553eab21f8917e0285f7
sha512: 96a71086cdd062b51ef94726ebcbf15482b70c56262555a915499bafc04aff959d122410af37214760eda8534b58232a64f6a8a0a8bb99aba6de0f94c739fe98
prepare:
- |
unzip RPi4_UEFI_Firmware.zip
rm RPi4_UEFI_Firmware.zip
mkdir /rpi4
mv ./* /rpi4
install:
- |
mkdir /tftp
ls /pkg/serials | while read serial; do mkdir /tftp/$serial && cp -r /rpi4/* /tftp/$serial && cp -r /pkg/serials/$serial/* /tftp/$serial/; done
# {{ else }}
- install:
- |
mkdir -p /tftp
# {{ end }}
finalize:
- from: /
to: /
Now that the EEPROM can network boot, we need to prepare the structure of our boot folder. Essentially what the bootloader will do is look for this folder on the network rather than on the SD card.
Visit the release page of RPi4 and grab
the latest RPi4_UEFI_Firmware_v*.zip (at the time of writing, v1.26 was used).
Extract the zip into a folder, the structure will look like the following:
.
├── RPI_EFI.fd
├── RPi4_UEFI_Firmware_v1.26.zip
├── Readme.md
├── bcm2711-rpi-4-b.dtb
├── bcm2711-rpi-400.dtb
├── bcm2711-rpi-cm4.dtb
├── config.txt
├── firmware
│ ├── LICENCE.txt
│ ├── Readme.txt
│ ├── brcmfmac43455-sdio.bin
│ ├── brcmfmac43455-sdio.clm_blob
│ └── brcmfmac43455-sdio.txt
├── fixup4.dat
├── overlays
│ └── miniuart-bt.dtbo
└── start4.elf
As a one time operation, we need to configure UEFI to do network booting by
default, remove the 3gb mem limit if it’s set and optionally set the CPU clock to
max.
Take these files and put them on the SD card and boot the Pi.
You will see the Pi logo, and the option to hit esc.
Disabled.Max.UEFI PXEv4 is the first boot option.Now that we have made the changes above, we need to persist these changes.
Go back to the home screen and hit reset to save the changes to disk.
When you hit reset, the settings will be saved to the RPI_EFI.fd file on the
SD card.
This is where we will run into a limitation that is explained in the
following issue: pftf/RPi4#59.
What this mean is that we need to create a RPI_EFI.fd file for each Pi that we want to use as server.
This is because the MAC address is also stored in the RPI_EFI.fd file,
which makes it invalid when you try to use it in a different Pi.
Plug the SD card back into your computer and extract the RPI_EFI.fd file from
it and place it into the raspberrypi4-uefi/serials/<serial>/.
The dir should look like this:
raspberrypi4-uefi/
├── pkg.yaml
└── serials
└─── XXXXXXXX
└── RPI_EFI.fd
Now that we have the RPI_EFI.fd of our Pi in the correct location, we must now
build a docker image containing the boot folder for the EEPROM.
To do this, run the following command in the pkgs repo:
make PLATFORM=linux/arm64 USERNAME=$USERNAME PUSH=true TARGETS=raspberrypi4-uefi
This will build and push the following image:
ghcr.io/$USERNAME/raspberrypi4-uefi:<tag>
If you need to change some other settings like registry etc, have a look in the Makefile to see the available variables that you can override.
The content of the /tftp folder in the image will be the following:
XXXXXXXX
├── RPI_EFI.fd
├── Readme.md
├── bcm2711-rpi-4-b.dtb
├── bcm2711-rpi-400.dtb
├── bcm2711-rpi-cm4.dtb
├── config.txt
├── firmware
│ ├── LICENCE.txt
│ ├── Readme.txt
│ ├── brcmfmac43455-sdio.bin
│ ├── brcmfmac43455-sdio.clm_blob
│ └── brcmfmac43455-sdio.txt
├── fixup4.dat
├── overlays
│ └── miniuart-bt.dtbo
└── start4.elf
To enable the 2 boot process, we need to include this EEPROM boot folder into the sidero’s tftp folder. To achieve this, we will use an init container using the image we created above to copy the contents of it into the tftp folder.
Create a file patch.yaml with the following contents:
spec:
template:
spec:
volumes:
- name: tftp-folder
emptyDir: {}
initContainers:
- image: ghcr.io/<USER>/raspberrypi4-uefi:v<TAG> # <-- change accordingly.
imagePullPolicy: Always
name: tftp-folder-setup
command:
- cp
args:
- -r
- /tftp
- /var/lib/sidero/
volumeMounts:
- mountPath: /var/lib/sidero/tftp
name: tftp-folder
containers:
- name: manager
volumeMounts:
- mountPath: /var/lib/sidero/tftp
name: tftp-folder
Followed by this command to apply the patch:
kubectl -n sidero-system patch deployments.apps sidero-controller-manager --patch "$(cat patch.yaml)"
With the patched metal controller, you should now be able to register the Pi4 to sidero by just connecting it to the network. From this point you can continue with the bootstrapping guide.
Sidero doesn’t require a lot of computing resources, so SBCs are a perfect fit to run the Sidero management cluster. In this guide, we are going to install Talos on Raspberry Pi4, deploy Sidero and other CAPI components.
Please see Talos documentation for additional information on installing Talos on Raspberry Pi4.
Download the clusterctl CLI from CAPI releases.
The minimum required version is 0.4.3.
Prepare the SD card with the Talos RPi4 image, and boot the RPi4.
Talos should drop into maintenance mode printing the acquired IP address.
Record the IP address as the environment variable SIDERO_ENDPOINT:
export SIDERO_ENDPOINT=192.168.x.x
Note: it makes sense to transform DHCP lease for RPi4 into a static reservation so that RPi4 always has the same IP address.
Generate Talos machine configuration for a single-node cluster:
talosctl gen config --config-patch='[{"op": "add", "path": "/cluster/allowSchedulingOnMasters", "value": true},{"op": "replace", "path": "/machine/install/disk", "value": "/dev/mmcblk0"}]' rpi4-sidero https://${SIDERO_ENDPOINT}:6443/
Submit the generated configuration to Talos:
talosctl apply-config --insecure -n ${SIDERO_ENDPOINT} -f controlplane.yaml
Merge client configuration talosconfig into default ~/.talos/config location:
talosctl config merge talosconfig
Update default endpoint and nodes:
talosctl config endpoints ${SIDERO_ENDPOINT}
talosctl config nodes ${SIDERO_ENDPOINT}
You can verify that Talos has booted by running:
$ talosctl version
talosctl version
Client:
Tag: v0.10.3
SHA: 21018f28
Built:
Go version: go1.16.3
OS/Arch: linux/amd64
Server:
NODE: 192.168.0.31
Tag: v0.10.3
SHA: 8f90c6a8
Built:
Go version: go1.16.3
OS/Arch: linux/arm64
Bootstrap the etcd cluster:
talosctl bootstrap
At this point, Kubernetes is bootstrapping, and it should be available once all the images are fetched.
Fetch the kubeconfig from the cluster with:
talosctl kubeconfig
You can watch the bootstrap progress by running:
talosctl dmesg -f
Once Talos prints [talos] boot sequence: done, Kubernetes should be up:
kubectl get nodes
Install Sidero with host network mode, exposing the endpoints on the node’s address:
SIDERO_CONTROLLER_MANAGER_HOST_NETWORK=true SIDERO_CONTROLLER_MANAGER_API_ENDPOINT=${SIDERO_IP} clusterctl init -i sidero -b talos -c talos
Watch the progress of installation with:
watch -n 2 kubectl get pods -A
Once images are downloaded, all pods should be in running state:
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cabpt-system cabpt-controller-manager-6458494888-d7lnm 1/1 Running 0 29m
cacppt-system cacppt-controller-manager-f98854db8-qgkf9 1/1 Running 0 29m
capi-system capi-controller-manager-58f797cb65-8dwpz 2/2 Running 0 30m
capi-webhook-system cabpt-controller-manager-85fd964c9c-ldzb6 1/1 Running 0 29m
capi-webhook-system cacppt-controller-manager-75c479b7f-5hw89 1/1 Running 0 29m
capi-webhook-system capi-controller-manager-7d596cc4cb-kjrfk 2/2 Running 0 30m
capi-webhook-system caps-controller-manager-79664cf677-zqbvw 1/1 Running 0 29m
cert-manager cert-manager-86cb5dcfdd-v86wr 1/1 Running 0 31m
cert-manager cert-manager-cainjector-84cf775b89-swk25 1/1 Running 0 31m
cert-manager cert-manager-webhook-7f9f4f8dcb-29xm4 1/1 Running 0 31m
kube-system coredns-fcc4c97fb-wkxkg 1/1 Running 0 35m
kube-system coredns-fcc4c97fb-xzqzj 1/1 Running 0 35m
kube-system kube-apiserver-talos-192-168-0-31 1/1 Running 0 33m
kube-system kube-controller-manager-talos-192-168-0-31 1/1 Running 0 33m
kube-system kube-flannel-qmlw6 1/1 Running 0 34m
kube-system kube-proxy-j24hg 1/1 Running 0 34m
kube-system kube-scheduler-talos-192-168-0-31 1/1 Running 0 33m
Verify Sidero installation and network setup with:
$ curl -I http://${SIDERO_ENDPOINT}:8081/tftp/ipxe.efi
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 1020416
Content-Type: application/octet-stream
Last-Modified: Thu, 03 Jun 2021 15:40:58 GMT
Date: Thu, 03 Jun 2021 15:41:51 GMT
Now Sidero is installed, and it is ready to be used.
Configure your DHCP server to PXE boot your bare metal servers from $SIDERO_ENDPOINT (see Bootstrapping guide on DHCP configuration).
SD cards are not very reliable, so make sure you are taking regular etcd backups, so that you can recover your Sidero installation in case of data loss.