š¶ Introduction
I will document my journey into the SRE world here, you will find notes from books I am reading, as well as other stuff I find relevant throughout the journey.
Becoming SRE
ā ļø These are notes taken from the book Becoming SRE by David N. Blank-Edelman O'Reilly. It is an splendid book and no amount of notes I take will make it justice, please go read it and just use this as a reference.
Table of Contents
- Introduction to SRE
- Becoming SRE for the Individual
Introduction to SRE
First things first
Site Reliability Engineering is an engineering discipline devoted to helpings orgs sustainably achieve the appropriate level of reliability in their systems, services and products.
3 key words from definition
- Reliability: even if you have the best software in the world, and the best
sales team, if the sw is not up when customers need it, you lose a lot of:
- Revenue
- Time
- Reputation
- Health
- Hiring
- Appropriate
- Usually 100% reliable is not achievable, it is likely that your dependencies are not 100% reliable.
- You need to use service level indicators/service level objectives (SLI/SLO) to help determine an appropriate level of reliability.
- Sustainable
- If the people in the org are burned out they wont be able to build reliable systems.
SRE vs Devops
-
SRE implements class devops
-
SRE is to reliability as devops is to delivery
-
Itās all about the direction
- Devops go from development to production operations
- SRE seems to start on prod operations, and going back to improve the result.
Basically these two phrases:
- Devops thinks, how can I deploy this from my laptop to production
- SRE starts at production, how can I make the environment more reliable.
SRE Mindset
It is born out of curiosity: How does a system work? How does it fail?
Zoom out and Zoom in as much as possible, how does theā¦ work
- The whole app, development and deployment process
- The entire service
- The infrastructure
- The physical overlay of the infra
- The sociotechnical context in which the infra runs
- The org context where the sociotechnical context resides in
Keeping Focus on the Customer
Small example, say you have a 100 servers running a front end pool, suddenly 14 systems fail and are not recovering. This situation is:
- No big deal; you can deal with it at your leisure
- Something that warrants your immediate attentions. Stop what you are doing and focus on this
- Crisis, even if 2 AM go wake the CEO
The answer, it depends, you need to ask, how does this affect the system as a whole. If the customers wont see anything diff then a. If visible by customers then b. If the service is dead in the water and this means no revenue, go wake the CEO.
So basically you need to ask: What is the intention of the system from the customerās perspective?
SREās Relationship with Failure
SRE are very collaborative. Reliability is a collaborative endeavour by nature.
The SRE mindset views errors as an opportunity to learn. Learning from failure is a core component of an SRE.
SRE treats errors as a signal and we like clear signals. With an SRE mindset errors can serve the helping understand the system better. This means we need to surface the errors in addition to eliminate them.
SRE feels ownership of the whole service, they do not say often ānot my code, not my problemā. This can be un arma de doble filo. Because of the yak shaving issue, where you go to do one task, say update a package, but then you check and for the new package you need to upgrade the OS, but to upgrade the OS you need toā¦ and you end up with a trimmer in your hands and a yak infront of you. Taking ownership of the whole range of services and so on, can lead to this. **
The Mindset in Motion
- How does a system work? ā How will the system work if we scale it?
- How does a system work? ā How can the system work with less operational load
- SREs are angered by toil and the almost allergic reaction it provokes in them.
- How does a system work? ā How can the system work reliable for more people
Culture
Support an enviorment where SREs can thrive.
- Celebrate the elimination of toil, give SREs opportunity to look for toil and come up with ideas on how to remove it.
- Support curosity
Other good way is to support the creation of documentation āit is not finished until it is documentedā
New hires
For new hires good some good tasks are:
- Find something unclear, missing or in some way needing improvment on the docs
and fix it.
- This has the benefit that they need to interact with the docs and read it, at a deep enough level that they can improve it.
- Taking our inline database of all systems on our network, and make sure it was correct by visiting the locations
Avoid
How to avoid the incident-handling train of the SRE will always fix everything.
A good question to anwser is āWho is getting smarter and what are we doing about it?ā The ideal would be that you are getting good new info about your systems and how they fail.
But if just the SREs are the ones that are learning, you are in reverse, you are going the opposite direction of culture you are hoping to create.
If the answer is close to: SREs, engineering personnel and relevant stake holders, you can go to the next question, What is the rest of the org doing with this knowledge?.
Cool Ideas to improve culture
- Start a postmortem group,
- Where someone does a writeup of an issue/outage that happened. Give it in
advance before the meeting, and discuss what could have been done diff,
or proposal on the arch, stuff like that.
- Good first questions are:
- What is missing from the write-up we have here?
- What do we still not know, or what may we never know?
- Why wasnāt this worse?
- Good first questions are:
- Where someone does a writeup of an issue/outage that happened. Give it in
advance before the meeting, and discuss what could have been done diff,
or proposal on the arch, stuff like that.
SRE Advocacy
There are two times where SRE Advocacy is most imporant
- early stages: you need to be able to tell people why they need an SRE
- expansion phase: āCool, youāve been able to set up a new SRE group. Now you have to get others to play with you. How do are you going to do that?ā
Humans are weird to be story-receiving machines. Letās go back to the definition
Site reliability engineering is the discipline devoted to helping orgs sustainably achieve an appropriate level of reliability in their systems, services and products.
You can tell stories aboutā¦
- Efficacy
- Partner suffering with reliability issues, SRE got involved and helped with X, Y and Z and now they are in a better place.
- Reputation
- How famous company X adopted SRE
- Possibility
- How comparable company X adopted SRE, how it went, issues, how they overcame. If they can you can do it to
- Surprise
- Story about an outage and the surprising result or finding uncovered by SRE postmortem process
- Transformation
- how things use to be, but now we are in a better place
- Puzzle
- X was a situations that made no sense; hereās how we solved the mystery step by step
Keep notes on things that happen, the brain is bad at memory stuff.
Add cliffhangers to your stories, āall pointed to human error, but something did not feel right..ā
āHere is how I failed and leveled up based on that experienceā
Becoming SRE for the Individual
Preparing to be an SRE
This is not like super mandatory, but will help you a lot with being an SRE:
Do I need to know how to code? Yes
- If you do not know how something is built your ability to understand it decreases
- Learning how to code will teach you how to debug/troubleshoot
- Many tools come in data formats that developers use in their day to day (JSON, YAML)
Fundamentals
Is always good to know about the Big O notation, this will let know people you know about data structures and efficiency.
- Single/Basic Systems
- You need to know how an OS works, about networking, permissions, protocols and how they fail.
- Distributed Systems
- Almost everything now is a distributed system
- Statistics and Data Visualization
- You need to understand and talk easily about, percentiles, standard statistical operations, aggregate and compound.
- The ability to improve reliability in most situations is predicated on how to have conversations about data
- Storytelling
- Post incidents reviews and post-mortems are literally tell a story
- Be a good person
Other nice to have
- Non-Abstract Large System Design (NALSD)
- Process of designing and reasoning about large systems
- Check highscalabilty.com and chapter 12 of The Site Reliability Workbook oreilly book/)
- Resilience Engineering
- Chaos Engineering and Performance Engineering
- Machine Learning and Artificial Intelligence
Getting to SRE fromā¦.
From Dev/SWE
You need to shift your focus to these areas:
- How does your code really function when in production?
- How will it behave when the internet can throw any input at it.
- What happens if the connection to the DB is slower than in your dev env.
- Failure nodes
- Do you have a good sense on how your code fails?
- How can you tell when it is about to fail?
- What have you built into the code to make figuring out whether your code is running well easy or possible?
- How easy have you made handle upgrades?
- and rollbacks?
- Have you written good docs on for an SRE audience?
Basically: How much do you think about running your systems in additions to building them?
From Sysadmin
Sysadmins live to serve, they bridge the gap between technology and the people that use it.
You already have a good well-exercised skill for troubleshooting and debugging.
Do exercises in SadServers, like leet code but for infrastructure.
Change you mindset from āmonitor all the thingsā to āmeasure reliability from the customer perspective, not the component perspectiveā
Using terms like ācontributing factorsā instead of āroot causeā can change both your internal and external way of looking at an issue. More on this on chapter 10.
Since you are already answering tickets, getting emails on things failing, you can start using that as a data set to start measure the reliability of the systems and how/why they fail.
More advice
Do not forget to track your progress, so you have something to look at when they reject you from an interview, or when you feel indifference to your ideas.
Hints for Getting Hired as an SRE
Some general advice of getting hired as an SRE
Looking closely at the job posting
Few thing to check:
- The tech mentioned
- Modernity of the tech
- How items hand together
- k8s/prometheus makes sense, k8s/nagios does not
- Mention of Ticketing System
- How quickly will things move in that env if its ticket based.
- Specific versions of sw
- they need a very specific thing
- Mix of on-premise and cloud products
- Are they all in the same env?
- Mention of programming languages
- Coding has meaning to them
- Heavily skewed toward CI/CD and env provisioning tools
- May have been a devops position, which is a diff mindset
- Presence or absence of a monitoring tech
- What connection if any would monitoring have to this role?
- Services that are consumed
- What am i getting into from dependencies perspective
- The human connection
- Look for an indication of the stakeholders or collaborators
Look for post-incidents reviews, they can be a helpful way of how they handle issues, whatās their SW stack and stuff like that. Do not bring that up in the interview unless they do.
Preparing for an SRE interview (Resources)
Depends (of course) of the posting, it could be more SWE focus, or CI/CD focus, but there are four things you need to study for the interviews:
-
NALSD (non-abstract large system design)
For systems that require scale (most of them)
Resources:
- The Site Reliability WorkbookĀ has a lovely chapter on the topic.
- There are a lot of talks on the topic USENIX
- Google has a public classroom for this https://sre.google/classroom/
-
Monitoring/observability
Good places to start:
- Practical MonitoringĀ by Mike Julian (OāReilly, 2017)
- Observability EngineeringĀ books by Charity Majors et al. (all from OāReilly)
If you expect to talk about SLIs and SLOs review
- Implementing Service Level Objectives(OāReilly, 2020).
-
Computing Fundamentals
- Computer science, computer networks, linux, distributed computing, stuff like that.
-
Troubleshooting and debugging
- Hopefully you have experience with this one, but sadservers.com is a good start
What to ask at the SRE Interview
Some good conversation starters:
-
Tell me about your monitoring system
This exposes all sort of info on organization, structure, collaboration, ownership, How decision are made, and so on.
Some follow up questions:
- Who owns monitoring in your org?
- How many monitoring systems are there in active use?
- Who (apps/services/teams) send data to those systems, and who access them?
- How easy it is to onboard a new app/service to your monitoring?
- What decisions are made using the data?
- Are there alerts generated fromt his system?
- What makes you happy and unhappy with your current system?
-
Tell me about your post-incidents review process
Here you are trying to see how intentional are they on learning from failure
- Do you hold post-incidents review after your outages?
- What is their purpose?
- Who is āin the roomā for them?
- How do you document your outages?
- Can you tell me (at whatever level of details you are comfortable) about a recent outage
- Do you have a sense of the most common classes of outage you have seen in the last N month? (config related, overload/resource-deprivation failures, code bug)
-
Tell me about you on-call setup
- Do people get time off after incidents?
- Who in the org participates in an on-call rotation (just SREs? Devs? Managers?)
- When was the last time you personally were āpagedā?
- Do people get paged equally often between work and off-work hours?
-
What problem does SRE exist to address in your org?
- If they cannot answer that: What are some ārecent winsā by SRE in the past 6 to 12 months?
-
Can SREs check in code to major repos in your org?
- Youāll see how involved is the SRE with dev work
A Day in the Life of an SRE
Because of the nature of the SRE role, it is hard to describe an average day, since it most days are different to each other. So instead of an average day, we have different modes SREs can relate to. Look at it like different hats an SRE will wear
-
Incident/Outage Mode
There will be days were most of your time you will be involved dealing with an accident. These days come with some feelings attached to them (fear, anxiety, and so on), the intensity of these feelings depend on the severity of the outage.
When on this mode you will be reacting not planning it is normal.
-
Post-incident Learning Mode
After the incident now you have the opportunity to review the memories of an outage and learn from it.
You are responsible for documenting it, in a way others can understand it. To do this you will have to investigate a bit, look for data in your monitoring systems, talk to your colleagues to discover what they knew and when.
-
Builder/Project/Learn Mode
This is when you actually have time to sit and...
- Devlop some code for services or SRE tasks
- Provision new envs or infra
- Improve monitoring/observability
- Removing toil
- Writing docs
- Learning a new tech in anticipation of using it some day
Of course, there will be times when you are doing boring stuff, but that can help you identify toil you need to remove.
-
Architecture Mode
Depends on your org, but, and SRE should be showing up to design and planning meetings where they are acting as a representative for reliability.
Be political about it, no one wants to hear, "this would have never happen if there was an SRE when this thing was designed". Appeal to the sense that everyone want to have their code in production be as reliable as possible.
-
Management Mode
If the response to, what did you do all day?, was, I went to meetings, do not worry, chances are you might be an SRE manager.
-
Planning Mode
Some portion of your day will be planning.
- Implementation plans
- Capacity planning
- Self-definitional work (goals of SRE team and stuff like that)
-
Collaboration Mode
The SRE role is relentlessly collaborative.
When you implement SLI/SLO (Service Level Indicators/Service Level Objectives) you will be working with: devs, PMs, stakeholders.
Another example of collaborative work, is what some people call pre-launch review. An SRE gets involved to revise the service being deployed in production, what is necessary for it to run reliably in production
Do not be a gatekeeper, share this work with the devs and the stakeholders, collaborate as much as possible
Finally, listen to the customers through monitoring work. The SLIs/SLOs (Service Level Indicators, Service Level Objectives) are meant to provide ongoing collaboration with the customer.
-
Recovery and Self-Care Mode
Burn out SREs are of no good to anyone. Because of the nature of SREs it can be easy to overextending yourself. But when we hear that someone is regularly working 60-75 hours, is not something to be proud of, that means there is a failure in the system and needs to be fixed.
You need to have recovery time.
-
On Balance
Balance is something good to strive for, but there are often situational factors that complicate the effort, for example, an early service vs mature service, new services are always noisier and require more reactive work. They also provide more toil to be stripped away, so maybe you expend more time in one of these modes than in others.
The idea is to see this as weather patterns, I know it is going to rain hard for some time, but I mentally prepare. Ideally things will level out. If not, you need to strive for it. SRE attempts to be a sustainable operations practice. If you realise this cannot be sustained maybe you need to start looking for a different job.
Establishing a Relationship to Toil
If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes are your systems grow.
Carla Geisser, Google SRE
What is toil?
First of all we need to define toil. Toil is not the work you do not like to do or simply repetitive chores. Toil is the kind of work when running a production service, that to tends to be:
-
Manual Work like manually running a script that automates a task is still manual, the time that a human spends running the script is still toil time
-
Repetitive Toil is work you do over and over.
-
Automatable If human judgment is essential for the task, there is a good chance it is not toil.
-
Tactical Toil is reactive rather than strategy-driven.
-
Does not bring enduring value If your service is in the same state after you finished a task, the task was probably toil.
-
O(n) with service growth If the work involved scales linearly with the size of the service (traffic volume, service size) is it probably toil.
We need to have less toil because it tends to expand, if left unchecked it can quickly fill 100% of everyone's time, and people will stop doing Engineering Work. What is that? Engineering work is novel and requires human judgment. It produces permanent improvement in your service and is guided by strategy.
Now that we know what toil is, lets see the relationship SREs have to it.
Whose Toil Are We Talking About?
Whose toil is it? On opposed to other parts of SRE where we are customer focuses, here we need to focus on our toil, not the customers one. Sure, they might have a connection for example, operational toil (ours) is exposed to the customer if they have to go into 4 steps to make a request. But we need to keep the main focus on the operational toil.
Why do SREs Care about toil?
An argument can be made that if you remove toil a system becomes more reliable, but the author suggests that sometimes this is not the case. An that SREs because of their nature, are inclined to eliminate toil because of the following reasons.
-
Aesthetics SREs want to eliminate toil because if offends their aesthetic sensibilities. Toil is inelegant, inefficient, unnecessary, suboptimal, hard to look at. That simply is a reason to remove it.
-
Money Orgs have many reasons to want their expensive people do work that is significant that make the revenue forward, meaning the antithesis of toil.
-
Job Satisfaction
Early vs Established Toil
When an app is developed it is likely to have more toil than once it is established. Why? Developers care about making a solution to a customer problem, not making their app be super operational. They can be, and that is why it is important that an SRE is in the room when planning the Architecture for the app, but it is likely that it will have more toil than an established one.
It is important to note this, because now we can mentally prepare for it, we now know that there will be a finite period of work with a lot of toil, but it is expected and it will end.
Dealing with Toil
Usually people just say, automate it, and the toil will go away. But the author propose the idea that similar to matter that toil is not created and cannot be destroyed just transformed.
When you are automating a task, the toil did not disappeared it just got transformed into a different thing: Complexity. Usually this is a wise bargain to take, but it is important to keep in mind that it has its trade offs.
Intermediate to Advanced Toil Reduction
It is important to keep track of the toil we remove on individual systems, management will love to hear that X system required N manual steps and now it was automated and require N-4 steps. But once you pass that first stage you also need to start thinking. How can I reduce the toil we are going to have?
Go Remove the Toil
That is pretty much it for this chapter, we defined what toil is and how to deal with it. Go and put in practice what you have read here.
āµļø k8s
Super surprised someone still writes the full Kuberenetes word instead of k8s. K8s looks way cooler
Specific k8s stuff
Things surrounding k8s
- ā helm (it has a really good intro on k8s)
š k8s up and running
ā ļø These are notes taken from the book Kubernetes Up and Running by Brendan Burns, Joe Beda, and Kelsey Hightower O'Reilly. Please go read it and just use this as reference.
Introduction
It has become the standard API for building cloud native applications. It provides the software needed for build and deploy, reliable, scalable distributed systems.
Distributed systems: Most services nowadays are delivered via the network, they have different moving parts in different machines talking to each other over the network via APIs.
Reliable: Since we rely a lot on these distributed systems, they cannot fail, even if some part of them crashes, they need to be kept up and running, and be fault-tolerant
Availability: They must still be available during maintenance, updates, software rollouts and so on.
Scalable: We live in a highly connected society, so these systems need to be able to grow their capacity without radical redesign of the distributed systems.
Most of the reasons people come to use containers and container orchestration like k8s, can be boil down to 5 things: velocity, scaling, abstracting your infra, efficiency, cloud native ecosystem
Development Velocity
Today software is delivered via de network more quickly than ever, but velocity is not just a matter of raw speed, people are more interested in a highly reliable service, all users expect constant uptime, even as the software is being updated.
Velocity is measures in term of the number of things you can ship while maintaining a highly available service.
k8s give you the tools to to do this, the core concepts to enable this are:
-
Immutability
Software used to be mutable (imperative), you installed
vim
usingdnf
and then one day you would dodnf update
and it would add modifications on top of the binary that you already have forvim
. This is what we call mutable software.On the other hand, immutable software, you would not add on top of the previous release, you would build the image from zero. Which is how containers work. Basically when you want to update something in a container, you do not go inside it and do
dnf update vim
you create a new image with the version of vim you want, destroy the old one, and create a container with the new one.This makes rollbacks way easier, and it keeps tracks of what changed from version to version.
-
Declarative Configuration
This is an extension of immutability but for the configuration. It basically means that you have a file where you defined the ideal state of your system. The idea of storing this files in source control is often called āinfrastructure as codeā.
The combination of declarative state stored in a version control system and the ability of k8s to make it match the state makes rollback super easy.
-
Self-Healing Systems
K8s will continuously take action to make sure the state on the declarative configuration is met. Say it has declared 3 replicas, then if you go and create one extra, it will kill it, and if you were to destroy one, it would create it.
This means that k8s will not only init your system but will guard it against any failures that might destabilize it.
Scaling Your Service and Your Teams
K8s is great for scaling because of its decoupled architecture.
Decoupling
In a decoupled architecture each components is separated from each other by defined APIs and service loads
This basically means that we can isolate each service.
- The API provide a buffer between implementer and consumer.
- Load balancing provide a buffer between running instances of each service.
When we decouple the services, it makes it easier to scale, since each small team can focus on a single microservice
Scaling for Applications and Clusters
Since your app is deployed in containers, which are immutable, and th configuration o is also declarative, scaling simply becomes a matter of changing a number in a configuration file. Or you can even tell k8s to do that for you.
Of course k8s will be limited to the resources you have, it will not create a physical computer on demand. But k8s makes it easier to scale the cluster itself as well.
Since a set of machines in a cluster are identical and the apps are decoupled from the specifics of the machine by containers, scaling a cluster is just a matter of provisioning the system and joining it to a cluster.
K8s can also come in handy when trying to forecast costs on growth and scaling. Basically, if you have different apps that need to scale from many teams, you can aggregate them all in a single k8s, and try to forecast the usage of the 3 apps together.
Scaling Development Teams
Teams are ideal when using the two-pizza team when (6 to 8 people), larger teams tend to have issues of hierarchy, poor visibility, infighting and so on.
K8s can help with that, since you can have multiple small teams working on different microservices and aggregate them all into a single k8s cluster.
K8s has different abstractions that make it easier to build decoupled microservices.
- pods: are groups of containers, can contain images developed by different teams into a single deployable unit.
- k8s services: provide load balancing, naming and discovery to isolate the microservices.
- namespaces: provide isolation and access control, so that each microservice can control the degree to which other services interact with it.
- ingress: easy to use front end that can combine microservices into a single API.
Separation of Concerns for Consistency and Scaling
K8s leads to greater consistency for the lower levels of the infrastructure. Basically the devs need to only worry about their app meeting the SLA (service level agreements) while the orchestration admin only need to worry about reaching the SLA from their side.
Abstracting You Infrastructure
When you build your app in terms of containers and deploy it via k8s APIs, you make moving your app between environments super easy. It is simply a question of sending the declarative config to a new cluster.
K8s make building deploying and managing your app truly portable across a wide variety of environments
Efficiency
Efficiency is measured by the ratio of the useful work performed by a machine.
K8s enables efficiency in multiple ways. Since developers do not need to use a whole machine for himself/herself, but rather one container, multiple users can share the same baremetal. Meaning less idle CPUs.
Cloud Native Ecosystem
K8s is open source and has a huge community building a thousand things on top of it. You can check the maturity of different projects in Cloud Native Computing Foundation. The maturities are sandbox incubating and graduated, going from less mature to most mature.
Summary
Fundamentally k8s was design to give developers more velocity efficiency and agility. Now that we have seen why to use it, lets see how to use it.
Deploying a k8s cluster
You can use a cloud provider, deploying on bare-metal quite hard, other good alternatives.
http://kind.sigs.k8s.io that has a really cool logo, and works on using containers for the nodes.
Go check the pages of cloud providers on how to deploy it there since it will change depending on the provider.
One thing to bare in mind is that in order to use kubectl
with your cluster
you need a ~/.kube/config
file, probably your cloud provider will give that
to you.
Also you can go to k8s cheatsheet (describe cluster) to see the basics commands on describing a cluster.
I will just cover some of the commands here, like
k describe nodes <node name>
At the top you can see basic node info:
Name: kind-control-plane
Roles: control-plane
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
We can see that it is linux on ARM
Next we can see info about the operation of the node
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False 29 Apr 2024 29 Apr 2024 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False 29 Apr 2024 29 Apr 2024 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False 29 Apr 2024 29 Apr 2024 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True 29 Apr 2024 29 Apr 2024 KubeletReady kubelet is posting ready status
Here we can see that the node is Ready, that it has sufficient memory, it has no disk pressure, and that is has sufficient PID.
Then we see info about the capacity of the system.
Capacity:
cpu: 1
memory: 1933908Ki
pods: 110
Allocatable:
cpu: 1
memory: 1933908Ki
pods: 110
Then info about software in the node like OS image, docker version all that.
System Info:
# more stuff
Kernel Version: 6.5.6-200.fc38.aarch64
OS Image: Debian GNU/Linux 12 (bookworm)
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.7.13
Kubelet Version: v1.29.2
Kube-Proxy Version: v1.29.2
Then info about the pods running on the node:
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-76f75df574-92286 100m (10%) 0 (0%) 70Mi (3%) 170Mi (9%) 23s
kube-system coredns-76f75df574-m2z2k 100m (10%) 0 (0%) 70Mi (3%) 170Mi (9%) 23s
kube-system etcd-kind-control-plane 100m (10%) 0 (0%) 100Mi (5%) 0 (0%) 39s
kube-system kindnet-qm29d 100m (10%) 100m (10%) 50Mi (2%) 50Mi (2%) 23s
kube-system kube-apiserver-kind-control-plane 250m (25%) 0 (0%) 0 (0%) 0 (0%) 39s
# more stuff
We can see that k8s tracks both requests
and upper limits
for resource for
each Pod. The difference is that resources requested by a pod are guaranteed to
be present in the node, while pod's limit is the max amount of given resource
that pod can consume
Cluster Components
Many components that make up the k8s cluster are actually deployed using k8s itself. All of them run in the kube-system namespace
-
k8s proxy
It routes network traffic to load-balanced services in the k8s cluster. It must be present on every node in the cluster
% k get daemonSets --namespace=kube-system kube-proxy
-
k8s dns
provides naming and discover for the services that are defined in the cluster
% k get deployments --namespace=kube-system coredns
Common kubectl
commands
I will repeat the commands here and in the page: k8s cheatsheet but I will go a bit more into detail here. I will also
alias k=kubectl
kubectl
is the way you talk with k8s API. Here we will go over basic
kubectl
commands.
Namespaces
Namespaces are like folders that hold a set of objects. They are used to organize the cluster.
By default the kubectl
command interacts with the default
namespace, but
you can specify which you want to use by -n
or --namespace=<ns>
.
You can also use --all-namespaces
flag to refer to all the ns.
Contexts
If you want to change the namespaces in a more permanent way, you can use
contexts. These get recorded in your ~/.kube/config
file. (That file also
stores how to find and auth to your cluster)
You can create a with a different default namespaces using
% kubectl config set-context my-context --namespace=mystuff
We need to tell kubectl
to start using it.
% kubectl config use-context my-context
You can also use context to manage different clusters users for auth using
--users
or --cluster
flags with set-context
read the man for more info
Viewing k8s API Objects
Basically k8s is an API and kubectl is just an http client. To see the k8s
objects (everything represented by a RESTful resource) we use k get <resource name>
By default it will throw to the STDOUT human readable stuff, but you can also
use -o json
or -o yaml
in case you want it in specific format.
Also when using awk
you may want to use --no-haders
.
You can also get multiple objects
k get pods,services
To get more info on a particular object
k describe <resource name> <obj name>
If you want to see like a mini man page of an specific k8s object:
% k explain pods
KIND: Pod
VERSION: v1
DESCRIPTION:
Pod is a collection of containers that can run on a host. This resource is
created by clients and scheduled onto hosts.
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values. More info:
Sometimes you want to continually observe the state of a k8s resource, like
when waiting for an app to restart of something you can use the --watch
flag:
k get pods -n kube-system --watch
Creating, updating and destroying k8s objects
Objects in the k8s API are represented as JSON or YAML files. We can query the server for those, or post them with an API request.
You use these JSON and YAML files to create, update or delete objects in your k8s server.
Create an object from obj.yaml
file.
% k apply -f obj.yaml
The yaml file will have the resource type of the object.
You can use the same command to update an object
% k apply -f obj.yaml
If nothing change it will not do anything, it will exit successfully, good for
for
loops. You can do --dry-run
to print objects to the terminal without
actually sending them.
To do interactive edits you can use
% k edit <resource-name> <obj-name>
# example k edit pod coredns-76f75df574-k97nj -n kube-system
It will open the yaml
in a text editor, wonce you save it will automatically
be uploaded back to the k8s cluster.
The apply
command also has a history of the previous configurations
edit-last-applied
, set-last-applied
and view-last-applied
for example:
% k apply -f myobj.yaml view-last-applied
To delete you can simply run
% k delete -f myobj.yaml
You can also delete an object without the file using
% k delete <resource-name> <obj-name>
Labeling and Annotating Objects
You can update the labels and annotations using label
and annotate
(who
would have thought).
For example to add the color=red
label to the pod named bar
you can run:
k label pods bar color=red
You can not rewrite unless using --overwrite
flag.
Finally you can remove a label by doing
k labels pods bar color-
Debugging Commands
You can check logs from a pod by:
k logs <pod-name>
If you have multiple container in your pods you can choose which container to
view using the -c
flag.
If you want to follow add -f
.
You can also use the exec
command to execute a command in a running
container.
k exec -it <pod-name> -- bash
If you do not have bash or a shell in your container, you can attach
to a
running process
k attach -it <pod-name>
You can also copy files to and from a container using cp
k cp <pod-name>:/path/to/remote/file /path/to/local/file
This will copy a file from the container to your local machine.
You can also forward traffic from your local system to the pod
k port-forward <pod name> 8080:80
Forwards traffic from the local machine on port 8080 to the remote container on port 80
If you want to see the last 10 events in a namespace
k get events
You can also stream events --watch
Finally, to check cluster resources being used
k top nodes
or
k top pods
These commands will only work if metrics servers are present, which they likely are.
Cluster Management
Cordon and drain a particular node:
- cordon: prevent future pods form being scheduled onto that machine
- drain: remove any pods that are currently running on that machine.
Useful for removing physical machine for repairs or upgrades. You would do k cordon
and then k drain
to safely remove the machine form the cluster.
Once the system is back online do k uncordon
there is no undrain
it will
naturally get back to normal.
Pods
K8s groups multiple containers into a single atomic unit called a pod (This also goes with the docker theme, since a group of whales is a pod lol)
A pod is a collection of app containers and volumes running in the same environment. Containers in a pod always land in the same machine.
Apps running in the same pod share, ip addresses and port, hostname, and can communicate to each other using interprocess communication channels. Meaning they share:
- network namespace
- UTS namespace
- IPC namespace
Thinking in Pods
What should go in a Pod? To answer this question you go look at your different containers and ask "Will these containers work correctly if they land on different machines?" If no, then a pod is the correct way of grouping them. If yes, you can use multiple pods.
An example, say you want to deploy a wordpress app with a mysql database. Would you put them in the same pod? I already read the book so I know the answer is no, but why?
The answer is no because both of those services do not need to scale together. The wordpress app itself is stateless, so you can create more wordpress pods if you are scaling. On the other hand a mysql database is much different to scale, it can be trickier, and you are likely to increase the resources dedicated to a single mysql pod.
The Pod Manifest
Pods are describe in a manifest which is a text-file representation of the k8s API object.
As you might remember from the fist chapter, k8s uses a declarative configuration, which basically means that you tell k8s your ideal state of the world and it will take action to ensure the state becomes a reality.
When you send a manifest to k8s, it will accept it and process it before
storing it in persistent storage (etcd)
. The scheduler will find pods that
have not been assigned to a node, and will place the pods onto those nodes
depending on the resources and constrains you specified in your manifest. K8s
will try to ensure pods from the same app are in different machines for
reliability.
You can see the pods with
k get pods
You can delete a pod with
k delete pods/<name>
Creating a manifest
Pod manifest should be treated in the same way a source code, adding comments to help explain the pod to new team members and stuff like that.
The manifests include a couple of key fields and attributes,
metadata
:name
: a unique identifier for the deploymentnamespace
: the Kubernetes namespace where the deployment will livelabels
: key-value pairs that can be used to filter and categorize objects
spec
:containers
:name
: a unique identifier for the containerimage
: the Docker image to use for this containercommand
: the default command to run in the containerargs
: any additional arguments to pass to the commandvolumeMounts
: mounts for persistent storage or other files
- volumes:
name
: a unique identifier for the volumepersistentVolumeClaim
: claims a Persistent Volume resource
selectors
: used to filter which pods are included in this deployment
Example
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2024-04-29T20:02:16Z"
generateName: coredns-76f75df574-
labels:
k8s-app: kube-dns
pod-template-hash: 76f75df574
name: coredns-76f75df574-k97nj
namespace: kube-system
# more stuff
uid: 4dd80223-de4c-49d7-b407-31908ae96b9e
spec:
# more stuff
containers:
- args:
- -conf
- /etc/coredns/Corefile
image: registry.k8s.io/coredns/coredns:v1.11.1
imagePullPolicy: IfNotPresent
# more stuff
volumeMounts:
- mountPath: /etc/coredns
name: config-volume
readOnly: true
# more stuff
volumes:
- configMap:
defaultMode: 420
items:
- key: Corefile
path: Corefile
name: coredns
# more stuff
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-04-29T20:02:19Z"
status: "True"
type: PodReadyToStartContainers
- lastProbeTime: null
# more stuff
You can create a pod from them by using k apply -f manifest.yaml
as seen in
previous sections.
Some Useful Commands
# get info
k get pods # list pods
k describe pods <podname> # more info on the k8s object
# manipulate pods
k delete pods/<podname> # delete pod, all info associated to it will be deleted
k logs <podname> # check logs
k exec <podname> -- date # run date command inside pod
k exec -it <podname> -- bash # interactive tty
k cp # keep in mind that copying files into a container is an antipattern.
Health Checks
When you are running an app in k8s, it is automatically kept alive by a process health check. This ensure that the app is always running, if it is not it k8s will restart it.
Sometimes, checking if the process is alive is not enough, maybe it was left hanging or something like that. K8s has something to check application liveness and other stuff for your app.
There are different kinds of probes here are some:
-
Liveness Probe This will check app specific stuff, for example checking if http is returning 200.
While the default response to a failed liveness is to restart you can change what happened with
restartPolicy
Here is an example of a
livenessProbe
spec: containers: - name: my-container image: my-image:latest livenessProbe: httpGet: path: /healthz port: 80 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5
-
Readiness Probe These probes check if an app is ready to serve user requests. Apps that fail are removed from the load balancers
-
Startup Probe Enables you to poll slowly from a slow-starting container, while also enabling responsive liveness checks once the container has initialized
Resource Management
K8s allow you to increase the overall utilization of the machines that are in the cluster.
Utilization is defined as the amount of resource actively being used divided by the amount of a resource that has been purchased
Example, if you purchase a one-core machine, and your app uses one tenth of a core, then your utilization is 10%
You can tell k8s the upper limit and lower limit of resources you want for an specific container. Keep in mind that this is on a per-container basis, the whole pod will the sum of the containers.
You can do this by setting up resources
in the manifest. The lower limit is
requests
and the upper limit is limits
.
spec:
containers:
- image: blabla:latest
name: jose
resouces:
cpu: "500m"
memory: "128Mi"
limit:
cpu: "1000m"
memory: "256Mi"
# more stuff
Persisting Data with Volumes
When a container restarts, any data in it will also be deleted. Which is good since we do not want to leave around cruft.
We can still add volumes to our pods though, that will persist this deletion
In order to do this, you need to two things to the manifest spec.volumes
,
which is an array of all volumes that the pod will need.
And the other is volumeMount
which specify inside the container definition
the file system to use.
So basically one is like, what volumes will the whole pod use, and the other is specific to each container.
spec:
containers:
- name: my-container
image: busybox:latest
volumeMounts:
- name: my-filesystem
mountPath: /mnt
subPath: data
volumes:
- name: my-filesystem
persistentVolumeClaim:
claimName: my-pvc
Labels and Annotations
K8s apps can grow in size and complexity real quick. We can use labels and annotations to help organize our k8s objects/resources as sets of things that map how we think about the app. Making it a bit more human accessible. Meaning we can group resources that make the most sense of the application.
We have two tools that help with this.
- Labels: key/value pairs, to attach identifying information to the k8s obj/resource. You will be able to query based on this.
- Annotations: set non-identifying information that libraries can leverage. These are not meant for querying
Labels
Labels give identifying metadata for k8s objects. They are key/value pairs of strings.
Keys are broken down into two parts
- Optional prefix name, separated by a slash.
- Key name, it must start and end with alphanumeric chars.
Some examples:
acme.com/app-version
-1.0.0
app.version
-1.0.0
Applying Labels
You can do it when doing the k run
command. (Doubt that is the way you are
using it but anyway):
k run alpaca-test \
--image=gcr.io/kuar-demo/kuard-amd64:green \
--labels="ver=1,app=alpaca,env=test"
You can also add it to the manifest of the resource you are using:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
labels:
ver: "1"
env: "test"
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
After the objects have been created, you can do --show-labels
k get deployments --show-labels
Modifying Labels
You can update the labels on objects after creating them:
k label deployemnts alpaca-test "canary=true"
Keep in mind that this will only update the label on the deployment itself, it wont affect any objects created by the deployment.
In case you feel lazy you can replace --show-labels
with -L
.
k get deployemnts -L canary
To remove a label you add a dash-suffix
k label deployments alpaca-test "canary-"
Label Selectors
Okay so, how do we query this stuff? Using boolean expressions
This will list only the pods that have ver label set to 2.
k get pods --selector="ver=2"
You can get a bit more creative:
This is the AND operator:
k get pods --selector="app=bandicoot,ver=2"
You can kind of do the OR by:
k get pods --selector="app in (alpaca,bandicoot)"
You can also check if a label has been set by just using the key name:
k get deployments --selector="canary"
Check if key is not in value1
or value2
:
k get pods --selector="key notin (value1, value2)"
You can combine them
k get pods --selector="ver=2,!canary"
Labels Selector in Manifests
Newer versions support doing something like this:
selector:
matchLabels:
app: alpaca
matchExpressions:
- {key: ver, operator: In, values: [1, 2]}
Annotations
This are only meant to assist tools and libraries, do not use them for queries.
If you are wondering if something should be a label or an annotations, add information to an object as an annotation and promote it to a label if you find yourself wanting to use it in a selector.
Usually annotations primary use is rolling deployments, they are used to track rollout status and provide the information for a roll back if needed.
Annotations are defined in the metadata
part of the manifest:
...
metadata:
annotations:
example.com/icon-url: "https://example.com/icon.png"
...
Service Discovery
K8s is a dynamic system. It has to manage placing pods on nodes, make sure they are running and more stuff. Due to this nature k8s makes it easy to run a lot of things, but it can be a bit hard to find these things.
What is Service Discovery?
A service is a method for exposing a network application that is running in one or more Pods in your cluster
Actually you use one type of service discover everyday, DNS, the thing that helps you resolve hostnames to IPs. It works great for the internet, but falls short in the k8s world.
Why? DNS caches a bunch of stuff, and in a k8s cluster pods are ephemeral, they can be killed and come back up in seconds, so due to this cache we could be pointing to something that is already dead.
While these pods that compose the backend may change, the frontend clients should not need to be aware of that, nor should they need to keep track of the of the backend themselves.
K8s has an object/resource specific to this issue.
The Service Object
A Service is an object, it has a manifest and all that. Say you have a set of
pods that each listen on port 5000 and are labeled as name=flask-app
. The
manifest would look something like this:
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app.kubernetes.io/name: flask-app
ports:
- protocol: TCP
port: 80
targetPort: 5000
K8s will assign the service a Cluster IP , and will be continuously scanning
for Pods that match the selector (the flask-app
part). Then it will make the
updates to the set of EndpointSlices
for the Service.
Using the type NodePorts
By using this type, in addition of a Cluster IP, the system picks a port on every node in the cluster to forward traffic to the service.
Meaning you can connect to any node in the cluster, and reach an specific
service. You can use NodePort
without knowing where any of the pods for that
service are running.
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: NodePort <--- here
selector:
app.kubernetes.io/name: MyApp
ports:
- port: 80
# By default and for convenience, the `targetPort` is set to
# the same value as the `port` field.
targetPort: 80
# Optional field
# By default and for convenience, the Kubernetes control plane
# will allocate a port from a range (default: 30000-32767)
nodePort: 30007
Now each request that is send to the service will be randomly directed to one of the pods that implements the service.
ConfigMaps and Secrets
A good practice is to make your container images reusable, making a general purpose one that can be used across applications. But we might also need to add some configuration to the image at runtime. Here is where ConfigMaps and Secrets come into play.
ConfigMaps
In a ConfigMap you can define env variables, command line arguments or even
small file systems for your containers. The configmap
is combined with the Pod
right before it is run.
Creating ConfigMaps
We can create a configmap
out of a file. Say for example we have a file
called config.txt
:
parameter1 = value1
parameter2 = value2
You can create a configmap
by doing:
% k create configmap config --from-file=config.txt
Or you can create literally from the command line
% k create configmap other-config --from-literal=some-param=some-value
Now you can list them
% k get cm
NAME DATA AGE
config 1 35s
some-config 1 2s
And get them as yaml
s
% k get cm config -o yaml
apiVersion: v1
data:
config.txt: |
param1 = value1
param2 = value2
kind: ConfigMap
metadata:
creationTimestamp: "2024-06-11T00:58:10Z"
name: config
namespace: default
resourceVersion: "2420"
uid: 7df23db6-7d2c-4db4-9b22-5fecd4bb456e
At the end of the day, the configmaps
are just some key/value pairs stores in
an object.
Using a ConfigMap
There are 3 ways of using them:
- File system:
- A ConfigMap can be mounted in a Pod. A file is created for each key, and the content is their respective values
- Command-line Argument:
- Creating command lines dynamically
- Environment Variable:
- It can set the value of an environment variable
Let's see each as an example
- File System Volumes
apiVersion: v1
kind: Pod
metadata
name: kuard-config
spec:
containers:
- name: test-container
image: gcr.io/kuar-demo/kaurd-amd64:blue
imagePullPolicy: Always
command:
- "/kuard"
volumeMounts:
# Mounting the ConfigMap as a set of files
- name: config-volume # <- HERE
mountPath: /config
volumes:
- name: config-volume # <- HERE
configMap:
name: config # <- name of the cm we just created
restartPolicy: Never
- Command-line Arguments
apiVersion: v1
kind: Pod
metadata
name: kuard-config
spec:
containers:
- name: test-container
image: gcr.io/kuar-demo/kaurd-amd64:blue
imagePullPolicy: Always
command:
- "/kuard"
- "$(EXTRA_PARAM)" <- NOTE HERE
env:
- name: EXTRA_PARAM
valueFrom:
configMapKeyRef
name: config # <- name of the cm we just created
key: param1
Note that k8s will perform the correct substitution with a special
$(<env-var-name> )
syntax
- Environment Variable:
apiVersion: v1
kind: Pod
metadata
name: kuard-config
spec:
containers:
- name: test-container
image: gcr.io/kuar-demo/kaurd-amd64:blue
imagePullPolicy: Always
command:
- "/kuard"
env:
- name: ENV_PARAM
valueFrom:
configMapKeyRef
name: config # <- name of the cm we just created
key: param2
Secrets
This are pretty similar to ConfigMaps but they are intended for sensitive information, for example, passwords, tokens, private keys, TLS certs.
This secrets are exposed to Pods via explicit declaration in the Pod manifest and the k8s API.
By default k8s stores secrets in plain text on
etcd
storage. If you have super sensitive information there, and a lot of people with admin access, might be worth encrypting it a bit more.
Creating Secrets
We can create a secret like any other resource
% k create secret generic my-secret --from-file=some.key
Instead of generic
-- depending on the secret -- we can use different types
From the manpage
Available Commands:
docker-registry Create a secret for use with a Docker registry
generic Create a secret from a local file, directory, or literal value
tls Create a TLS secret
If we describe it
% k describe secret my-secret
Name: my-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
some.key: 1984 bytes
The Pods consume the secrets.
Consuming Secrets
We can call the API directly, but more often we will access secrets from a
Secrets Volume. These volumes are created at pod creation time, and are
stored in tmpfs
(meaning memory not disk).
Each secret is in a different file under the target mount point. So if we mount
the my-secret
secret in the /keys
directory, it would looks something like
/keys/some.key
.
The Pod manifest would end up looking something like this:
apiVersion: v1
kind: Pod
metadata
name: kuard-secret
spec:
containers:
- name: test-container
image: gcr.io/kuar-demo/kaurd-amd64:blue
imagePullPolicy: Always
volumeMounts:
- name: keys
mountPath: /keys
readOnly: true
volumes:
- name: keys
secret:
secretName: my-secret # <- name of the secret we just created
Private Container Registries
You can also create a secret for for use with a Docker registry. Instead of
generic
you would use docker-registry
, with --docker-username
,
--docker-password
and --docker-email
and the manifest would look something
like this:
apiVersion: v1
kind: Pod
metadata
name: kuard-secret
spec:
containers:
- name: test-container
image: gcr.io/kuar-demo/kaurd-amd64:blue
imagePullPolicy: Always
volumeMounts:
- name: keys
mountPath: /keys
readOnly: true
imagePullSecrets:
- name: my-image-pull-secrets-from-registry # <- name of that secret
volumes:
- name: keys
secret:
secretName: my-secret
Naming Constrains
Key names need to be valid variable names. Just use normal names and you will be fine.
-
valid
.token
secret.token
config_file
-
not valid
token..key
auth token.key
_token
Just not use spaces, double dots and stuff like that.
Managing ConfigMaps and Secrets
The usual stuff, get
, delete
, create
. Just a note on creating:
--from-file=<filename>
: keys in file will be keys in secret--from-file=<key>=<filename>
: keys in file will be keys in secret- Instead of:
You would getData ==== <filename>: ---- param1 = value1 param2 = value2
Data ==== <key>: ---- param1 = value1 param2 = value2
- Instead of:
--from-file=<directory>
: loads all files in a directory
Also you can recreate and update like:
kubectl create secret generic kuard-tls \
--from-file=kuard.crt --from-file=kuard.key \
--dry-run -o yaml | kubectl replace -f -
Neat trick from the book Kubernetes Up and Running by Brendan Burns, Joe Beda, and Kelsey Hightower O'Reilly.
š k8s cheatsheet
Some useful commands
using kind?
in you mac? with podman?
podman machine start; export KIND_EXPERIMENTAL_PROVIDER=podman; kind create cluster
Describe Cluster
-
Check version
% k version # what do you think it does? :) *Client Version: v1.28.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.2*
-
Check cluster status
% k get componentstatuses # will be deprectated Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy ok
or talk directly to the endpoint
% k get --raw='/readyz?verbose' [+]ping ok [+]log ok [+]etcd ok [+]etcd-readiness ok [+]informer-sync ok # more messages readyz check passed
-
List k8s nodes
% k get nodes NAME STATUS ROLES AGE VERSION kind-control-plane Ready control-plane 13s v1.29.2 node01 Ready <none> 13s v1.29.2
In k8s nodes are separated into
control-panel
nodes, like the API server, scheduler, etc. which manage the cluster, andworker
nodes where your containers will run. -
Get more information about a specific node (prints bunch of info)
% k describe nodes kind-control-plane
more info on the ouput here
-
See the proxies if they are under the API object named DaemonSet
% k get daemonSets --namespace=kube-system kube-proxy
-
See DNS k8s deployment
% k get deployments --namespace=kube-system coredns
-
Wait for nodes to be ready
% k wait --for=condition=Ready nodes --all node/node1 condition met node/node2 condition met ...
Manage your context
-
Change namespace for current context
% kubectl config set-context --current --namespace=<some ns>
ā helm
ā ļø These are notes taken from the book Learning Helm by Matt Butcher, Matt Farina and Josh Dolitsky on O'Reilly Please go read it and just use this as reference.
Introduction
Helm is the package manager for k8s. Let us do a super quick dive into what k8s is an all that just to give the context necessary to understand helm.
Context for Helm (containers & k8s)
Sometime ago someone thought of a new way of making apps, instead of them being a big monolithic service, we could split everything into small discrete standalone services. Then we could join those services using API over the network and build the application that way. That is how microservices were born.
Containers
To help with this the concept of containers was introduced, it is different of a VM because the VM runs an entire OS on on a host machine. On the other hand, a container has its own file system, but it uses the same OS kernel as the host.
A container is a program together with its dependencies and environment, they are packaged together into a container image which can be build based on a file where you specify the packages you want to run, and how to set the environments. The cool part is that all those instructions are not compiled into a binary or anything, but they are packaged into discrete layers.
So if a host has an image with five layers, and there is another host that needs the same image, it will just fetch the layers it does not already have.
Say you have three images using fedora-minimal:37
, well it would reuse that
layer on all those three.
Those images are stored in a registry, the registry tells hosts which layers compose an image. So the host can only download the layers it need from the registry.
A registry identifies an image by three things:
- name: basically a string
fedora
orfedora-minimal
- tag: usually the version
latest
orv1
- digest: a hash of the image, since the tags are mutable.
They look like name:tag@digest
.
Kubernetes (or k8s like the cool kids call it)
So with all these container stuff, some questions begin to arise:
- How do we best execute lots of containers?
- How can they work together?
- How do we manage memory, cpu, network and storage?
Well a bunch of companies tried to create this orchestration of containers technology to answer those questions, at the end as per 2024 seems like Google won, and we now all use āµļø K8s.(which is greek for ship's capitan or something like that)
K8s won because it introduced two concepts that people really liked.
-
Declarative infrastructure Basically you tell k8s what your desired state of the cluster is, and it will work to make that happen.
-
Reconciliation loop How does k8s work behinds the scenes to reach the declarative configuration we set? Using a reconciliation loop.
In a reconciliation loop, the scheduler says: "Here is what the user wrote as his/her desired state. Here is the current state. They are not the same, lets reconcile them."
Say you specified storage, and the scheduler sees that we do not have storage yet, it will create units of storage and attach them to your containers.
Pods
We do not deal directly with containers when setting up our k8s cluster.
We use a pods. A pod which is basically a group of containers. These are
defined in a manifest (yaml
or json
, but most people use yaml
)
apiVersion: v1 # we can see that this will be a v1 Pod
kind: Pod
metadata:
name: example-pod
spec:
containers:
- image: "fedora-minimal:latest"
- name: example-fedora
A Pod can have 1 or more containers. The containers that help preconfigurate stuff for the main one, are called init containers.
The ones that run alongside the main container are called sidecar containers
ConfigMaps
Basically in a pod you describe the configuration the containers need, like
network ports files system mount points. You can store configuration
information for k8s in ConfigMaps
and password and stuff like that in
Secret
.
Here is an example:
apiVersion: v1
kind: ConfigMap
metadata:
name: configuration-data
data: # inside here we will declare aribitrary key/value stuff
backgroundColor: blue
title: Learning Helm
For Secret
is pretty much the same as for ConfigMaps
but the values in data
must be Base64 encoded.
Pods then can be linked to the ConfigMaps like this:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
volumes:
# note that this is not the metadata name you used in the ConfigMap
- name: my-configuration
configMap:
name: configuration-data # same name as in metadata of ConfigMap
containers:
- image: "fedora-minimal:latest"
name: example-fedora
env:
- name: TITLE # the env variable will be title
valueFrom:
configMapKeyRef:
name: configuration-data # name for the volume
key: title # key in the actual ConfigMap
Deployments
Pretty cool stuff, but we might not just one to run one instance of our
container, therefore we can use something called Deployments
.
A Deployment
describe an application as a collection of identical pods,we can
tell k8s to create our app with a single pod and scale it up to five pods.
apiVersion: apps/v1 # apps/v1 Deployment
kind: Deployment
metadata:
name: example-deployment
labels:
app: my-deployment
spec:
replicas: 3 # we want three replicas of the following template
selector:
matchLabels:
app: my-deployment
template:
metadata:
labels:
app: my-deployment
spec:
containers :
-image: "fedora-minimal"
name: "example-fedora"
Service
A Service
is a persistent network resource, that persists even if the pod or
pods attached to it go away.
apiVersion: v1
kind: Service
metadata:
name: example-service
spec:
selector:
app: my-deployment
ports:
- protocol: TCP
port: 80
targetPort: 8080
Here we define a service for the pods with the app: my-deployment
, telling
that traffic port 80 of this Service
will be routed to 8080.
Helm's Goal
So we have seen the building block of K8s, say you want to deploy a WordPress
app, you would need a Deployment
for the containers, then a ConfigMap
and
Secret
for the password, maybe some Service
objects and so on. That sounds
like a lot of yaml
.
The core idea of Helm is that all those objects can be packaged to be
installed
, updated
and deleted
together.
Basically helm is the package manger of kuberentes. Helm will allow you to spin up all the k8s objects necessary for an app by just using a few commands.
Here are some of the things that helm can do:
- Provide package repos (similar to
dnf
orapt
). - Familiar install, upgrade, delete commands.
- Helm has a method for configuring the packages before installing them.
- Helm can also let you know what already is installed.
You can also limit the installation of packages to a specific namespace, so you can install the same package in different namespaces in the same k8s cluster. (What is a namespace?)
Helm also provides reusability, it uses charts for this. A chart provides a pattern for producing the same k8s manifests. But the cool part is that you can also add more configuration to those charts.
Helm provides patterns for storing the initial configuration plus the changes
you did. Helm encourages k8s users to package their yaml
into charts so that
these descriptors can be reused.
One last thing, keep in mind that helm is not a configuration tool, it helps but it there are software specialized on that like ansible, puppet or chef.
Helm Architecture
Here are the main components that helm uses.
K8s Resource/Object
These are the Pods
, ConfigMap
, Deployment
, we have seen throughout the
chapter. K8s has a lot of these objects, you can even define custom ones using
custom resource definition (CRD).
All the resources share some elements
apiVersion: apps/v1 # api and version of the resource
kind: Deployment # resource type
metadata: # top-level data on the resource/object
name: example-deployment # req for all objects
labels: # used for creating query-able handles
some-name: some-value
annotations: # authors to attach their own keys and values
some-name: some-value
Charts
A package is called a chart. The idea es that k8s meaning captian, helm being the steering mechanism of the ship. The chart plots the way k8s apps should be installed.
A Chart is a set of files and directories that describe how to install the different k8s resource/objects
A chart contains
Chart.yaml
: describes the chart (name, description, authors)templates
directory: Inside all the k8s manifests potentially annotated with templating directivesvalues.yaml
file that provides the default configuration. You can override during installation.
These are the basic stuff for an unpacked chart, a packed chart is just a tar ball with all this inside.
Resources, Installations and Releases
When a helm chart is installed:
- helm reads the charts (will download if necessary)
- It sends the values into the templates generating the k8s manifests
- The manifests are sent to k8s
- K8s creates the requested resources inside the cluster
One last concept release. A release is created each time we use helm to modify the installation
Using Helm
Helm is a cli, you can install it with your favorite package manger or build it from source it is written in golang.
You can check the version by
helm version
Helm will use the same KUBECONFIG file you have configured for kubectl
, it
will look in the same places, though you can specify a path for one.
The most common workflow is:
- Add a chart repo
- Find a chart to install
- Install a Helm chart
- See the list of what is installed
- Upgrade your installation
- Delete your installation
Adding a Chart Repo
A Helm Chart is an individual package that can be installed into your k8s cluster. You can find then at chart repositories.
You can find popular repositories in the artifact hub
By default helm
does not have any repo added, so you need to look for one
there.
Bitnamis's official Helm charts are one of the best well-curated charts repos. (Some Bitnami devs are among the core contributors who design the helm repo system)
To add a repo you do helm repo add
so:
% helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
Now if you do
% helm repo list
NAME URL
bitnami https://charts.bitnami.com/bitnami
You will see it there. After that we can look for specific charts.
% helm search repo drupal
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/drupal 18.0.2 10.2.5 Drupal is one of the most versatile open source...
You can also search label and descriptions
% helm search repo content
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/drupal 18.0.2 10.2.5 Drupal is one of the most versatile open source...
bitnami/harbor 21.1.2 2.10.2 Harbor is an open source trusted cloud-native r...
bitnami/nginx 16.0.6 1.25.5 NGINX Open Source is a web server that can be a...
bitnami/owncloud 12.2.11 10.11.0 DEPRECATED ownCloud is an open source content c...
bitnami/wordpress 22.2.2 6.5.2 WordPress is the world's most popular blogging ...
The chart version is the version of well, the chart. On the other hand the app version is the version of the software it would install.
Installing a package
% helm install mysite bitnami/drupal
NAME: mysite
LAST DEPLOYED: Wed May 1 16:40:16 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
# more text
Typical stuff, just a few things to keep in mind.
-
Difference between an installation and a chart.
An installation of a chart is a specific instance of the chart One chart may have many installations
-
You can repeat instance names (
mysite
), but it must be on different namespaces.
You can set values specific to your installation, you can set them directly
from the command line with --set
for example. (this works with both set
install
and upgrade
)
% helm install mysite bitnami/drupal --set drupalUsername=admin
You can also have them in a yaml
file, which is the recommended approach.
% helm upgrade mysite bitnami/drupal --values values.yaml
For example values.yaml
would look like this:
drupalUsername: admin
drupalEmail: [email protected]
mariadb:
db:
name: "my-database"
Listing your Installations
% helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
mysite default 1 2024-05-01 16:40:16.50636 -0600 CST deployed drupal-18.0.2 10.2.5
Not much to say here, does what expected.
Upgrading an Installation
So there are two types of changes:
- upgrade version of the chart
- upgrade configuration of the chart
Every time we perform an upgrade we are doing a new release of the same installation.
Helm will attempt to to alter only the bare minimum, so if you only changes one simple configuration variable, it will not like restart everything and all that.
To restart stuff just use kubectl
.
To update your chart with a new version you can
% helm repo update
% helm upgrade mysite bitnami/drupal
Configuration Values
If you do this:
% helm install mysite bitnami/drupal --values values.yaml
% helm upgrade mysite bitnami/drupal
Chances are you lost your values, so it is good to always send the yaml
% helm install mysite bitnami/drupal --values values.yaml
% helm upgrade mysite bitnami/drupal --values values.yaml
You can use helm get values mysite
to see the values sent on the last install
or upgrade.
You could also use:
% helm upgrade mysite bitnami/drupal --reuse-values
But it is not recommended
Uninstalling charts
Not much to say here
% helm uninstall mysite
Works as you would expect, if you would do
% helm list
You will not see it there.
Lastly, you can see a special record that contain release information.
% k get secret
Helm stores there the info.
If you uninstall you loose the history, so be careful. You could helm uninstall --keep-history
. Good if you plan on doing helm rollback
.
That is pretty much the basics on helm.
What is Fluentd
It is a event/data collection tool. We can have multiple applications with tons of different logs formats. Fluentd tries to unify them.
It is extensible, the core program is small, and does only the main stuff.
- Divide & Conquer
- Buffering & Retries
- Error Handling
- Message Routing
- Parallelism
It delegate the rest to the users via plugins, reading data, parsing data, buffering data, write data.
It is reliable, it is important to not loose data. Fluentd tries to move the data in small pieces and checks if the data passed successfully if not it retries.
Fluentd can tag the data and route it according to the label.
Use Cases
Simple Forwarding
You have some simple files that and logs from a port and want to forward them to mongodb. Here is a simple configuration example.
logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
logs from client libraries
<source>
type forward
port 24224
</source>
store logs to ES and HDFS
<source backend.* >
type mongo
database fluent
collection test
</source>
Lambda Architecture
š Elastic Stack (ELK)
Table of contents
ELK is great for Centralized Logging, this enable us to not waste time trying to find where the issues are, we have all in one place. We also can
Kibana is the visualization of the ELK stack, you can visualize the data on real time.
Components
It has several open source add-ons, but the main ones are these:
- Beats:
- Light weight way of getting data into the ELK stack.
- Single purpose tools.
- Logstash:
- Transform data as it comes into structured data that will be stored.
- Very resource intensive
- Example: is like cutting carrots š„ as you get from the store and saving them to the fridge, instead of cutting them until you are about to cook
- Elasticsearch:
- The part of the stack that stores the data itself
- Usually deployed by a cluster, it has redundant copies so it can work if something fails
- Kibana:
- Web front end for the ELK stack
- Quick birdās eye view across your infra
Logstash
Three block of code in config
- Input: get data in
- Output put data somewhere, usually elasticsearch
- Filter block: where data is transformed and relocated, where we can parse lines.
Example of config file.
input { stdin {} }
filter {
mutate {
add_field => { "greeting" => "Hello %{message}" }
}
}
output { stdout {} }
This will take input from STDIN
and output it to STDOUT
, anything we type would be returned in a structured format with a greeting
field.
Logstash Plugins
It has more than 100 packages installed with logstash
/usr/share/logstash/bin/logstash-plugins list
for listing them
-
Beats: Input plugin to work with
beats
input { beats { port => 5044 } }
-
File:
- Get input from a file, or multiple, you can use wildcards
input { file { path=> "/var/log/kern.log" } }
-
elasticsearch
- this is an output plugin, you can point to multiple servers
output { elasticsearch { host => ["localhost:9200"] } }
Filters
-
Grok
- convert unstructured data to structured data
- based on regex, but it has many abstractions in case you want to use them
filter { grok { match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM}:request...." } } }
-
Mutate
- let you transform the data
filter { grok { match => { "message" => "%{TIMESTAMP:t} new users: %{GREEDYDATA:users}" # 2024-01-23 new users: jose,pedro,maria } } mutate { split => {"users" => ","} } }
In this example grok will get the time stamp of the message and will get all the string after
new users:
then we use mutate to split the users and have it stored as a list so when the data shows up in elasticsearch it looks like an array
GrokConstructor
grokconstructor.appspot.com Helper for creating grok filters.
You can see also some grok patterns too.
Basically you can paste the logs there build your filter from there
Beats
Why use beats? Logstash is overkill for a lot of things people use it for.
It uses JRuby (basically ruby that runs on the JVM), and by default it is configured to set aside 1GiB of memory for the JVM. That is too much to just send things to Elastic
Beats (written in Go) is way more šŖ½lightweight, it has been design to be. Becuase of this can be run everywhere.
There are several beats
- Heartbeat (Uptime)
- Filebeat (Logs and text data)
- Metricbeat (Metrics)
- Packetbeat (Network data)
- Winlogbeat (Windows events logs)
- Auditbeat (audit logs and data)
- Functionbeat (serverless data)