šŸ›¶ Introduction

I will document my journey into the SRE world here, you will find notes from books I am reading, as well as other stuff I find relevant throughout the journey.

Becoming SRE

āš ļø These are notes taken from the book Becoming SRE by David N. Blank-Edelman O'Reilly. It is an splendid book and no amount of notes I take will make it justice, please go read it and just use this as a reference.

Table of Contents

Introduction to SRE

First things first

Site Reliability Engineering is an engineering discipline devoted to helpings orgs sustainably achieve the appropriate level of reliability in their systems, services and products.

3 key words from definition

  • Reliability: even if you have the best software in the world, and the best sales team, if the sw is not up when customers need it, you lose a lot of:
    • Revenue
    • Time
    • Reputation
    • Health
    • Hiring
  • Appropriate
    • Usually 100% reliable is not achievable, it is likely that your dependencies are not 100% reliable.
    • You need to use service level indicators/service level objectives (SLI/SLO) to help determine an appropriate level of reliability.
  • Sustainable
    • If the people in the org are burned out they wont be able to build reliable systems.

SRE vs Devops

  1. SRE implements class devops

  2. SRE is to reliability as devops is to delivery

  3. Itā€™s all about the direction

    sre-vs-devops.png Source: Becoming SRE O'Reilly

  • Devops go from development to production operations
  • SRE seems to start on prod operations, and going back to improve the result.

Basically these two phrases:

  • Devops thinks, how can I deploy this from my laptop to production
  • SRE starts at production, how can I make the environment more reliable.

SRE Mindset

It is born out of curiosity: How does a system work? How does it fail?

Zoom out and Zoom in as much as possible, how does theā€¦ work

  • The whole app, development and deployment process
  • The entire service
  • The infrastructure
  • The physical overlay of the infra
  • The sociotechnical context in which the infra runs
  • The org context where the sociotechnical context resides in

Keeping Focus on the Customer

Small example, say you have a 100 servers running a front end pool, suddenly 14 systems fail and are not recovering. This situation is:

  1. No big deal; you can deal with it at your leisure
  2. Something that warrants your immediate attentions. Stop what you are doing and focus on this
  3. Crisis, even if 2 AM go wake the CEO

The answer, it depends, you need to ask, how does this affect the system as a whole. If the customers wont see anything diff then a. If visible by customers then b. If the service is dead in the water and this means no revenue, go wake the CEO.

So basically you need to ask: What is the intention of the system from the customerā€™s perspective?

SREā€™s Relationship with Failure

SRE are very collaborative. Reliability is a collaborative endeavour by nature.

The SRE mindset views errors as an opportunity to learn. Learning from failure is a core component of an SRE.

SRE treats errors as a signal and we like clear signals. With an SRE mindset errors can serve the helping understand the system better. This means we need to surface the errors in addition to eliminate them.

SRE feels ownership of the whole service, they do not say often ā€œnot my code, not my problemā€. This can be un arma de doble filo. Because of the yak shaving issue, where you go to do one task, say update a package, but then you check and for the new package you need to upgrade the OS, but to upgrade the OS you need toā€¦ and you end up with a trimmer in your hands and a yak infront of you. Taking ownership of the whole range of services and so on, can lead to this. **

The Mindset in Motion

  • How does a system work? ā†’ How will the system work if we scale it?
  • How does a system work? ā†’ How can the system work with less operational load
    • SREs are angered by toil and the almost allergic reaction it provokes in them.
  • How does a system work? ā†’ How can the system work reliable for more people

Culture

Support an enviorment where SREs can thrive.

  • Celebrate the elimination of toil, give SREs opportunity to look for toil and come up with ideas on how to remove it.
  • Support curosity

Other good way is to support the creation of documentation ā€œit is not finished until it is documentedā€

New hires

For new hires good some good tasks are:

  • Find something unclear, missing or in some way needing improvment on the docs and fix it.
    • This has the benefit that they need to interact with the docs and read it, at a deep enough level that they can improve it.
  • Taking our inline database of all systems on our network, and make sure it was correct by visiting the locations

Avoid

How to avoid the incident-handling train of the SRE will always fix everything.

A good question to anwser is ā€œWho is getting smarter and what are we doing about it?ā€ The ideal would be that you are getting good new info about your systems and how they fail.

But if just the SREs are the ones that are learning, you are in reverse, you are going the opposite direction of culture you are hoping to create.

If the answer is close to: SREs, engineering personnel and relevant stake holders, you can go to the next question, What is the rest of the org doing with this knowledge?.

Cool Ideas to improve culture

  • Start a postmortem group,
    • Where someone does a writeup of an issue/outage that happened. Give it in advance before the meeting, and discuss what could have been done diff, or proposal on the arch, stuff like that.
      • Good first questions are:
        • What is missing from the write-up we have here?
        • What do we still not know, or what may we never know?
        • Why wasnā€™t this worse?

SRE Advocacy

There are two times where SRE Advocacy is most imporant

  • early stages: you need to be able to tell people why they need an SRE
  • expansion phase: ā€œCool, youā€™ve been able to set up a new SRE group. Now you have to get others to play with you. How do are you going to do that?ā€

Humans are weird to be story-receiving machines. Letā€™s go back to the definition

Site reliability engineering is the discipline devoted to helping orgs sustainably achieve an appropriate level of reliability in their systems, services and products.

You can tell stories aboutā€¦

  • Efficacy
    • Partner suffering with reliability issues, SRE got involved and helped with X, Y and Z and now they are in a better place.
  • Reputation
    • How famous company X adopted SRE
  • Possibility
    • How comparable company X adopted SRE, how it went, issues, how they overcame. If they can you can do it to
  • Surprise
    • Story about an outage and the surprising result or finding uncovered by SRE postmortem process
  • Transformation
    • how things use to be, but now we are in a better place
  • Puzzle
    • X was a situations that made no sense; hereā€™s how we solved the mystery step by step

Keep notes on things that happen, the brain is bad at memory stuff.

Add cliffhangers to your stories, ā€œall pointed to human error, but something did not feel right..ā€

ā€œHere is how I failed and leveled up based on that experienceā€

Becoming SRE for the Individual

Preparing to be an SRE

This is not like super mandatory, but will help you a lot with being an SRE:

Do I need to know how to code? Yes

  • If you do not know how something is built your ability to understand it decreases
  • Learning how to code will teach you how to debug/troubleshoot
  • Many tools come in data formats that developers use in their day to day (JSON, YAML)

Fundamentals

Is always good to know about the Big O notation, this will let know people you know about data structures and efficiency.

  • Single/Basic Systems
    • You need to know how an OS works, about networking, permissions, protocols and how they fail.
  • Distributed Systems
    • Almost everything now is a distributed system
  • Statistics and Data Visualization
    • You need to understand and talk easily about, percentiles, standard statistical operations, aggregate and compound.
    • The ability to improve reliability in most situations is predicated on how to have conversations about data
  • Storytelling
    • Post incidents reviews and post-mortems are literally tell a story
  • Be a good person

Other nice to have

  • Non-Abstract Large System Design (NALSD)
  • Resilience Engineering
  • Chaos Engineering and Performance Engineering
  • Machine Learning and Artificial Intelligence

Getting to SRE fromā€¦.

From Dev/SWE

You need to shift your focus to these areas:

  • How does your code really function when in production?
    • How will it behave when the internet can throw any input at it.
    • What happens if the connection to the DB is slower than in your dev env.
  • Failure nodes
    • Do you have a good sense on how your code fails?
    • How can you tell when it is about to fail?
  • What have you built into the code to make figuring out whether your code is running well easy or possible?
  • How easy have you made handle upgrades?
    • and rollbacks?
  • Have you written good docs on for an SRE audience?

Basically: How much do you think about running your systems in additions to building them?

From Sysadmin

Sysadmins live to serve, they bridge the gap between technology and the people that use it.

You already have a good well-exercised skill for troubleshooting and debugging.

Do exercises in SadServers, like leet code but for infrastructure.

Change you mindset from ā€œmonitor all the thingsā€ to ā€œmeasure reliability from the customer perspective, not the component perspectiveā€

Using terms like ā€œcontributing factorsā€ instead of ā€œroot causeā€ can change both your internal and external way of looking at an issue. More on this on chapter 10.

Since you are already answering tickets, getting emails on things failing, you can start using that as a data set to start measure the reliability of the systems and how/why they fail.

More advice

Do not forget to track your progress, so you have something to look at when they reject you from an interview, or when you feel indifference to your ideas.

Hints for Getting Hired as an SRE

Some general advice of getting hired as an SRE

Looking closely at the job posting

Few thing to check:

  • The tech mentioned
    • Modernity of the tech
    • How items hand together
      • k8s/prometheus makes sense, k8s/nagios does not
    • Mention of Ticketing System
      • How quickly will things move in that env if its ticket based.
    • Specific versions of sw
      • they need a very specific thing
    • Mix of on-premise and cloud products
      • Are they all in the same env?
    • Mention of programming languages
      • Coding has meaning to them
    • Heavily skewed toward CI/CD and env provisioning tools
      • May have been a devops position, which is a diff mindset
    • Presence or absence of a monitoring tech
      • What connection if any would monitoring have to this role?
    • Services that are consumed
      • What am i getting into from dependencies perspective
  • The human connection
    • Look for an indication of the stakeholders or collaborators

Look for post-incidents reviews, they can be a helpful way of how they handle issues, whatā€™s their SW stack and stuff like that. Do not bring that up in the interview unless they do.

Preparing for an SRE interview (Resources)

Depends (of course) of the posting, it could be more SWE focus, or CI/CD focus, but there are four things you need to study for the interviews:

  • NALSD (non-abstract large system design)

    For systems that require scale (most of them)

    Resources:

    • The Site Reliability WorkbookĀ has a lovely chapter on the topic.
    • There are a lot of talks on the topic USENIX
    • Google has a public classroom for this https://sre.google/classroom/

  • Monitoring/observability

    Good places to start:

    • Practical MonitoringĀ by Mike Julian (Oā€™Reilly, 2017)
    • Observability EngineeringĀ books by Charity Majors et al. (all from Oā€™Reilly)

    If you expect to talk about SLIs and SLOs review

    • Implementing Service Level Objectives(Oā€™Reilly, 2020).

  • Computing Fundamentals

    • Computer science, computer networks, linux, distributed computing, stuff like that.

  • Troubleshooting and debugging

    • Hopefully you have experience with this one, but sadservers.com is a good start

What to ask at the SRE Interview

Some good conversation starters:

  • Tell me about your monitoring system

    This exposes all sort of info on organization, structure, collaboration, ownership, How decision are made, and so on.

    Some follow up questions:

    • Who owns monitoring in your org?
    • How many monitoring systems are there in active use?
    • Who (apps/services/teams) send data to those systems, and who access them?
    • How easy it is to onboard a new app/service to your monitoring?
    • What decisions are made using the data?
    • Are there alerts generated fromt his system?
    • What makes you happy and unhappy with your current system?

  • Tell me about your post-incidents review process

    Here you are trying to see how intentional are they on learning from failure

    • Do you hold post-incidents review after your outages?
    • What is their purpose?
    • Who is ā€œin the roomā€ for them?
    • How do you document your outages?
    • Can you tell me (at whatever level of details you are comfortable) about a recent outage
    • Do you have a sense of the most common classes of outage you have seen in the last N month? (config related, overload/resource-deprivation failures, code bug)

  • Tell me about you on-call setup

    • Do people get time off after incidents?
    • Who in the org participates in an on-call rotation (just SREs? Devs? Managers?)
    • When was the last time you personally were ā€œpagedā€?
    • Do people get paged equally often between work and off-work hours?

  • What problem does SRE exist to address in your org?

    • If they cannot answer that: What are some ā€˜recent winsā€™ by SRE in the past 6 to 12 months?

  • Can SREs check in code to major repos in your org?

    • Youā€™ll see how involved is the SRE with dev work

A Day in the Life of an SRE

Because of the nature of the SRE role, it is hard to describe an average day, since it most days are different to each other. So instead of an average day, we have different modes SREs can relate to. Look at it like different hats an SRE will wear

  • Incident/Outage Mode

    There will be days were most of your time you will be involved dealing with an accident. These days come with some feelings attached to them (fear, anxiety, and so on), the intensity of these feelings depend on the severity of the outage.

    When on this mode you will be reacting not planning it is normal.

  • Post-incident Learning Mode

    After the incident now you have the opportunity to review the memories of an outage and learn from it.

    You are responsible for documenting it, in a way others can understand it. To do this you will have to investigate a bit, look for data in your monitoring systems, talk to your colleagues to discover what they knew and when.

  • Builder/Project/Learn Mode

    This is when you actually have time to sit and...

    • Devlop some code for services or SRE tasks
    • Provision new envs or infra
    • Improve monitoring/observability
    • Removing toil
    • Writing docs
    • Learning a new tech in anticipation of using it some day

    Of course, there will be times when you are doing boring stuff, but that can help you identify toil you need to remove.

  • Architecture Mode

    Depends on your org, but, and SRE should be showing up to design and planning meetings where they are acting as a representative for reliability.

    Be political about it, no one wants to hear, "this would have never happen if there was an SRE when this thing was designed". Appeal to the sense that everyone want to have their code in production be as reliable as possible.

  • Management Mode

    If the response to, what did you do all day?, was, I went to meetings, do not worry, chances are you might be an SRE manager.

  • Planning Mode

    Some portion of your day will be planning.

    • Implementation plans
    • Capacity planning
    • Self-definitional work (goals of SRE team and stuff like that)

  • Collaboration Mode

    The SRE role is relentlessly collaborative.

    When you implement SLI/SLO (Service Level Indicators/Service Level Objectives) you will be working with: devs, PMs, stakeholders.

    Another example of collaborative work, is what some people call pre-launch review. An SRE gets involved to revise the service being deployed in production, what is necessary for it to run reliably in production

    Do not be a gatekeeper, share this work with the devs and the stakeholders, collaborate as much as possible

    Finally, listen to the customers through monitoring work. The SLIs/SLOs (Service Level Indicators, Service Level Objectives) are meant to provide ongoing collaboration with the customer.

  • Recovery and Self-Care Mode

    Burn out SREs are of no good to anyone. Because of the nature of SREs it can be easy to overextending yourself. But when we hear that someone is regularly working 60-75 hours, is not something to be proud of, that means there is a failure in the system and needs to be fixed.

    You need to have recovery time.

  • On Balance

    Balance is something good to strive for, but there are often situational factors that complicate the effort, for example, an early service vs mature service, new services are always noisier and require more reactive work. They also provide more toil to be stripped away, so maybe you expend more time in one of these modes than in others.

    The idea is to see this as weather patterns, I know it is going to rain hard for some time, but I mentally prepare. Ideally things will level out. If not, you need to strive for it. SRE attempts to be a sustainable operations practice. If you realise this cannot be sustained maybe you need to start looking for a different job.

Establishing a Relationship to Toil

If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes are your systems grow.

Carla Geisser, Google SRE

What is toil?

First of all we need to define toil. Toil is not the work you do not like to do or simply repetitive chores. Toil is the kind of work when running a production service, that to tends to be:

  • Manual Work like manually running a script that automates a task is still manual, the time that a human spends running the script is still toil time

  • Repetitive Toil is work you do over and over.

  • Automatable If human judgment is essential for the task, there is a good chance it is not toil.

  • Tactical Toil is reactive rather than strategy-driven.

  • Does not bring enduring value If your service is in the same state after you finished a task, the task was probably toil.

  • O(n) with service growth If the work involved scales linearly with the size of the service (traffic volume, service size) is it probably toil.

We need to have less toil because it tends to expand, if left unchecked it can quickly fill 100% of everyone's time, and people will stop doing Engineering Work. What is that? Engineering work is novel and requires human judgment. It produces permanent improvement in your service and is guided by strategy.

Now that we know what toil is, lets see the relationship SREs have to it.

Whose Toil Are We Talking About?

Whose toil is it? On opposed to other parts of SRE where we are customer focuses, here we need to focus on our toil, not the customers one. Sure, they might have a connection for example, operational toil (ours) is exposed to the customer if they have to go into 4 steps to make a request. But we need to keep the main focus on the operational toil.

Why do SREs Care about toil?

An argument can be made that if you remove toil a system becomes more reliable, but the author suggests that sometimes this is not the case. An that SREs because of their nature, are inclined to eliminate toil because of the following reasons.

  • Aesthetics SREs want to eliminate toil because if offends their aesthetic sensibilities. Toil is inelegant, inefficient, unnecessary, suboptimal, hard to look at. That simply is a reason to remove it.

  • Money Orgs have many reasons to want their expensive people do work that is significant that make the revenue forward, meaning the antithesis of toil.

  • Job Satisfaction

Early vs Established Toil

When an app is developed it is likely to have more toil than once it is established. Why? Developers care about making a solution to a customer problem, not making their app be super operational. They can be, and that is why it is important that an SRE is in the room when planning the Architecture for the app, but it is likely that it will have more toil than an established one.

It is important to note this, because now we can mentally prepare for it, we now know that there will be a finite period of work with a lot of toil, but it is expected and it will end.

Dealing with Toil

Usually people just say, automate it, and the toil will go away. But the author propose the idea that similar to matter that toil is not created and cannot be destroyed just transformed.

When you are automating a task, the toil did not disappeared it just got transformed into a different thing: Complexity. Usually this is a wise bargain to take, but it is important to keep in mind that it has its trade offs.

Intermediate to Advanced Toil Reduction

It is important to keep track of the toil we remove on individual systems, management will love to hear that X system required N manual steps and now it was automated and require N-4 steps. But once you pass that first stage you also need to start thinking. How can I reduce the toil we are going to have?

Go Remove the Toil

That is pretty much it for this chapter, we defined what toil is and how to deal with it. Go and put in practice what you have read here.

ā›µļø k8s

Super surprised someone still writes the full Kuberenetes word instead of k8s. K8s looks way cooler

Specific k8s stuff

Things surrounding k8s

šŸ‹ k8s up and running

āš ļø These are notes taken from the book Kubernetes Up and Running by Brendan Burns, Joe Beda, and Kelsey Hightower O'Reilly. Please go read it and just use this as reference.

Introduction

It has become the standard API for building cloud native applications. It provides the software needed for build and deploy, reliable, scalable distributed systems.

Distributed systems: Most services nowadays are delivered via the network, they have different moving parts in different machines talking to each other over the network via APIs.

Reliable: Since we rely a lot on these distributed systems, they cannot fail, even if some part of them crashes, they need to be kept up and running, and be fault-tolerant

Availability: They must still be available during maintenance, updates, software rollouts and so on.

Scalable: We live in a highly connected society, so these systems need to be able to grow their capacity without radical redesign of the distributed systems.

Most of the reasons people come to use containers and container orchestration like k8s, can be boil down to 5 things: velocity, scaling, abstracting your infra, efficiency, cloud native ecosystem

Development Velocity

Today software is delivered via de network more quickly than ever, but velocity is not just a matter of raw speed, people are more interested in a highly reliable service, all users expect constant uptime, even as the software is being updated.

Velocity is measures in term of the number of things you can ship while maintaining a highly available service.

k8s give you the tools to to do this, the core concepts to enable this are:

  • Immutability

    Software used to be mutable (imperative), you installed vim using dnf and then one day you would do dnf update and it would add modifications on top of the binary that you already have for vim. This is what we call mutable software.

    On the other hand, immutable software, you would not add on top of the previous release, you would build the image from zero. Which is how containers work. Basically when you want to update something in a container, you do not go inside it and do dnf update vim you create a new image with the version of vim you want, destroy the old one, and create a container with the new one.

    This makes rollbacks way easier, and it keeps tracks of what changed from version to version.

  • Declarative Configuration

    This is an extension of immutability but for the configuration. It basically means that you have a file where you defined the ideal state of your system. The idea of storing this files in source control is often called ā€œinfrastructure as codeā€.

    The combination of declarative state stored in a version control system and the ability of k8s to make it match the state makes rollback super easy.

  • Self-Healing Systems

    K8s will continuously take action to make sure the state on the declarative configuration is met. Say it has declared 3 replicas, then if you go and create one extra, it will kill it, and if you were to destroy one, it would create it.

    This means that k8s will not only init your system but will guard it against any failures that might destabilize it.

Scaling Your Service and Your Teams

K8s is great for scaling because of its decoupled architecture.

Decoupling

In a decoupled architecture each components is separated from each other by defined APIs and service loads

This basically means that we can isolate each service.

  • The API provide a buffer between implementer and consumer.
  • Load balancing provide a buffer between running instances of each service.

When we decouple the services, it makes it easier to scale, since each small team can focus on a single microservice

Scaling for Applications and Clusters

Since your app is deployed in containers, which are immutable, and th configuration o is also declarative, scaling simply becomes a matter of changing a number in a configuration file. Or you can even tell k8s to do that for you.

Of course k8s will be limited to the resources you have, it will not create a physical computer on demand. But k8s makes it easier to scale the cluster itself as well.

Since a set of machines in a cluster are identical and the apps are decoupled from the specifics of the machine by containers, scaling a cluster is just a matter of provisioning the system and joining it to a cluster.

K8s can also come in handy when trying to forecast costs on growth and scaling. Basically, if you have different apps that need to scale from many teams, you can aggregate them all in a single k8s, and try to forecast the usage of the 3 apps together.

Scaling Development Teams

Teams are ideal when using the two-pizza team when (6 to 8 people), larger teams tend to have issues of hierarchy, poor visibility, infighting and so on.

K8s can help with that, since you can have multiple small teams working on different microservices and aggregate them all into a single k8s cluster.

K8s has different abstractions that make it easier to build decoupled microservices.

  • pods: are groups of containers, can contain images developed by different teams into a single deployable unit.
  • k8s services: provide load balancing, naming and discovery to isolate the microservices.
  • namespaces: provide isolation and access control, so that each microservice can control the degree to which other services interact with it.
  • ingress: easy to use front end that can combine microservices into a single API.

Separation of Concerns for Consistency and Scaling

K8s leads to greater consistency for the lower levels of the infrastructure. Basically the devs need to only worry about their app meeting the SLA (service level agreements) while the orchestration admin only need to worry about reaching the SLA from their side.

Abstracting You Infrastructure

When you build your app in terms of containers and deploy it via k8s APIs, you make moving your app between environments super easy. It is simply a question of sending the declarative config to a new cluster.

K8s make building deploying and managing your app truly portable across a wide variety of environments

Efficiency

Efficiency is measured by the ratio of the useful work performed by a machine.

K8s enables efficiency in multiple ways. Since developers do not need to use a whole machine for himself/herself, but rather one container, multiple users can share the same baremetal. Meaning less idle CPUs.

Cloud Native Ecosystem

K8s is open source and has a huge community building a thousand things on top of it. You can check the maturity of different projects in Cloud Native Computing Foundation. The maturities are sandbox incubating and graduated, going from less mature to most mature.

Summary

Fundamentally k8s was design to give developers more velocity efficiency and agility. Now that we have seen why to use it, lets see how to use it.

Deploying a k8s cluster

You can use a cloud provider, deploying on bare-metal quite hard, other good alternatives.

http://kind.sigs.k8s.io that has a really cool logo, and works on using containers for the nodes.

Go check the pages of cloud providers on how to deploy it there since it will change depending on the provider.

One thing to bare in mind is that in order to use kubectl with your cluster you need a ~/.kube/config file, probably your cloud provider will give that to you.

Also you can go to k8s cheatsheet (describe cluster) to see the basics commands on describing a cluster.

I will just cover some of the commands here, like

k describe nodes <node name>

At the top you can see basic node info:

Name:               kind-control-plane
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=arm64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=arm64

We can see that it is linux on ARM

Next we can see info about the operation of the node

Conditions:
Type             Status  LastHeartbeatTime  LastTransitionTime Reason                       Message
----             ------  -----------------  ------------------ ------                       -------
MemoryPressure   False   29 Apr 2024        29 Apr 2024        KubeletHasSufficientMemory   kubelet has sufficient memory available
DiskPressure     False   29 Apr 2024        29 Apr 2024        KubeletHasNoDiskPressure     kubelet has no disk pressure
PIDPressure      False   29 Apr 2024        29 Apr 2024        KubeletHasSufficientPID      kubelet has sufficient PID available
Ready            True    29 Apr 2024        29 Apr 2024        KubeletReady                 kubelet is posting ready status

Here we can see that the node is Ready, that it has sufficient memory, it has no disk pressure, and that is has sufficient PID.

Then we see info about the capacity of the system.

Capacity:
cpu:             1
memory:          1933908Ki
pods:            110
Allocatable:
cpu:             1
memory:          1933908Ki
pods:            110

Then info about software in the node like OS image, docker version all that.

System Info:
# more stuff
Kernel Version:             6.5.6-200.fc38.aarch64
OS Image:                   Debian GNU/Linux 12 (bookworm)
Operating System:           linux
Architecture:               arm64
Container Runtime Version:  containerd://1.7.13
Kubelet Version:            v1.29.2
Kube-Proxy Version:         v1.29.2

Then info about the pods running on the node:

Non-terminated Pods:          (9 in total)
Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
---------                   ----                                          ------------  ----------  ---------------  -------------  ---
kube-system                 coredns-76f75df574-92286                      100m (10%)    0 (0%)      70Mi (3%)        170Mi (9%)     23s
kube-system                 coredns-76f75df574-m2z2k                      100m (10%)    0 (0%)      70Mi (3%)        170Mi (9%)     23s
kube-system                 etcd-kind-control-plane                       100m (10%)    0 (0%)      100Mi (5%)       0 (0%)         39s
kube-system                 kindnet-qm29d                                 100m (10%)    100m (10%)  50Mi (2%)        50Mi (2%)      23s
kube-system                 kube-apiserver-kind-control-plane             250m (25%)    0 (0%)      0 (0%)           0 (0%)         39s
# more stuff

We can see that k8s tracks both requests and upper limits for resource for each Pod. The difference is that resources requested by a pod are guaranteed to be present in the node, while pod's limit is the max amount of given resource that pod can consume

Cluster Components

Many components that make up the k8s cluster are actually deployed using k8s itself. All of them run in the kube-system namespace

  • k8s proxy

    It routes network traffic to load-balanced services in the k8s cluster. It must be present on every node in the cluster

    % k get daemonSets --namespace=kube-system kube-proxy

  • k8s dns

    provides naming and discover for the services that are defined in the cluster

    % k get deployments --namespace=kube-system coredns

Common kubectl commands

I will repeat the commands here and in the page: k8s cheatsheet but I will go a bit more into detail here. I will also alias k=kubectl

kubectl is the way you talk with k8s API. Here we will go over basic kubectl commands.

Namespaces

Namespaces are like folders that hold a set of objects. They are used to organize the cluster.

By default the kubectl command interacts with the default namespace, but you can specify which you want to use by -n or --namespace=<ns>.

You can also use --all-namespaces flag to refer to all the ns.

Contexts

If you want to change the namespaces in a more permanent way, you can use contexts. These get recorded in your ~/.kube/config file. (That file also stores how to find and auth to your cluster)

You can create a with a different default namespaces using

% kubectl config set-context my-context --namespace=mystuff

We need to tell kubectl to start using it.

% kubectl config use-context my-context

You can also use context to manage different clusters users for auth using --users or --cluster flags with set-context read the man for more info

Viewing k8s API Objects

Basically k8s is an API and kubectl is just an http client. To see the k8s objects (everything represented by a RESTful resource) we use k get <resource name>

By default it will throw to the STDOUT human readable stuff, but you can also use -o json or -o yaml in case you want it in specific format.

Also when using awk you may want to use --no-haders.

You can also get multiple objects

k get pods,services

To get more info on a particular object

k describe <resource name> <obj name>

If you want to see like a mini man page of an specific k8s object:

% k explain pods
KIND:       Pod
VERSION:    v1

DESCRIPTION:
    Pod is a collection of containers that can run on a host. This resource is
    created by clients and scheduled onto hosts.

FIELDS:
  apiVersion	<string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:

Sometimes you want to continually observe the state of a k8s resource, like when waiting for an app to restart of something you can use the --watch flag:

k get pods -n kube-system --watch

Creating, updating and destroying k8s objects

Objects in the k8s API are represented as JSON or YAML files. We can query the server for those, or post them with an API request.

You use these JSON and YAML files to create, update or delete objects in your k8s server.

Create an object from obj.yaml file.

% k apply -f obj.yaml

The yaml file will have the resource type of the object.

You can use the same command to update an object

% k apply -f obj.yaml

If nothing change it will not do anything, it will exit successfully, good for for loops. You can do --dry-run to print objects to the terminal without actually sending them.

To do interactive edits you can use

% k edit <resource-name> <obj-name>
# example k edit pod coredns-76f75df574-k97nj  -n kube-system

It will open the yaml in a text editor, wonce you save it will automatically be uploaded back to the k8s cluster.

The apply command also has a history of the previous configurations edit-last-applied, set-last-applied and view-last-applied for example:

% k apply -f myobj.yaml view-last-applied

To delete you can simply run

% k delete -f myobj.yaml

You can also delete an object without the file using

% k delete <resource-name> <obj-name>

Labeling and Annotating Objects

You can update the labels and annotations using label and annotate (who would have thought).

For example to add the color=red label to the pod named bar you can run:

k label pods bar color=red

You can not rewrite unless using --overwrite flag.

Finally you can remove a label by doing

k labels pods bar color-

Debugging Commands

You can check logs from a pod by:

k logs <pod-name>

If you have multiple container in your pods you can choose which container to view using the -c flag.

If you want to follow add -f.

You can also use the exec command to execute a command in a running container.

k exec -it <pod-name> -- bash

If you do not have bash or a shell in your container, you can attach to a running process

k attach -it <pod-name>

You can also copy files to and from a container using cp

k cp <pod-name>:/path/to/remote/file /path/to/local/file

This will copy a file from the container to your local machine.

You can also forward traffic from your local system to the pod

k port-forward <pod name> 8080:80

Forwards traffic from the local machine on port 8080 to the remote container on port 80

If you want to see the last 10 events in a namespace

k get events

You can also stream events --watch

Finally, to check cluster resources being used

k top nodes

or

k top pods

These commands will only work if metrics servers are present, which they likely are.

Cluster Management

Cordon and drain a particular node:

  • cordon: prevent future pods form being scheduled onto that machine
  • drain: remove any pods that are currently running on that machine.

Useful for removing physical machine for repairs or upgrades. You would do k cordon and then k drain to safely remove the machine form the cluster.

Once the system is back online do k uncordon there is no undrain it will naturally get back to normal.

Pods

K8s groups multiple containers into a single atomic unit called a pod (This also goes with the docker theme, since a group of whales is a pod lol)

A pod is a collection of app containers and volumes running in the same environment. Containers in a pod always land in the same machine.

Apps running in the same pod share, ip addresses and port, hostname, and can communicate to each other using interprocess communication channels. Meaning they share:

  • network namespace
  • UTS namespace
  • IPC namespace

Thinking in Pods

What should go in a Pod? To answer this question you go look at your different containers and ask "Will these containers work correctly if they land on different machines?" If no, then a pod is the correct way of grouping them. If yes, you can use multiple pods.

An example, say you want to deploy a wordpress app with a mysql database. Would you put them in the same pod? I already read the book so I know the answer is no, but why?

The answer is no because both of those services do not need to scale together. The wordpress app itself is stateless, so you can create more wordpress pods if you are scaling. On the other hand a mysql database is much different to scale, it can be trickier, and you are likely to increase the resources dedicated to a single mysql pod.

The Pod Manifest

Pods are describe in a manifest which is a text-file representation of the k8s API object.

As you might remember from the fist chapter, k8s uses a declarative configuration, which basically means that you tell k8s your ideal state of the world and it will take action to ensure the state becomes a reality.

When you send a manifest to k8s, it will accept it and process it before storing it in persistent storage (etcd). The scheduler will find pods that have not been assigned to a node, and will place the pods onto those nodes depending on the resources and constrains you specified in your manifest. K8s will try to ensure pods from the same app are in different machines for reliability.

You can see the pods with

k get pods

You can delete a pod with

k delete pods/<name>

Creating a manifest

Pod manifest should be treated in the same way a source code, adding comments to help explain the pod to new team members and stuff like that.

The manifests include a couple of key fields and attributes,

  • metadata:
    • name: a unique identifier for the deployment
    • namespace: the Kubernetes namespace where the deployment will live
    • labels: key-value pairs that can be used to filter and categorize objects
  • spec:
    • containers:
      • name: a unique identifier for the container
      • image: the Docker image to use for this container
      • command: the default command to run in the container
      • args: any additional arguments to pass to the command
      • volumeMounts: mounts for persistent storage or other files
    • volumes:
      • name: a unique identifier for the volume
      • persistentVolumeClaim: claims a Persistent Volume resource
  • selectors: used to filter which pods are included in this deployment

Example

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-04-29T20:02:16Z"
  generateName: coredns-76f75df574-
  labels:
    k8s-app: kube-dns
    pod-template-hash: 76f75df574
  name: coredns-76f75df574-k97nj
  namespace: kube-system
  # more stuff
  uid: 4dd80223-de4c-49d7-b407-31908ae96b9e
spec:
  # more stuff
  containers:
  - args:
    - -conf
    - /etc/coredns/Corefile
    image: registry.k8s.io/coredns/coredns:v1.11.1
    imagePullPolicy: IfNotPresent
    # more stuff

    volumeMounts:
    - mountPath: /etc/coredns
      name: config-volume
      readOnly: true
    # more stuff

  volumes:
  - configMap:
      defaultMode: 420
      items:
      - key: Corefile
        path: Corefile
      name: coredns
      # more stuff

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-04-29T20:02:19Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
  # more stuff

You can create a pod from them by using k apply -f manifest.yaml as seen in previous sections.

Some Useful Commands

# get info
k get pods  # list pods
k describe pods <podname> # more info on the k8s object

# manipulate pods
k delete pods/<podname> # delete pod, all info associated to it will be deleted
k logs <podname> # check logs
k exec <podname> -- date # run date command inside pod
k exec -it <podname> -- bash # interactive tty
k cp # keep in mind that copying files into a container is an antipattern.

Health Checks

When you are running an app in k8s, it is automatically kept alive by a process health check. This ensure that the app is always running, if it is not it k8s will restart it.

Sometimes, checking if the process is alive is not enough, maybe it was left hanging or something like that. K8s has something to check application liveness and other stuff for your app.

There are different kinds of probes here are some:

  • Liveness Probe This will check app specific stuff, for example checking if http is returning 200.

    While the default response to a failed liveness is to restart you can change what happened with restartPolicy

    Here is an example of a livenessProbe

    spec:
  containers:
  - name: my-container
    image: my-image:latest
    livenessProbe:
      httpGet:
        path: /healthz
        port: 80
        scheme: HTTP
        initialDelaySeconds: 30
        timeoutSeconds: 5

  • Readiness Probe These probes check if an app is ready to serve user requests. Apps that fail are removed from the load balancers

  • Startup Probe Enables you to poll slowly from a slow-starting container, while also enabling responsive liveness checks once the container has initialized

Resource Management

K8s allow you to increase the overall utilization of the machines that are in the cluster.

Utilization is defined as the amount of resource actively being used divided by the amount of a resource that has been purchased

Example, if you purchase a one-core machine, and your app uses one tenth of a core, then your utilization is 10%

You can tell k8s the upper limit and lower limit of resources you want for an specific container. Keep in mind that this is on a per-container basis, the whole pod will the sum of the containers.

You can do this by setting up resources in the manifest. The lower limit is requests and the upper limit is limits.

spec:
    containers:
    - image: blabla:latest
      name: jose
      resouces:
        cpu: "500m"
        memory: "128Mi"
      limit:
        cpu: "1000m"
        memory: "256Mi"
    # more stuff

Persisting Data with Volumes

When a container restarts, any data in it will also be deleted. Which is good since we do not want to leave around cruft.

We can still add volumes to our pods though, that will persist this deletion

In order to do this, you need to two things to the manifest spec.volumes, which is an array of all volumes that the pod will need.

And the other is volumeMount which specify inside the container definition the file system to use.

So basically one is like, what volumes will the whole pod use, and the other is specific to each container.

spec:
  containers:
  - name: my-container
    image: busybox:latest
    volumeMounts:
    - name: my-filesystem
      mountPath: /mnt
      subPath: data
  volumes:
  - name: my-filesystem
    persistentVolumeClaim:
      claimName: my-pvc

Labels and Annotations

K8s apps can grow in size and complexity real quick. We can use labels and annotations to help organize our k8s objects/resources as sets of things that map how we think about the app. Making it a bit more human accessible. Meaning we can group resources that make the most sense of the application.

We have two tools that help with this.

  • Labels: key/value pairs, to attach identifying information to the k8s obj/resource. You will be able to query based on this.
  • Annotations: set non-identifying information that libraries can leverage. These are not meant for querying

Labels

Labels give identifying metadata for k8s objects. They are key/value pairs of strings.

Keys are broken down into two parts

  • Optional prefix name, separated by a slash.
  • Key name, it must start and end with alphanumeric chars.

Some examples:

  • acme.com/app-version - 1.0.0
  • app.version - 1.0.0

Applying Labels

You can do it when doing the k run command. (Doubt that is the way you are using it but anyway):

k run alpaca-test \
  --image=gcr.io/kuar-demo/kuard-amd64:green \
  --labels="ver=1,app=alpaca,env=test"

You can also add it to the manifest of the resource you are using:

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  labels:
    ver: "1"
    env: "test"
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80

After the objects have been created, you can do --show-labels

k get deployments --show-labels

Modifying Labels

You can update the labels on objects after creating them:

k label deployemnts alpaca-test "canary=true"

Keep in mind that this will only update the label on the deployment itself, it wont affect any objects created by the deployment.

In case you feel lazy you can replace --show-labels with -L.

k get deployemnts -L canary

To remove a label you add a dash-suffix

k label deployments alpaca-test "canary-"

Label Selectors

Okay so, how do we query this stuff? Using boolean expressions

This will list only the pods that have ver label set to 2.

k get pods --selector="ver=2"

You can get a bit more creative:

This is the AND operator:

k get pods --selector="app=bandicoot,ver=2"

You can kind of do the OR by:

k get pods --selector="app in (alpaca,bandicoot)"

You can also check if a label has been set by just using the key name:

k get deployments --selector="canary"

Check if key is not in value1 or value2:

k get pods --selector="key notin (value1, value2)"

You can combine them

k get pods --selector="ver=2,!canary"

Labels Selector in Manifests

Newer versions support doing something like this:

selector:
  matchLabels:
    app: alpaca
  matchExpressions:
    - {key: ver, operator: In, values: [1, 2]}

Annotations

This are only meant to assist tools and libraries, do not use them for queries.

If you are wondering if something should be a label or an annotations, add information to an object as an annotation and promote it to a label if you find yourself wanting to use it in a selector.

Usually annotations primary use is rolling deployments, they are used to track rollout status and provide the information for a roll back if needed.

Annotations are defined in the metadata part of the manifest:

...
metadata:
  annotations:
    example.com/icon-url: "https://example.com/icon.png"
...

Service Discovery

K8s is a dynamic system. It has to manage placing pods on nodes, make sure they are running and more stuff. Due to this nature k8s makes it easy to run a lot of things, but it can be a bit hard to find these things.

What is Service Discovery?

A service is a method for exposing a network application that is running in one or more Pods in your cluster

Actually you use one type of service discover everyday, DNS, the thing that helps you resolve hostnames to IPs. It works great for the internet, but falls short in the k8s world.

Why? DNS caches a bunch of stuff, and in a k8s cluster pods are ephemeral, they can be killed and come back up in seconds, so due to this cache we could be pointing to something that is already dead.

While these pods that compose the backend may change, the frontend clients should not need to be aware of that, nor should they need to keep track of the of the backend themselves.

K8s has an object/resource specific to this issue.

The Service Object

A Service is an object, it has a manifest and all that. Say you have a set of pods that each listen on port 5000 and are labeled as name=flask-app. The manifest would look something like this:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app.kubernetes.io/name: flask-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000

K8s will assign the service a Cluster IP , and will be continuously scanning for Pods that match the selector (the flask-app part). Then it will make the updates to the set of EndpointSlices for the Service.

Using the type NodePorts

By using this type, in addition of a Cluster IP, the system picks a port on every node in the cluster to forward traffic to the service.

Meaning you can connect to any node in the cluster, and reach an specific service. You can use NodePort without knowing where any of the pods for that service are running.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: NodePort <--- here
  selector:
    app.kubernetes.io/name: MyApp
  ports:
    - port: 80
      # By default and for convenience, the `targetPort` is set to
      # the same value as the `port` field.
      targetPort: 80
      # Optional field
      # By default and for convenience, the Kubernetes control plane
      # will allocate a port from a range (default: 30000-32767)
      nodePort: 30007

Now each request that is send to the service will be randomly directed to one of the pods that implements the service.

ConfigMaps and Secrets

A good practice is to make your container images reusable, making a general purpose one that can be used across applications. But we might also need to add some configuration to the image at runtime. Here is where ConfigMaps and Secrets come into play.

ConfigMaps

In a ConfigMap you can define env variables, command line arguments or even small file systems for your containers. The configmap is combined with the Pod right before it is run.

Creating ConfigMaps

We can create a configmap out of a file. Say for example we have a file called config.txt:

parameter1 = value1
parameter2 = value2

You can create a configmap by doing:

% k create configmap config --from-file=config.txt

Or you can create literally from the command line

% k create configmap other-config --from-literal=some-param=some-value

Now you can list them

% k get cm
NAME               DATA   AGE
config             1      35s
some-config        1      2s

And get them as yamls

% k get cm config -o yaml
apiVersion: v1
data:
  config.txt: |
    param1 = value1
    param2 = value2
kind: ConfigMap
metadata:
  creationTimestamp: "2024-06-11T00:58:10Z"
  name: config
  namespace: default
  resourceVersion: "2420"
  uid: 7df23db6-7d2c-4db4-9b22-5fecd4bb456e

At the end of the day, the configmaps are just some key/value pairs stores in an object.

Using a ConfigMap

There are 3 ways of using them:

  • File system:
    • A ConfigMap can be mounted in a Pod. A file is created for each key, and the content is their respective values
  • Command-line Argument:
    • Creating command lines dynamically
  • Environment Variable:
    • It can set the value of an environment variable

Let's see each as an example

  • File System Volumes
apiVersion: v1
kind: Pod
metadata
  name: kuard-config
spec:
  containers:
    - name: test-container
      image: gcr.io/kuar-demo/kaurd-amd64:blue
      imagePullPolicy: Always
      command:
        - "/kuard"
    volumeMounts:
    # Mounting the ConfigMap as a set of files
    - name: config-volume # <- HERE
      mountPath: /config
  volumes:
    - name: config-volume # <- HERE
      configMap:
        name: config # <- name of the cm we just created
  restartPolicy: Never
  • Command-line Arguments
apiVersion: v1
kind: Pod
metadata
  name: kuard-config
spec:
  containers:
    - name: test-container
      image: gcr.io/kuar-demo/kaurd-amd64:blue
      imagePullPolicy: Always
      command:
        - "/kuard"
        - "$(EXTRA_PARAM)" <- NOTE HERE
      env:
        - name: EXTRA_PARAM
            valueFrom:
            configMapKeyRef
            name: config # <- name of the cm we just created
            key: param1

Note that k8s will perform the correct substitution with a special $(<env-var-name> ) syntax

  • Environment Variable:
apiVersion: v1
kind: Pod
metadata
  name: kuard-config
spec:
  containers:
    - name: test-container
      image: gcr.io/kuar-demo/kaurd-amd64:blue
      imagePullPolicy: Always
      command:
        - "/kuard"
      env:
        - name: ENV_PARAM
            valueFrom:
            configMapKeyRef
            name: config # <- name of the cm we just created
            key: param2

Secrets

This are pretty similar to ConfigMaps but they are intended for sensitive information, for example, passwords, tokens, private keys, TLS certs.

This secrets are exposed to Pods via explicit declaration in the Pod manifest and the k8s API.

By default k8s stores secrets in plain text on etcd storage. If you have super sensitive information there, and a lot of people with admin access, might be worth encrypting it a bit more.

Creating Secrets

We can create a secret like any other resource

% k create secret generic my-secret --from-file=some.key

Instead of generic -- depending on the secret -- we can use different types

From the manpage

Available Commands:
  docker-registry   Create a secret for use with a Docker registry
  generic           Create a secret from a local file, directory, or literal value
  tls               Create a TLS secret

If we describe it

% k describe secret my-secret
Name:         my-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
some.key:  1984 bytes

The Pods consume the secrets.

Consuming Secrets

We can call the API directly, but more often we will access secrets from a Secrets Volume. These volumes are created at pod creation time, and are stored in tmpfs (meaning memory not disk).

Each secret is in a different file under the target mount point. So if we mount the my-secret secret in the /keys directory, it would looks something like /keys/some.key.

The Pod manifest would end up looking something like this:

apiVersion: v1
kind: Pod
metadata
  name: kuard-secret
spec:
  containers:
    - name: test-container
      image: gcr.io/kuar-demo/kaurd-amd64:blue
      imagePullPolicy: Always
    volumeMounts:
    - name: keys
      mountPath: /keys
      readOnly: true
  volumes:
    - name: keys
      secret:
        secretName: my-secret # <- name of the secret we just created

Private Container Registries

You can also create a secret for for use with a Docker registry. Instead of generic you would use docker-registry, with --docker-username, --docker-password and --docker-email and the manifest would look something like this:

apiVersion: v1
kind: Pod
metadata
  name: kuard-secret
spec:
  containers:
    - name: test-container
      image: gcr.io/kuar-demo/kaurd-amd64:blue
      imagePullPolicy: Always
    volumeMounts:
    - name: keys
      mountPath: /keys
      readOnly: true
  imagePullSecrets:
  - name: my-image-pull-secrets-from-registry # <- name of that secret
  volumes:
    - name: keys
      secret:
        secretName: my-secret

Naming Constrains

Key names need to be valid variable names. Just use normal names and you will be fine.

  • valid

    • .token
    • secret.token
    • config_file

  • not valid

    • token..key
    • auth token.key
    • _token

Just not use spaces, double dots and stuff like that.

Managing ConfigMaps and Secrets

The usual stuff, get, delete, create. Just a note on creating:

  • --from-file=<filename>: keys in file will be keys in secret
  • --from-file=<key>=<filename>: keys in file will be keys in secret
    • Instead of: 
        Data
  ====
  <filename>:
  ----
  param1 = value1
  param2 = value2
      You would get 
        Data
  ====
  <key>:
  ----
  param1 = value1
  param2 = value2
  • --from-file=<directory>: loads all files in a directory

Also you can recreate and update like:

kubectl create secret generic kuard-tls \
  --from-file=kuard.crt --from-file=kuard.key \
  --dry-run -o yaml | kubectl replace -f -

Neat trick from the book Kubernetes Up and Running by Brendan Burns, Joe Beda, and Kelsey Hightower O'Reilly.

šŸ“ k8s cheatsheet

Some useful commands

using kind?

in you mac? with podman?

podman machine start;
export KIND_EXPERIMENTAL_PROVIDER=podman;
kind create cluster

Describe Cluster

  • Check version

    % k version  # what do you think it does? :)
*Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2*

  • Check cluster status

    % k get componentstatuses # will be deprectated
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   ok

    or talk directly to the endpoint

    % k get --raw='/readyz?verbose'
[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
[+]informer-sync ok
 # more messages
 readyz check passed

  • List k8s nodes

    % k get nodes
NAME                 STATUS     ROLES           AGE   VERSION
kind-control-plane   Ready   control-plane   13s   v1.29.2
node01               Ready   <none>          13s   v1.29.2

    In k8s nodes are separated into control-panel nodes, like the API server, scheduler, etc. which manage the cluster, and worker nodes where your containers will run.

  • Get more information about a specific node (prints bunch of info)

    % k describe nodes kind-control-plane

    more info on the ouput here

  • See the proxies if they are under the API object named DaemonSet

    % k get daemonSets --namespace=kube-system kube-proxy

  • See DNS k8s deployment

    % k get deployments --namespace=kube-system coredns

  • Wait for nodes to be ready

    % k wait --for=condition=Ready nodes --all
node/node1 condition met
node/node2 condition met
...

Manage your context

  • Change namespace for current context

    % kubectl config set-context --current --namespace=<some ns>

āŽˆ helm

āš ļø These are notes taken from the book Learning Helm by Matt Butcher, Matt Farina and Josh Dolitsky on O'Reilly Please go read it and just use this as reference.

Introduction

Helm is the package manager for k8s. Let us do a super quick dive into what k8s is an all that just to give the context necessary to understand helm.

Context for Helm (containers & k8s)

Sometime ago someone thought of a new way of making apps, instead of them being a big monolithic service, we could split everything into small discrete standalone services. Then we could join those services using API over the network and build the application that way. That is how microservices were born.

Containers

To help with this the concept of containers was introduced, it is different of a VM because the VM runs an entire OS on on a host machine. On the other hand, a container has its own file system, but it uses the same OS kernel as the host.

A container is a program together with its dependencies and environment, they are packaged together into a container image which can be build based on a file where you specify the packages you want to run, and how to set the environments. The cool part is that all those instructions are not compiled into a binary or anything, but they are packaged into discrete layers.

So if a host has an image with five layers, and there is another host that needs the same image, it will just fetch the layers it does not already have.

Say you have three images using fedora-minimal:37, well it would reuse that layer on all those three.

Those images are stored in a registry, the registry tells hosts which layers compose an image. So the host can only download the layers it need from the registry.

A registry identifies an image by three things:

  • name: basically a string fedora or fedora-minimal
  • tag: usually the version latest or v1
  • digest: a hash of the image, since the tags are mutable.

They look like name:tag@digest.

Kubernetes (or k8s like the cool kids call it)

So with all these container stuff, some questions begin to arise:

  • How do we best execute lots of containers?
  • How can they work together?
  • How do we manage memory, cpu, network and storage?

Well a bunch of companies tried to create this orchestration of containers technology to answer those questions, at the end as per 2024 seems like Google won, and we now all use ā›µļø K8s.(which is greek for ship's capitan or something like that)

K8s won because it introduced two concepts that people really liked.

  • Declarative infrastructure Basically you tell k8s what your desired state of the cluster is, and it will work to make that happen.

  • Reconciliation loop How does k8s work behinds the scenes to reach the declarative configuration we set? Using a reconciliation loop.

    In a reconciliation loop, the scheduler says: "Here is what the user wrote as his/her desired state. Here is the current state. They are not the same, lets reconcile them."

    Say you specified storage, and the scheduler sees that we do not have storage yet, it will create units of storage and attach them to your containers.

Pods

We do not deal directly with containers when setting up our k8s cluster. We use a pods. A pod which is basically a group of containers. These are defined in a manifest (yaml or json, but most people use yaml)

apiVersion: v1 # we can see that this will be a v1 Pod
kind: Pod
metadata:
    name: example-pod
spec:
    containers:
    - image: "fedora-minimal:latest"
    - name: example-fedora

A Pod can have 1 or more containers. The containers that help preconfigurate stuff for the main one, are called init containers.

The ones that run alongside the main container are called sidecar containers

ConfigMaps

Basically in a pod you describe the configuration the containers need, like network ports files system mount points. You can store configuration information for k8s in ConfigMaps and password and stuff like that in Secret.

Here is an example:

apiVersion: v1
kind: ConfigMap
metadata:
    name: configuration-data
data: # inside here we will declare aribitrary key/value stuff
    backgroundColor: blue
    title: Learning Helm

For Secret is pretty much the same as for ConfigMaps but the values in data must be Base64 encoded.

Pods then can be linked to the ConfigMaps like this:

apiVersion: v1
kind: Pod
metadata:
    name: example-pod
spec:
    volumes:
    # note that this is not the metadata name you used in the ConfigMap
    - name: my-configuration
      configMap:
        name: configuration-data # same name as in metadata of ConfigMap
    containers:
    - image: "fedora-minimal:latest"
      name: example-fedora
      env:
        - name: TITLE # the env variable will be title
          valueFrom:
            configMapKeyRef:
                name: configuration-data # name for the volume
                key: title # key in the actual ConfigMap

Deployments

Pretty cool stuff, but we might not just one to run one instance of our container, therefore we can use something called Deployments.

A Deployment describe an application as a collection of identical pods,we can tell k8s to create our app with a single pod and scale it up to five pods.

apiVersion: apps/v1 # apps/v1 Deployment
kind: Deployment
metadata:
    name: example-deployment
    labels:
        app: my-deployment
spec:
    replicas: 3 # we want three replicas of the following template
    selector:
        matchLabels:
            app: my-deployment
    template:
        metadata:
            labels:
                app: my-deployment
        spec:
           containers :
            -image: "fedora-minimal"
             name: "example-fedora"

Service

A Service is a persistent network resource, that persists even if the pod or pods attached to it go away.

apiVersion: v1
kind: Service
metadata:
    name: example-service
spec:
    selector:
      app: my-deployment
    ports:
      -  protocol: TCP
         port: 80
         targetPort: 8080

Here we define a service for the pods with the app: my-deployment, telling that traffic port 80 of this Service will be routed to 8080.

Helm's Goal

So we have seen the building block of K8s, say you want to deploy a WordPress app, you would need a Deployment for the containers, then a ConfigMap and Secret for the password, maybe some Service objects and so on. That sounds like a lot of yaml.

The core idea of Helm is that all those objects can be packaged to be installed, updated and deleted together.

Basically helm is the package manger of kuberentes. Helm will allow you to spin up all the k8s objects necessary for an app by just using a few commands.

Here are some of the things that helm can do:

  • Provide package repos (similar to dnf or apt).
  • Familiar install, upgrade, delete commands.
  • Helm has a method for configuring the packages before installing them.
  • Helm can also let you know what already is installed.

You can also limit the installation of packages to a specific namespace, so you can install the same package in different namespaces in the same k8s cluster. (What is a namespace?)

Helm also provides reusability, it uses charts for this. A chart provides a pattern for producing the same k8s manifests. But the cool part is that you can also add more configuration to those charts.

Helm provides patterns for storing the initial configuration plus the changes you did. Helm encourages k8s users to package their yaml into charts so that these descriptors can be reused.

One last thing, keep in mind that helm is not a configuration tool, it helps but it there are software specialized on that like ansible, puppet or chef.

Helm Architecture

Here are the main components that helm uses.

K8s Resource/Object

These are the Pods, ConfigMap, Deployment, we have seen throughout the chapter. K8s has a lot of these objects, you can even define custom ones using custom resource definition (CRD).

All the resources share some elements

apiVersion: apps/v1 # api and version of the resource
kind: Deployment # resource type
metadata: # top-level data on the resource/object
    name: example-deployment # req for all objects
    labels: # used for creating query-able handles
        some-name: some-value
    annotations: # authors to attach their own keys and values
        some-name: some-value

Charts

A package is called a chart. The idea es that k8s meaning captian, helm being the steering mechanism of the ship. The chart plots the way k8s apps should be installed.

A Chart is a set of files and directories that describe how to install the different k8s resource/objects

A chart contains

  • Chart.yaml: describes the chart (name, description, authors)
  • templates directory: Inside all the k8s manifests potentially annotated with templating directives
  • values.yaml file that provides the default configuration. You can override during installation.

These are the basic stuff for an unpacked chart, a packed chart is just a tar ball with all this inside.

Resources, Installations and Releases

When a helm chart is installed:

  1. helm reads the charts (will download if necessary)
  2. It sends the values into the templates generating the k8s manifests
  3. The manifests are sent to k8s
  4. K8s creates the requested resources inside the cluster

One last concept release. A release is created each time we use helm to modify the installation

Using Helm

Helm is a cli, you can install it with your favorite package manger or build it from source it is written in golang.

You can check the version by

helm version

Helm will use the same KUBECONFIG file you have configured for kubectl, it will look in the same places, though you can specify a path for one.

The most common workflow is:

  1. Add a chart repo
  2. Find a chart to install
  3. Install a Helm chart
  4. See the list of what is installed
  5. Upgrade your installation
  6. Delete your installation

Adding a Chart Repo

A Helm Chart is an individual package that can be installed into your k8s cluster. You can find then at chart repositories.

You can find popular repositories in the artifact hub

By default helm does not have any repo added, so you need to look for one there.

Bitnamis's official Helm charts are one of the best well-curated charts repos. (Some Bitnami devs are among the core contributors who design the helm repo system)

To add a repo you do helm repo add so:

% helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

Now if you do

% helm repo list
NAME    URL
bitnami https://charts.bitnami.com/bitnami

You will see it there. After that we can look for specific charts.

% helm search repo drupal
NAME            CHART VERSION   APP VERSION     DESCRIPTION
bitnami/drupal  18.0.2          10.2.5          Drupal is one of the most versatile open source...

You can also search label and descriptions

% helm search repo content
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/drupal          18.0.2          10.2.5          Drupal is one of the most versatile open source...
bitnami/harbor          21.1.2          2.10.2          Harbor is an open source trusted cloud-native r...
bitnami/nginx           16.0.6          1.25.5          NGINX Open Source is a web server that can be a...
bitnami/owncloud        12.2.11         10.11.0         DEPRECATED ownCloud is an open source content c...
bitnami/wordpress       22.2.2          6.5.2           WordPress is the world's most popular blogging ...

The chart version is the version of well, the chart. On the other hand the app version is the version of the software it would install.

Installing a package

% helm install mysite bitnami/drupal
NAME: mysite
LAST DEPLOYED: Wed May  1 16:40:16 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
# more text

Typical stuff, just a few things to keep in mind.

  • Difference between an installation and a chart.

    An installation of a chart is a specific instance of the chart One chart may have many installations

  • You can repeat instance names (mysite), but it must be on different namespaces.

You can set values specific to your installation, you can set them directly from the command line with --set for example. (this works with both set install and upgrade)

% helm install mysite bitnami/drupal --set drupalUsername=admin

You can also have them in a yaml file, which is the recommended approach.

% helm upgrade mysite bitnami/drupal --values values.yaml

For example values.yaml would look like this:

drupalUsername: admin
drupalEmail: [email protected]
mariadb:
    db:
        name: "my-database"

Listing your Installations

% helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
mysite  default         1               2024-05-01 16:40:16.50636 -0600 CST     deployed        drupal-18.0.2   10.2.5

Not much to say here, does what expected.

Upgrading an Installation

So there are two types of changes:

  • upgrade version of the chart
  • upgrade configuration of the chart

Every time we perform an upgrade we are doing a new release of the same installation.

Helm will attempt to to alter only the bare minimum, so if you only changes one simple configuration variable, it will not like restart everything and all that.

To restart stuff just use kubectl.

To update your chart with a new version you can

% helm repo update
% helm upgrade mysite bitnami/drupal

Configuration Values

If you do this:

% helm install mysite bitnami/drupal --values values.yaml
% helm upgrade mysite bitnami/drupal

Chances are you lost your values, so it is good to always send the yaml

% helm install mysite bitnami/drupal --values values.yaml
% helm upgrade mysite bitnami/drupal --values values.yaml

You can use helm get values mysite to see the values sent on the last install or upgrade.

You could also use:

% helm upgrade mysite bitnami/drupal --reuse-values

But it is not recommended

Uninstalling charts

Not much to say here

% helm uninstall mysite

Works as you would expect, if you would do

% helm list

You will not see it there.

Lastly, you can see a special record that contain release information.

% k get secret

Helm stores there the info.

If you uninstall you loose the history, so be careful. You could helm uninstall --keep-history. Good if you plan on doing helm rollback.

That is pretty much the basics on helm.

source

What is Fluentd

It is a event/data collection tool. We can have multiple applications with tons of different logs formats. Fluentd tries to unify them.

It is extensible, the core program is small, and does only the main stuff.

  • Divide & Conquer
  • Buffering & Retries
  • Error Handling
  • Message Routing
  • Parallelism

It delegate the rest to the users via plugins, reading data, parsing data, buffering data, write data.

It is reliable, it is important to not loose data. Fluentd tries to move the data in small pieces and checks if the data passed successfully if not it retries.

Fluentd can tag the data and route it according to the label.

Use Cases

Simple Forwarding

You have some simple files that and logs from a port and want to forward them to mongodb. Here is a simple configuration example.

logs from a file

<source>
    type tail
    path /var/log/httpd.log 
    format apache2
    tag web.access
</source>

logs from client libraries

<source>
    type forward 
    port 24224
</source>

store logs to ES and HDFS

<source backend.* >
    type mongo 
    database fluent
    collection test
</source>

Lambda Architecture

šŸ“ˆ Elastic Stack (ELK)

Table of contents

ELK is great for Centralized Logging, this enable us to not waste time trying to find where the issues are, we have all in one place. We also can

Kibana is the visualization of the ELK stack, you can visualize the data on real time.

Components

It has several open source add-ons, but the main ones are these:

  • Beats:
    • Light weight way of getting data into the ELK stack.
    • Single purpose tools.
  • Logstash:
    • Transform data as it comes into structured data that will be stored.
    • Very resource intensive
    • Example: is like cutting carrots šŸ„• as you get from the store and saving them to the fridge, instead of cutting them until you are about to cook
  • Elasticsearch:
    • The part of the stack that stores the data itself
    • Usually deployed by a cluster, it has redundant copies so it can work if something fails
  • Kibana:
    • Web front end for the ELK stack
    • Quick birdā€™s eye view across your infra

Logstash

Three block of code in config

  • Input: get data in
  • Output put data somewhere, usually elasticsearch
  • Filter block: where data is transformed and relocated, where we can parse lines.

Example of config file.

input { stdin {} }

filter {
	mutate {
		add_field => { "greeting" => "Hello %{message}" }
	}
}

output { stdout {} }

This will take input from STDIN and output it to STDOUT, anything we type would be returned in a structured format with a greeting field.

Logstash Plugins

It has more than 100 packages installed with logstash

/usr/share/logstash/bin/logstash-plugins list for listing them

  • Beats: Input plugin to work with beats

    input { beats { port => 5044 } }

  • File:

    • Get input from a file, or multiple, you can use wildcards
    input { file { path=> "/var/log/kern.log" } }

  • elasticsearch

    • this is an output plugin, you can point to multiple servers
    output { elasticsearch { host => ["localhost:9200"] } }

Filters

  • Grok

    • convert unstructured data to structured data
    • based on regex, but it has many abstractions in case you want to use them
    filter {
	grok {
		match => {
			"message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM}:request...."
		}
	}
}

  • Mutate

    • let you transform the data
    filter {
	grok {
		match => {
			"message" => "%{TIMESTAMP:t} new users: %{GREEDYDATA:users}"
			# 2024-01-23 new users: jose,pedro,maria
		}
	}

	mutate {
		split => {"users" => ","}
	}
}

    In this example grok will get the time stamp of the message and will get all the string after new users: then we use mutate to split the users and have it stored as a list so when the data shows up in elasticsearch it looks like an array

GrokConstructor

grokconstructor.appspot.com Helper for creating grok filters.

You can see also some grok patterns too.

Basically you can paste the logs there build your filter from there

grok.png

Beats

Why use beats? Logstash is overkill for a lot of things people use it for.

It uses JRuby (basically ruby that runs on the JVM), and by default it is configured to set aside 1GiB of memory for the JVM. That is too much to just send things to Elastic

Beats (written in Go) is way more šŸŖ½lightweight, it has been design to be. Becuase of this can be run everywhere.

There are several beats

  • Heartbeat (Uptime)
  • Filebeat (Logs and text data)
  • Metricbeat (Metrics)
  • Packetbeat (Network data)
  • Winlogbeat (Windows events logs)
  • Auditbeat (audit logs and data)
  • Functionbeat (serverless data)