Background:

First, With the rush of releasing Binge, endless zoom meetings and editing all those k8s yaml files, I have been itching to write a for loop (ie..do some coding) .

Second, I am tired of parsing/grepping/awking those kubectl output but I still had to go to my newrelic UI to see which pod is running on my EKS spot vs ondemand nodegroup and I am a commandline kinda guy and its too difficult to do in commandline.

So I decided to write a small tool just for myself, but didn’t feel like doing it in groovy which is my default scripting language (hey I come from java!). A few month ago, I picked up a GO book but realized it will take me more than a weekend to be productive in it and frankly I didn’t like the syntax. I like my language to be more abstract (again, I come from java!), so I picked up a python cheatsheet and started hacking.

I must admit its a great stress reliever when you are trying to deploy ~1000 pods running ~100 microservices in a fresh new EKS cluster for the first time in your organization and the whole internet is there to curse you if the platform is not delivering 🙂 (thats a different story for another day)

Result:

I hacked together a few lines of python code (again, my python skills are those coming from a cheatsheet and random googling over a weekend so no judging). But the output kk (kubectl++) came out to be quite handy and I find myself using it very often so I though I’d put it in my github.

https://github.com/sajid-moinuddin/kk

The idea behind is simple, you get all node info, all pod info and programatically join them and create a virtual PodNode Resource (just like any k8s resource). Now you can do things like:

#get me pod,namespace,nodename,spot/ondemand,pod-resources,restartcount from namespace x/y/z and by the way, exclude the daemonsets and format nicely as I say 

kk get podnode  
-f 'metadata.namespace=streamtech/content/commerce' 
-e 'metadata.owner_references[0].kind=DaemonSet' 
-o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:35,pod.spec.containers[0].resources.limits:35,pod.status.container_statuses[0].restart_count'

offers-api            Running             spot                commerce            ip-10-100-58-11    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   
offers-api            Running             spot                commerce            ip-10-100-62-11    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   
offers-api            Running             spot                commerce            ip-10-100-63-3.    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   
offers-api            Running             ondemand            commerce            ip-10-100-60-74    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   

(venv) ➜  kk git:(master) kk -hk(kubectl)++ ... only better
Usage:
kk get (podnode|pn) [options]

Options:
    -h --help    show this
    -l STRING    label selector
    -f STRING    field selector
    -e STRING    exclude
    -o output    pydash format plus padding info, ie. node.metadata.name:62
    --offline    do not fetch new data, work on the last fetched data
    --json       print the -o elements in json format
    -w           watch mode

Example:
#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:35,pod.spec.containers[0].resources.limits:35,pod.status.container_statuses[0].restart_count' | sort -k3

#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:30,pod.spec.containers[0].resources.limits:30'

#kk get podnode -f 'metadata.namespace=streamtech/content/commerce'  -e 'metadata.owner_references[0].kind=DaemonSet'   -o 'pod_name:62,node.metadata.labels.lifecycle,namespace,node.metadata.name:62' -w

#kk get podnode -f 'metadata.namespace=streamtech/content/commerce'  -e 'metadata.owner_references[0].kind=DaemonSet'   -o 'pod_name,node.metadata.labels.lifecycle,namespace,node.metadata' --json

I have used the python-builder library which looked like maven. I am yet to spend any time to understand how to publish this as a pip module (I havent spend much time understanding python modules and packages yet…)

NOTE:
>> this can be a very expensive operation on your kubectl api so use this with care in prod environment.
>> It also has a –offline mode (this works on the last fetched data) which can be used to first get a dump of the data and look at them at different angel
>> You can also use a –json format that will dump the raw jason

And I just finished Binging 4 Seasons of  Halt & Catch Fire in last 3 weeks: …..In my defence I was contributing to the testing effort 🙂

https://binge.com.au/shows/show-halt-and-catch-fire!1716

IOC is an age old concept we are so familiar with via spring framework. This enables us to invert the flow of control and avoid tight coupling between the modules of the system.

Conductor is a Microservice Orchestration Framework from Netflix. It helps us to avoid Choreography in microservice. If you are not familiar with Orchestration vs Choreography concepts, I highly encourage you to pickup a copy of Building Microservice , or have a quick read:

https://stackoverflow.com/questions/4127241/orchestration-vs-choreography 

Essentially Conductor is a IOC framework for your microservices. You take the flow of control out of your microservices and push it in the “Conductor WorkFlow“.

NOTE: there are other frameworks implementing similar concepts : https://github.com/ing-bank/baker
https://camunda.com/products/bpmn-engine/ 

Now Imagine you have Multiple Microservices Deployed in Cloud and OnPrem Datacentre and a flow as below:

Only problem is, you don’t have any direct internet connection from the CLOUD to your onprem hosted Service B . Why you have that service B onprem you might ask? May be its talking to an onprem legacy database, or a million $ video transcoder that you can’t export to the cloud. (Onprem datacentres are here to stay, stop dreaming!!!)

This is why IOC comes in play, The famous Hollywood principle: “Don’t Call Me, I’ll Call You” at its finest! Instead of the FLOW calling  Service B, Service B can call the FLOW for its Designated Task. Having something like Conductor which enables you to invert this dependency is priceless (otherwise you’d be filling forms and chasing people for months trying to open up corporate firewall to the internet) :

NOTE: The task queue is logical component here, there is no kafka/activemq , Conductor uses a simple Redis / Mysql Database to maintain the Queue and provides nice REST api to GET/PUT tasks. You can compare this to event driven architecture/pub sub model with IOC in place (think Polling !). You achieve similar benefit but without the complexity of having a message BUS.

People keep talking about k8s, but the story is really about CI/CD, k8s is just the enabler! Took me about 3 years to finally appreciate this. Without the modern CI/CD for your k8s cluster, you are just driving the plane on the road and thinking you are flying.

Its so hilarious and painfully true it took me a while to realize I am reading a technical book!

“Improving daily work is even more important than doing daily work.”
― Gene Kim

One hundred years from now, historians will look back at this decade and conclude that something transformative happened: how we worked inside Development and IT Operations changed completely.… I predict that historians will call this decade the “Cambrian explosion for IT,” a time of incredible innovation and disruption when, 50 years after the birth of computing, we finally learned what technology was good for.
—John Willis, Cohost of “DevOps Cafe”