Background:
First, With the rush of releasing Binge, endless zoom meetings and editing all those k8s yaml files, I have been itching to write a for
loop (ie..do some coding) .
Second, I am tired of parsing/grepping/awking those kubectl output but I still had to go to my newrelic UI to see which pod is running on my EKS spot vs ondemand nodegroup and I am a commandline kinda guy and its too difficult to do in commandline.
So I decided to write a small tool just for myself, but didn’t feel like doing it in groovy which is my default scripting language (hey I come from java!). A few month ago, I picked up a GO book but realized it will take me more than a weekend to be productive in it and frankly I didn’t like the syntax. I like my language to be more abstract (again, I come from java!), so I picked up a python cheatsheet and started hacking.
I must admit its a great stress reliever when you are trying to deploy ~1000 pods running ~100 microservices in a fresh new EKS cluster for the first time in your organization and the whole internet is there to curse you if the platform is not delivering ๐ (thats a different story for another day)
Result:
I hacked together a few lines of python code (again, my python skills are those coming from a cheatsheet and random googling over a weekend so no judging). But the output kk (kubectl++) came out to be quite handy and I find myself using it very often so I though I’d put it in my github.
https://github.com/sajid-moinuddin/kk
The idea behind is simple, you get all node info, all pod info and programatically join them and create a virtual PodNode
Resource (just like any k8s resource). Now you can do things like:
#get me pod,namespace,nodename,spot/ondemand,pod-resources,restartcount from namespace x/y/z and by the way, exclude the daemonsets and format nicely as I say
kk get podnode
-f 'metadata.namespace=streamtech/content/commerce'
-e 'metadata.owner_references[0].kind=DaemonSet'
-o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:35,pod.spec.containers[0].resources.limits:35,pod.status.container_statuses[0].restart_count'
offers-api Running spot commerce ip-10-100-58-11 {'cpu': '2', 'memory': '4Gi'} {'cpu': '6', 'memory': '4Gi'} 0
offers-api Running spot commerce ip-10-100-62-11 {'cpu': '2', 'memory': '4Gi'} {'cpu': '6', 'memory': '4Gi'} 0
offers-api Running spot commerce ip-10-100-63-3. {'cpu': '2', 'memory': '4Gi'} {'cpu': '6', 'memory': '4Gi'} 0
offers-api Running ondemand commerce ip-10-100-60-74 {'cpu': '2', 'memory': '4Gi'} {'cpu': '6', 'memory': '4Gi'} 0
(venv) โ kk git:(master) kk -hk(kubectl)++ ... only better
Usage:
kk get (podnode|pn) [options]
Options:
-h --help show this
-l STRING label selector
-f STRING field selector
-e STRING exclude
-o output pydash format plus padding info, ie. node.metadata.name:62
--offline do not fetch new data, work on the last fetched data
--json print the -o elements in json format
-w watch mode
Example:
#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:35,pod.spec.containers[0].resources.limits:35,pod.status.container_statuses[0].restart_count' | sort -k3
#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:30,pod.spec.containers[0].resources.limits:30'
#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name:62,node.metadata.labels.lifecycle,namespace,node.metadata.name:62' -w
#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name,node.metadata.labels.lifecycle,namespace,node.metadata' --json
I have used the python-builder library which looked like maven. I am yet to spend any time to understand how to publish this as a pip module (I havent spend much time understanding python modules and packages yet…)
NOTE:
>> this can be a very expensive operation on your kubectl api so use this with care in prod environment.
>> It also has a –offline mode (this works on the last fetched data) which can be used to first get a dump of the data and look at them at different angel
>> You can also use a –json format that will dump the raw jason