I was watching the above video and it triggered some thinking:

Back in 2019 Atlassian coined the term “Brilliant Jerks”, and as always rest of the industry overbought it :), very similarly when google did their publication of “5 Keys of Successful Google Team“, the only thing that people remembered from that is “Psychological Safety”, the very next item on it is “Dependability: Can we count on each other to do high-quality work on time?” but “Dependability,Quality” is not talked about as often as “Psychological Safety“. Again very similar to how people cheering / blindly copying the Spotify Model, it never actually worked for them as the rest of the world thinks: https://www.jeremiahlee.com/posts/failed-squad-goals/

In this era of Political Correctness and Instagram Celebs, Optics seems to matter most and it can get tricky for “Code wins argument” , if you were the boss of Linus Torvalds, you would have hard time keeping him in the job. If you don’t believe me, here is some of the sample: 

Diplomatic Programmer

And

Linus Torvalds on his insults

Imagine defending him in an HR meeting 😀

Ok, don’t get me wrong, in no way I am approving this behavior and suggesting we should all follow his footsteps (there is only one Linus), but my point is, it can be easy to swing on another side of extremes, where strongly opinionated people get voted as “Brilliant Jerks” and more mediocre play it nice people win the argument gets promoted.

I think as managers/leaders, it is most critical now than ever in the industry to protect/grow our so called “brilliant jerks” (u dont need to be brilliant to be a jerk) and train them to win the code argument, otherwise, we will end up in a mediocre group who just pat each other on the back and play nice. Many a time I have seen a good engineer with a good argument not being able to sell the idea or being sidelined just because he/she bought too much into the “Code wins Argument”. Code / Idea don’t just sell itself and it is up to us engineers to play salesman and defender of the “Code wins Argument”. And us managers should promote that environment where people can start an argument and still feel psychologically safe without being concerned about making someone else psychologically unsafe. At the end, its just code! Like they say in the linux community, talk is cheap, show me the code! Now more than ever, its important for the technical people to rise up and get into decision making process and decision makers to get educated and understand technology and understand the merit / difference of good vs bad argument and not just rely on optics.

I feel psychologically unsafe when there is no debate in the room and everyone playing nice 🙂 , for me most rewarding experience is having a strong argument and being overruled by another stronger argument or vice versa.

Recently I had a lot of fun writing: HPATuner

First My GO skills are very naive but I wouldn’t allow GO to stop me from writing a controller specially when its made so easy by KubeBuilder. Also it does promise to solve a lot of the issues we have been trying to solve over the years (see the github README for more details).

But the most fun was when I tried to add decent functional test coverage and add proper CI/CD process. This was specially needed because we are not a GO team and people trying to maintain it should have enough confidence making future changes.

• I learned to use Kind to write functional integration tests. Up until now I always treated as a play thing not realizing how powerful it can be.
• I learned how powerful Docker-In-Docker in a kubernetes pod to build/test K8s components
• Once you get comfortable dealing with k8s api and controller constructs, the possibilities are endless! I am already thinking some other custom controller to solve some cross cutting concerns we have across PODs. You haven’t truly appreciated powers of k8s until you have written a controller for it. I love k8s even more now 🙂 (lol if thats possible)
• Last but not least, I learned my initial impression of GO being a weired language is still true 🙂 , but how powerful it is doing functional coding in a staticically typed language. I have switched to groovy/python because I didn’t like Java’s awkward functional constructs but I always missed the static typing. I could get used to GO once i get past my mental barrier of dealing with pointers. The fact that its natively compiled is also quite impressive.

Lastly we are not a GO Shop so couldn’t find anyone yet to do a proper code review for the project. Anyone in k8s/GO world in my network want to spend some time giving some feedback/pointers for improvement will be much appreciated. In return this hpatuner can be a very powerful adition to your k8s platform!

After coming out of a month vacation where I stayed min 200 feet away from any computer/slack/outlook/chrome, I couldn’t speak highly enough about Dr. Shimi Kang’s Ted Talk:

…..then again I needed science(a ted talk from a PHD) to take a break 🙂

15 years ago, just one year into my 1st job, I submitted my resignation to work on a new startup. One of my seniors was very surprised and asked me why! I was super excited to work on VOIP/SIP and explained that to him (little did I know I’d spend next 7 years of my career debugging UDP pakets in etherial, yes wireshark was called etherial back then). My senior smiled and replied, “you are still young, after a while you will realize everything is ultimately just input/output”

15 years down in my career, while I dont fully agree with that, I do appreciate where he was coming from. The other day I had 4 session in 2 day with 4 different teams who has very different domain from each other:

• 1st one was how to collect and process and analyze millions of set-top-box sending usage data
• 2nd session was about the next step design of a “metadata-hub” for efficiently storing and distributing video metadata for all the video platforms we have.
• 3rd was about collecting video click data and feeding it into the recommendation system and using the recommendations on the web.
• 4th was about a jenkins pipeline and building a jenkins helm chart itself(egg vs chicken)

The whole time (actually starting from 2nd meeting), I couldn’t stop thinking about that quote “everything is eventually input/output”.

Lol I am not sure whether to take this mental state of mine positively or negatively. Have I reached some new architect high where I am seeing the superior abstraction in everything or I got too old and boring and every problem looks same to me 🙂

Trying to look a bit deeper, I realized, with all the OSS frameworks/libraries and aws building blocks(or any public cloud offering) our problem space shifted a lot. All we do these days is glue different services / tools together to deliver business value. And who does the glueing? Your Pipelines !!! Its all about the Pipelines!!! Just looking at the small team we have, we got so many pipeline execution framework running in production at this moment: Conductor, AirFlow, AWS Steps, Jenkins-X, Argo (kubeflow pipelines), Activiti (I know too many!!!, but its about right tool for right job 🙂 )

So yes, if I had to rephrase my senior 15 years later, I’d say, Its all Input/Output And the pipelines in between.

or may be I am missing my SIP/RFC3261 days too much!!!

Again, me trying to learn some python (because I want to talk to my Anki Vektor) and make something useful so I can use day to day. I must say, this code has been very instrumental in me keeping an eye on every k8s events and find some gotchas from them when we went live in large scale EKS production platform.

https://github.com/sajid-moinuddin/watchman

PS: as a java developer, once you go past the odd syntax of passing around self , its probably the easiest language to pickup

Background:

First, With the rush of releasing Binge, endless zoom meetings and editing all those k8s yaml files, I have been itching to write a for loop (ie..do some coding) .

Second, I am tired of parsing/grepping/awking those kubectl output but I still had to go to my newrelic UI to see which pod is running on my EKS spot vs ondemand nodegroup and I am a commandline kinda guy and its too difficult to do in commandline.

So I decided to write a small tool just for myself, but didn’t feel like doing it in groovy which is my default scripting language (hey I come from java!). A few month ago, I picked up a GO book but realized it will take me more than a weekend to be productive in it and frankly I didn’t like the syntax. I like my language to be more abstract (again, I come from java!), so I picked up a python cheatsheet and started hacking.

I must admit its a great stress reliever when you are trying to deploy ~1000 pods running ~100 microservices in a fresh new EKS cluster for the first time in your organization and the whole internet is there to curse you if the platform is not delivering 🙂 (thats a different story for another day)

Result:

I hacked together a few lines of python code (again, my python skills are those coming from a cheatsheet and random googling over a weekend so no judging). But the output kk (kubectl++) came out to be quite handy and I find myself using it very often so I though I’d put it in my github.

https://github.com/sajid-moinuddin/kk

The idea behind is simple, you get all node info, all pod info and programatically join them and create a virtual PodNode Resource (just like any k8s resource). Now you can do things like:

#get me pod,namespace,nodename,spot/ondemand,pod-resources,restartcount from namespace x/y/z and by the way, exclude the daemonsets and format nicely as I say 

kk get podnode  
-f 'metadata.namespace=streamtech/content/commerce' 
-e 'metadata.owner_references[0].kind=DaemonSet' 
-o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:35,pod.spec.containers[0].resources.limits:35,pod.status.container_statuses[0].restart_count'

offers-api            Running             spot                commerce            ip-10-100-58-11    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   
offers-api            Running             spot                commerce            ip-10-100-62-11    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   
offers-api            Running             spot                commerce            ip-10-100-63-3.    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   
offers-api            Running             ondemand            commerce            ip-10-100-60-74    {'cpu': '2', 'memory': '4Gi'}          {'cpu': '6', 'memory': '4Gi'}          0                   

(venv) ➜  kk git:(master) kk -hk(kubectl)++ ... only better
Usage:
kk get (podnode|pn) [options]

Options:
    -h --help    show this
    -l STRING    label selector
    -f STRING    field selector
    -e STRING    exclude
    -o output    pydash format plus padding info, ie. node.metadata.name:62
    --offline    do not fetch new data, work on the last fetched data
    --json       print the -o elements in json format
    -w           watch mode

Example:
#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:35,pod.spec.containers[0].resources.limits:35,pod.status.container_statuses[0].restart_count' | sort -k3

#kk get podnode -f 'metadata.namespace=streamtech/content/commerce' -e 'metadata.owner_references[0].kind=DaemonSet' -o 'pod_name:62,pod.status.phase,node.metadata.labels.lifecycle,namespace,node.metadata.name:15,pod.spec.containers[0].resources.requests:30,pod.spec.containers[0].resources.limits:30'

#kk get podnode -f 'metadata.namespace=streamtech/content/commerce'  -e 'metadata.owner_references[0].kind=DaemonSet'   -o 'pod_name:62,node.metadata.labels.lifecycle,namespace,node.metadata.name:62' -w

#kk get podnode -f 'metadata.namespace=streamtech/content/commerce'  -e 'metadata.owner_references[0].kind=DaemonSet'   -o 'pod_name,node.metadata.labels.lifecycle,namespace,node.metadata' --json

I have used the python-builder library which looked like maven. I am yet to spend any time to understand how to publish this as a pip module (I havent spend much time understanding python modules and packages yet…)

NOTE:
>> this can be a very expensive operation on your kubectl api so use this with care in prod environment.
>> It also has a –offline mode (this works on the last fetched data) which can be used to first get a dump of the data and look at them at different angel
>> You can also use a –json format that will dump the raw jason

And I just finished Binging 4 Seasons of  Halt & Catch Fire in last 3 weeks: …..In my defence I was contributing to the testing effort 🙂

https://binge.com.au/shows/show-halt-and-catch-fire!1716

IOC is an age old concept we are so familiar with via spring framework. This enables us to invert the flow of control and avoid tight coupling between the modules of the system.

Conductor is a Microservice Orchestration Framework from Netflix. It helps us to avoid Choreography in microservice. If you are not familiar with Orchestration vs Choreography concepts, I highly encourage you to pickup a copy of Building Microservice , or have a quick read:

https://stackoverflow.com/questions/4127241/orchestration-vs-choreography 

Essentially Conductor is a IOC framework for your microservices. You take the flow of control out of your microservices and push it in the “Conductor WorkFlow“.

NOTE: there are other frameworks implementing similar concepts : https://github.com/ing-bank/baker
https://camunda.com/products/bpmn-engine/ 

Now Imagine you have Multiple Microservices Deployed in Cloud and OnPrem Datacentre and a flow as below:

Only problem is, you don’t have any direct internet connection from the CLOUD to your onprem hosted Service B . Why you have that service B onprem you might ask? May be its talking to an onprem legacy database, or a million $ video transcoder that you can’t export to the cloud. (Onprem datacentres are here to stay, stop dreaming!!!)

This is why IOC comes in play, The famous Hollywood principle: “Don’t Call Me, I’ll Call You” at its finest! Instead of the FLOW calling  Service B, Service B can call the FLOW for its Designated Task. Having something like Conductor which enables you to invert this dependency is priceless (otherwise you’d be filling forms and chasing people for months trying to open up corporate firewall to the internet) :

NOTE: The task queue is logical component here, there is no kafka/activemq , Conductor uses a simple Redis / Mysql Database to maintain the Queue and provides nice REST api to GET/PUT tasks. You can compare this to event driven architecture/pub sub model with IOC in place (think Polling !). You achieve similar benefit but without the complexity of having a message BUS.

 

People keep talking about k8s, but the story is really about CI/CD, k8s is just the enabler! Took me about 3 years to finally appreciate this. Without the modern CI/CD for your k8s cluster, you are just driving the plane on the road and thinking you are flying.