Saturday, July 7, 2018

Creating an admin account on Kubernetes

I spent a bunch of time Googling how to do this so I figured it could help someone else if posted the steps to add an admin account on a Kubernetes cluster managed with kops.

k8s has service accounts but that's not what you want to create an admin account — equivalent to having root privileges on the cluster.  Instead you simply need to create a certificate/key pair for the user and sign it with the master's CA (certificate authority).

In this example we'll create an account for user foobar.
  1. Create a private key:
    openssl genrsa -out foobar.key 2048
    For extra security you can also opt for 4096 bits for the key but for some reason kops defaults to 2048 right now.
  2. Create a CSR (Certificate Signing Request)
    openssl req -new -key foobar.key -out foobar.csr -subj '/CN=foobar/O=system:masters'
    The CN (Common Name) contains the user name and the O (Organization Name) must be system:masters to be a super-user.
  3. Fetch the master's private key from S3, from the bucket kops was configured to use:
    aws s3 sync $KOPS_STATE_STORE/$NAME/pki pki
    Here the variables $KOPS_STATE_STORE and $NAME are the ones referred to in the kops documentation. For example:
    aws s3 sync s3://prefix-example-com-state-store/ pki
    All the PKI files will be downloaded from S3 into the local pki directory.
  4. Issue the certificate using the master's CA: openssl x509 -req -in foobar.csr -CA pki/issued/ca/*.crt -CAkey pki/private/ca/*.key -CAcreateserial -out foobar.crt -days 3650
At this point you could give the private key (foobar.key) and the certificate (foobar.crt) to the user, but if you want to be a bit nicer and generate a self-contained kubectl config for them, here's how:
kubectl --kubeconfig=kcfg config \
  set-credentials $NAME --client-key=foobar.key --client-certificate=foobar.crt --embed-certs=true
kubectl --kubeconfig=kcfg config \
  set-cluster $NAME --embed-certs=true --server= --certificate-authority pki/issued/ca/*.crt
kubectl --kubeconfig=kcfg config \
  set-context $NAME --cluster=$NAME --user=$NAME
kubectl --kubeconfig=kcfg config \
  use-context $NAME
You can then hand over the kcfg file to the user and they could use it directly as their ~/.kube/config if they don't already have one.

Don't forget to rm -rf pki to delete the files you downloaded from S3.

Tuesday, January 9, 2018

Why I left Arista Networks

5 years ago, I wrote a blog post on why I joined Arista Networks back in 2012.  As I am now suddenly and unexpectedly leaving the company, I figured I'd write a bit of a retrospective and perhaps bring some closure to this otherwise fairly quiet blog.  I know that the original blog post has been used by candidates considering to join Arista, and even though I didn't write it with this in mind originally, I wanted to give a bit of an update to those considering to join the company in 2018 and beyond.

Why I left Arista

I was very happy and thriving at Arista and wasn't looking for a change.  But I guess change was looking for me and somehow managed to convince me to join a new startup as co-founder.  I won't say much more on that topic for now but it's one of those opportunities that was too big to pass up on.  It's not in the networking industry, so not competitive with Arista.

I really struggled with this change, it took some massive amount of questioning things to accept the idea to leave such a great company, with a great team, working on great projects, to throw myself into the unknown and push myself way outside of my comfort zone.  But I felt like I had to try, I had to seize this opportunity.

Arista in 2018

Everything that I wrote in my original blog post is still true as far as I'm concerned.  The big difference is that in the meantime Arista has established itself as one of the truly remarkable success stories in recent Silicon Valley history.

Now “Arista Networks” may not be a household name like Google or Facebook, but make no mistake, Arista's success in the networking industry is on the same track as Google's success in search or Facebook's success in social media.

Many others have tried (or are trying) to claim a piece of the networking cake dominated by Cisco, and I really cannot think of any other company succeeding in any meaningful way in that space.  If anything, previously established players have all but disappeared (e.g. Force10, Brocade) or become largely irrelevant (e.g. Extreme).  As the two remaining industry giants, Cisco and Juniper, are tumbling, steadily losing market share and focus, the brightest rising star in the datacenter networking industry has been Arista.  And yet Arista still only commands a low double digit market share, so there is a lot of room to grow further while also strategically expanding the TAM (Total Addressable Market).

There are a number of tailwinds benefiting the company:
  • Competitors still can't get their act together and continue to overpromise and underdeliver.  Quality issues continue to plague them.  Arista manages its roadmap carefully and will not hesitate to say "no" to a customer if they cannot commit to what the customer is asking for, rather than promise something that they know cannot be delivered on time or at all.  Quality remains paramount and the team is constantly trying to improve automated testing processes to ensure that every new release that comes out is better than the previous one and that no regression sneaks back into the code.  This includes things like automatically running tests based on what code changed by leveraging code-coverage information gleaned during earlier test runs, automatically triaging and root-causing unexplained test failures, and more.  There is a strong emphasis on building/improving tools and creating a development environment where everyone can be productive [1].
  • The routing industry is collapsing in the datacenter networking industry. This trend started a couple years ago and should by now be clear to anybody in the industry.  The gap between a "switch" and a "router" has been shrinking steadily to the point that we now commonly see datacenter switches play the role of edge peering boxes, backbone routers, cross-datacenter interconnects, etc.  This is hurting Juniper particularly bad, because this space was their bread and butter.  But with the wrong hardware and the wrong software, they cannot compete with the density and cost per port of commodity hardware.  The only lead they kept, and mostly the only differences that remain between switches and routers, are in specialized routing software.  And since Arista is a software company, not a hardware company, the team has been hard at work to implement routing features and scale the routing code way beyond what has ever been done on datacenter networking platforms.  This is probably one of the biggest boost to Arista's TAM and much work remains to be done in that space to close that gap fully.  It's very exciting.
  • Arista has been leading innovation in the networking industry. Whenever a new chip comes out, Arista is often the first to make it bridge a packet, sometimes before the chip vendor has done it themselves.  On many occasions, Arista has managed to push the hardware at a scale that exceeds the data sheet of the underlying hardware.  This is only made possible by Arista's edge on the software front.  Furthermore, Arista has influenced chip design with the silicon vendors they partner with to further widen the gap between the cost/performance of commodity hardware and vendor-proprietary ASICs like those designed at great cost by Cisco and Juniper.  Arista has been leading industry standards like 25/50G and more recently 200/400G, with the new OSFP initiative.  Arista was the first to take to market new technologies like VXLAN, internet-scale routing in a sub-$20k 1RU top of rack switch, streaming telemetry and network programmability, etc.
  • Arista's execution has been flawless.  The company faced some pretty serious challenges, including a set of massive lawsuits from the 800 pound gorilla with a virtually unlimited legal budget that would stop at nothing to slow them down or tarnish their image.  Despite all this, the company kept its head down and its focus, fought fearlessly for what was right, and managed to deliver 14 consecutive "beat and raise" quarters that turned it into a Wall Street darling.  This is really a function of the amazing exec team that has been at the helm of the company.
  • Arista is in the segment of the networking industry that is growing the fastest. There are a lot of products and areas in the overall networking industry but datacenter networking is the one growing the fastest, because everything is going to the cloud, and the cloud runs on this stuff.  Arista has managed to remain laser focused on this specific segment of the industry, slowly expanding into connected areas where opportunities existed to go after some low hanging fruits (e.g. tap aggregation, routing, and more).  Arista is present at a large scale in virtually all the major cloud environments out there.  Again, the name might not quite have the mindshare of a Google or a Facebook, but these days it's virtually impossible to use the Internet without going through Arista devices.
And while the headcount has more than quintupled since I joined, the company has managed to remain surprisingly apolitical and bullshit-free.  There have been growing pains, for sure, and it's not like everything is perfect and just happy rainbow unicorns either, but the company culture is essentially unchanged, and that's what actually matters.

So it was really, really, really freaking hard to say goodbye.  I've been lucky to be very happy everywhere I worked in my career, but to this point Arista has been by far the best company I've worked at.

So... As Douglas Adams would say: So long, and thanks for all the fish.

[1] A footnote worthwhile adding regarding the emphasis on tooling.  Ken Duda, one of the co-founders, is very involved in developer tools.  After becoming a Go fanboy he spent months working on a new way to put together development workspaces using Docker containers.  There are several people working with him on this new tool now and it has become the de-facto standard way of managing Arista's massive workspaces, which comprise millions of lines of code and often need to pull in tens of gigabytes of stuff.  This has saved everybody a lot of time and helped support / enable changes to the CI (Continuous Integration) workflow.

Additional disclaimer for this post: the views expressed in this blog are my own, and Arista didn't review/approve/endorse anything I wrote here.